Suspecting the Unsuspected. Extracting and Analyzing Log Anomalies

Simon27 May 2022

Problem

Security teams are asked to find malicious users in logs without knowing what an attack looks like and without labels. The usual shortcut, a list of top talkers (actors with the most events), catches crawlers and brute-force bursts. But heavy legitimate users fill the top of that list and hide actors who do rare things in small numbers. Expert-assigned action weights need a clear idea of what is normal, and a machine-learning model needs training and serving infrastructure.

Example from the article: in a simulated shop with 8 users, a power user with 41 actions tops the count ranking, while malicious users sit in the middle. In real Google Workspace API logs, an insider who scripted document search ranked 104th by action count.

Solution in brief

Give each event a weight equal to its information content, −log2p(x), where p(x) is how often that action occurs in a reference period. Sum the weights per actor and sort. Rare actions count a lot, frequent ones such as search count little, and the whole thing runs as one SQL query.

Questions this case answers
  • How to rank actors in an unlabeled event log by how unlikely their action mix is, when raw activity volume is dominated by a few heavy but harmless users
  • How to find unusual or malicious users in event logs when nobody can say in advance what an attack looks like?
  • How to rank actors by how unexpected their actions are, instead of by how many actions they performed?
  • How to build log anomaly detection with SQL only, without training and serving a model?

Tools and methods

  • Information content (surprisal) scoring — weights each action by −log2 of its probability, summed per actor.
  • z-score — puts the summed bits on a common scale for thresholds.
  • Top-talkers ranking — the baseline, every event weighs 1.
  • BigQuery SQL — lookup table and scoring in one query.
  • pandas — prototype and comparison of the two rankings.
Continue reading — it's free

Google or a one-time email code. No password.