Skip to main content

The technologies change. The paradigm does not.

Why Event-by-Event Detection Misses Insider Threats

物語

For two decades, the security industry has repeated the same pattern: ingest massive volume, build detections for undesirable behavior, display the results, and pray someone acts on them.

A security engineering portfolio project by Vyshyvka Studio

VolumeDetectDisplayPray
VolumeDetectDisplayPray

The Deluge

A mid-size enterprise generates approximately one billion security-relevant events per day across 100+ data sources. Firewalls, endpoints, identity providers, cloud APIs, email gateways, DNS resolvers, CASB proxies. Each vendor produces its own log format. Each log contains its own schema.

1B+
Events / Day
100+
Data Sources
80+
Log Formats
2 TB+
Storage / Day

Event Sources

Network & Firewall
400M
Endpoint Telemetry
200M
DNS Queries
150M
Cloud API Calls
120M
Authentication Events
80M
Email/DLP Events
50M

No human can read a billion events. So we built machines to read them for us. The machines produced alerts. Step one of the paradigm: volume → detect.

VolumeDetectDisplayPray

The Detections

Three generations of detection technology. Each one genuinely better than the last. None of them solve the problem — because the problem isn't detection quality.

The first generation response: write detection rules. If X happens more than Y times in Z minutes, fire an alert. Thousands of rules. Thousands of SIGMA signatures. Thousands of correlation searches.

Brute Force Detection
IF failed_logins > 5 IN 10min THEN alert
847 alerts/day94% false positive
Data Exfiltration
IF egress_bytes > 500MB IN 1hr THEN alert
234 alerts/day89% false positive
Lateral Movement
IF new_host_login > 3 IN 24hr THEN alert
1,203 alerts/day97% false positive
Privilege Escalation
IF sudo_events > 10 IN 1hr THEN alert
412 alerts/day91% false positive
Impossible Travel
IF geo_distance > 500mi IN 1hr THEN alert
89 alerts/day78% false positive
~5,000 rules
Total Detectors
~5,000
Alerts / Day
92%
False Positive Rate

5,000 alerts per day. 92% false positives. 400 real signals buried in 4,600 false alarms. Step two of the paradigm: detect → display.

VolumeDetectDisplayPray

The Graveyard

Step 3: Display. If you build it, they will come.

Alerts arrive in emails, Slack channels, SIEM dashboards, ticketing systems. They pile up. Channels get muted. Emails get filtered. Dashboards go stale. The SOC analyst learns to ignore.

OOutlooksecurity-alerts@corp.com
Filter
Focused
Other
1,247
SIEM AlertBot2:41 PM
[CRITICAL] Brute force detected on prod-db-03
This alert requires immediate investigation. Source IP has been flagged in...
SIEM AlertBot2:38 PM
[HIGH] Suspicious login from unrecognized geo
This alert requires immediate investigation. Source IP has been flagged in...
SIEM AlertBot2:34 PM
[HIGH] Lateral movement alert — finance-ws-12
This alert requires immediate investigation. Source IP has been flagged in...
SIEM AlertBot2:31 PM
[MEDIUM] DLP policy violation — outbound USB
This alert requires immediate investigation. Source IP has been flagged in...
SIEM AlertBot2:28 PM
[CRITICAL] Privilege escalation attempt on DC-01
This alert requires immediate investigation. Source IP has been flagged in...
SIEM AlertBot2:22 PM
[MEDIUM] Anomalous DNS query pattern detected
This alert requires immediate investigation. Source IP has been flagged in...
1,242 more unread
slack|# security-alerts
🔇
New
AB
AlertBotAPP2:34 PM
P1

Brute force: 47 failed logins to prod-db-03 from 185.234.xx.xx in 3 minutes

👀 2✅ 1
AB
AlertBotAPP2:31 PM
P2

UEBA: Entity risk score for jsmith exceeded threshold (0.91)

AB
AlertBotAPP2:28 PM
P2

Impossible travel: auth from NYC → Singapore in 12 minutes

AB
AlertBotAPP2:22 PM
P3

DLP: 340MB outbound transfer to personal cloud storage

847 more messages today
Jira|Projects/SEC/Board
SEC Queue — 423 Open
KeySummaryPStatusAge
SEC-4821Investigate brute force on prod-db-03To Do3d
SEC-4819Review suspicious geo login — jsmithTo Do5d
SEC-4815Lateral movement finance-ws-12To Do7d
SEC-4808Anomalous DNS exfil patternOpen12d
SEC-4801USB policy violation — marketing deptOpen18d
418 more open tickets
splunk>Security Posture
index=security sourcetype=alert | stats count by severity
Last 24h ▾
Alert Severity Distribution▼ ⟳ ⋯
47
Critical
234
High
1,892
Medium
Indexing Load94%
4.2 days
MTTR
423
Backlog
18 months
Avg Tenure
40%/year
Turnover Rate
~500/day
Alerts / Analyst
~30 seconds
Time / Alert

The alerts are not being ignored because analysts are lazy. They are being ignored because no human workflow can absorb this volume. This is a systems failure, not a people failure. Step three of the paradigm: display → pray.

VolumeDetectDisplayPray

The Prayer

Step 4: Hope someone acts. Will they?

What happens after the alert is displayed? A human must decide. One alert out of five thousand. Thirty seconds to judge. And 400 more waiting behind it.

The Triage Decision

Alert arrives (one of 5,000 today)
├──
Obviously critical?5%
Investigate
├──
Known false positive pattern?60%
Close
├──
Not sure, 400 more to go30%
Acknowledge, move on
├──
Lost in the noise5%
Never seen

The Dwell Time Gap

Day 1Day 21 — Breach Discovered
Investigated (<5%)~105,000 alerts generated
21 days
Avg Dwell Time
~105,000
Alerts During Dwell
~30 seconds
Time per Alert
<5%
Investigation Probability

The Evidence

Target (2013)

FireEye alerts were generated. SOC team did not investigate. The paradigm displayed. Nobody prayed hard enough.

$162M

Equifax (2017)

IDS alerts for the Apache Struts exploit were missed for 78 days. The pipeline detected. The paradigm failed at display → pray.

$1.4B

SolarWinds (2020)

Anomalous network activity was logged but not correlated for 9 months. Volume was ingested. Detection existed. The paradigm failed at the last step.

$100M+

Every major breach in the last decade was preceded by alerts that nobody acted on. Not because analysts are lazy — because the paradigm produces more signal than any human can process. The prayer never gets answered.

VolumeDetectDisplayPray

The Fix

This time is different! Except it isn't.

After each breach, the industry responds with better technology. The current wave promises that large language models, retrieval-augmented generation, MCP tool servers, and agentic workflows will finally solve the problem. The technology is genuinely impressive. The architecture is identical.

Cross-Log LLM ContextVolume

LLMs ingest logs from dozens of sources simultaneously, understanding semantic relationships across firewalls, identity providers, and cloud APIs that no regex or statistical model could correlate.

The ingestion is genuinely better — semantic correlation across log sources is a real capability rules never had. But the output is still a classification: is this event suspicious? The pipe is smarter. The question is the same.

RAG + MCP RetrievalDetect

Vector databases retrieve relevant runbooks, past incidents, and threat intel. MCP tool servers give agents live access to org context — asset inventories, employee directories, change tickets.

The detection context is richer — the classifier genuinely has more information than any prior generation. But it still classifies each event in isolation. A user authenticating from a new location is evaluated without the access pattern that preceded it. Richer context per event is not the same as understanding across events.

Agentic TriageDisplay

AI agents perform multi-step investigation: querying data sources, building timelines, correlating evidence across systems, testing hypotheses, and producing structured incident reports.

The investigation is real — agents correlate evidence and reason about attack chains in ways dashboards never could. But the output is still a finding routed to a queue that grows faster than humans can drain it. A better investigation upstream does not remove the bottleneck downstream.

Autonomous ResponsePray

Agents execute remediation playbooks, isolate hosts, revoke credentials, and close tickets without human intervention.

The automation is real — playbooks execute faster than any human. But automated response built on per-event classification inherits every false assumption at machine speed. When the paradigm is wrong about a threat, the automation doesn't just miss it — it closes the ticket and moves on.

Every claim is true. The technology is genuinely better at each step. But better execution of the same paradigm does not change the paradigm. A smarter pipe is still a pipe.

VolumeDetectDisplayPray

The Fragility

More components. More connections. More ways to fail silently.

Each generation adds complexity. The circuit boards below show the actual dependency graphs — every component that must work correctly for the system to function. Click the button to see what happens when a single component fails.

Log ParsersSIEMRulesRoutingDashboard
5 components

Visible — you know when it breaks.

e.g.Failed logins > 5 in 10 minTraffic to known C2 IPOutbound transfer > 1 GBPort scan from internal hostPrivilege escalation pattern

The log parsers break — a vendor pushes a firmware update, the syslog format changes. Immediately, the SIEM can't ingest. Rules stop evaluating because there's nothing to evaluate. Alert routing goes quiet. The dashboard goes dark. Every downstream component fails in seconds, and everyone knows — the SOC sees empty screens, the CISO gets a call. Five components, one point of failure, total visibility. This is a pipeline you can reason about.

This graph is shared across all rules. One pipeline, many rules. When it breaks, it breaks for everything — but at least there's only one thing to fix.

THE FRAGILITY

Each generation makes the machinery more complex and the failure modes less visible. Gen 1 fails loudly — dashboards go dark, rules stop firing. Gen 2 fails silently — models drift, scores degrade, nobody notices. Gen 3 fails misleadingly — agents confabulate, produce confident wrong answers, and the system looks like it's working perfectly while it's not.

VolumeDetectDisplayPray

The Paradox

We built a machine more complex than the problem it was designed to solve.

The problem was simple: too many events for humans to process. Three generations later, the solution is thousands of fragile, interdependent components in service of a question that was wrong from the start.

Total running components

Gen 1You know when it breaks.5
Gen 2You don't know when it breaks.~156
Gen 3It looks like it's working when it isn't.86,000+

The problem was volume — too many events for humans to read. The solution is machinery so complex no human can reason about it. The cure is harder to operate, debug, and trust than the disease. Each generation deepens the trap: the paradigm's only answer is another detection.

The Paradigm's Question

Is this event bad?

Rules classify with regex. ML classifies with statistics. LLMs classify with context. Three generations of increasingly complex, increasingly fragile machinery — all in service of binary event classification. The architecture changes. The question never does.

A Different Question

Does this story make sense?

Not 'is this login suspicious?' but 'does this person's behavior over the last three months tell a coherent story?' Not event classification — narrative comprehension. A question the paradigm's architecture cannot even ask.

Alert Fatigue Curve

Attention % vs Alerts/Day
10
50
100
500
1k
2.5k
5k

The paradox is not that detection doesn't work — detection works exactly as designed. It classifies events. The paradox is that no amount of event classification can catch a threat where no single event is suspicious. The most dangerous things happening on your network are not bad events. They are authorized, normal-looking, policy-compliant events that tell a story no one is reading.

VolumeDetectDisplayPray

Volume. Detect. Display. Pray.

The paradigm has not changed in twenty years.

Better rules did not fix it. ML did not fix it. LLMs will not fix it.

What if we stopped asking 'is this event bad?'

What if we asked 'does this behavior make sense?'

Not 5,000 alerts. Five stories that broke.

See The Idea →

See it in practice

According to the SEC, unauthorized changes to fourteen Two Sigma investment models contributed to approximately $165 million in client harm over 21 months. The sequence crossed control boundaries without being understood as a whole.

Read the Case Study