Lakshya

Security Engineering & Tooling · Chapter 4 of 8

Runtime, endpoints, and why SIEM economics are the architecture

The CrowdStrike outage as engineers should tell it, what eBPF actually changed, and why every SIEM programme becomes an argument about which logs you can afford not to collect.

6 min read1 diagramAll 8 chapters

One expansion first, because the rest of the chapter leans on it. A SIEM — Security Information and Event Management — collects logs from everything you own (servers, laptops, firewalls, cloud services, applications), keeps them in one searchable place, and runs rules over them looking for attacks. The reason it exists is correlation: a failed sign-in means nothing, but a failed sign-in, then a success from a new device, then a mailbox rule forwarding externally is an attack — and no single system can see it, because each holds only its own piece.

The July 2024 outage, told correctly

WHAT ACTUALLY HAPPENED — NOT THE NULL-POINTER STORY EVERYONE REPEATS 1The designRapid Response Content — behavioural config shipped SEPARATELY from sensor code, to push detections fast2March 2024An IPC Template Type defines 21 input fields. The sensor supplies 20. Not validated at compile time. Ships LATENT3Mar–Apr 2024Template Instances use WILDCARD matching on field 21, so the missing input is never dereferenced. Dormant four months419 Jul 2024Channel File 291 is the first instance to use a NON-WILDCARD criterion on the 21st field5The crashContent Interpreter reads out of bounds past a 20-element array → unhandled exception IN KERNEL CONTEXT → BSOD, everywhere, at once THE WIDELY-REPEATED VERSION IS WRONG Not a null-pointer deref from a file of zeros. All-zero files on disk are a SYMPTOM of the crash — written as the machine died. THE LESSON IS RELEASE ENGINEERING, NOT SECURITY Mature staged rollout for sensor CODE. Effectively none for CONTENT — which a kernel-context parser interpreted. If your data can crash your parser, your data is code, and it needs your code’s release process. Six distinct failures were enumerated: no compile-time field count check · no runtime bounds check · a Content Validator logic error ·template testing that never varied the matching criteria · instance validation that never ran the Interpreter · no staged deployment forcontent.
A good interview discriminator, because the wrong version is repeated in vendor blogs and explainer videos, and the vendor published a specific rebuttal of it.

What changed afterwards is worth knowing because it is what you would have asked for: runtime bounds checks in the Content Interpreter; template testing across varied matching criteria; instance validation that actually exercises the Interpreter; staggered canary deployment for Rapid Response Content with monitoring gating each ring; customer control over when and where content updates land; and independent third-party code and process reviews.

The commercial lesson is the uncomfortable one. The stock fell about 45% over eighteen days. Two years later the recovery is unambiguous — ending ARR of $5.51B growing 24%, a record first-quarter net-new ARR, four consecutive quarters of revenue acceleration. The largest IT outage in history cost the responsible vendor essentially no market share. Switching costs in endpoint security are enormous, and that should inform how much weight you put on “we would just replace them” as a risk control.

What eBPF actually changed

Before it, runtime visibility meant a kernel module — crash it and you take the host, as above — or ptrace-style interception, which was too slow to run everywhere. eBPF gives verified, sandboxed, JIT-compiled in-kernel programs with bounded execution. The verifier is the whole point: a bad eBPF program is rejected at load time rather than panicking the machine. That is the direct architectural answer to the failure mode above, and it is why endpoint vendors are being pushed out of the kernel.

  • Falco — created at Sysdig in 2016, donated to the CNCF in 2018, graduated February 2024, with maintainers now including Amazon, Apple, IBM and Red Hat. The de facto Kubernetes threat-detection engine.
  • Tetragon — part of Cilium. Its differentiator is in-kernel enforcement via LSM hooks: it can block, not just alert, without a round trip to userspace. That matters, because a detection that must leave the kernel to decide has already lost the race against a fast attacker.
  • Tracee — Aqua's equivalent, and part of the same open-source-scanner-as-loss-leader pattern.

What eBPF still cannot do: distinguish malicious from legitimate use of a legitimate syscall without context. curl | bash in a CI runner is a build step; in a production pod it is an incident. The kernel sees identical syscalls. Runtime security is a context problem wearing a telemetry costume — which is why the false-positive burden lands here too.

SIEM — the pricing model is the architecture

The defining constraint: SIEM is priced on ingest volume while security value comes from coverage. Those pull in opposite directions, so every SIEM programme becomes a negotiation about which logs you can afford not to collect — and the engineering discipline that creates, pipeline filtering and routing and tiering, is now a named job function.

PlatformModelIndicative list (all heavily discounted in practice)
Microsoft SentinelPer-GB analytics tier + a cheap data-lake tier~$4.30/GB pay-as-you-go, falling to ~$2.05 at very high commitment. Eligible M365 E5 security logs ingest free
Splunk CloudPer GB/day, tapering with commit~$1,000 per GB/day/year around 50 GB/day. Enterprise Security typically adds 1.5–2×
Google SecOpsData volume against a package capThe famous flat per-employee unlimited-ingest model no longer exists. Anyone quoting it is quoting 2021
PantherVolume meter~$50,000/year per TB of monthly ingestion
Elastic SecurityResource-based, self-hostableThe main lever in cost-driven migrations

“How would you cut SIEM cost?” — and the honest caveat

The wrong answer is “reduce retention”. The fix is architectural: split detection from retention. Keep a small, high-value, real-time set in the SIEM; land everything else in cheap object storage; query it federated when investigating.

This is now a product direction, not a theory: Splunk shipped federated search over Snowflake (GA July 2026), Microsoft has the Sentinel data-lake tier, and SentinelOne bought a pipeline vendor for $225M explicitly “to help customers decouple data ingestion from SIEM platforms.” That acquisition rationale is the clearest statement of the trend from anyone with money at stake.

Now say the cost out loud, because most candidates do not. Reference architectures claim 70–80% savings, and the saving is real — but you are trading query latency and detection coverage for it. Detections do not run on data you did not ingest. Move the wrong source to cold storage and you find out during an incident, when the query takes forty minutes.

Which makes it a detection-engineering decision, not a finance one — and it is routinely made by finance. Saying that is the senior half of the answer.

Check yourself

What actually caused the CrowdStrike outage?A template declaring 21 input fields against sensor code supplying 20, unvalidated at compile time, latent for four months until the first non-wildcard match on field 21 caused an out-of-bounds read in kernel context.
What is the null-byte story?Wrong, and specifically rebutted by the vendor. All-zero channel files on disk are a symptom — files written just before the machine died — not the cause.
What is the generalisable lesson?If your data can crash your parser, your data is code and needs your code's release process. There was staged rollout for sensor code and effectively none for content.
Why is the eBPF verifier the point?A bad eBPF program is rejected at load time rather than panicking the machine. It is the direct architectural answer to a kernel module taking down the fleet.
What can runtime security not decide?Whether legitimate syscalls are malicious. curl | bash is a build step in CI and an incident in production; the kernel sees the same calls.
How do you cut SIEM cost properly?Split detection from retention — high-value sources in the SIEM, verbose ones in object storage, queried federated. And say the cost: detections don't run on data you didn't ingest.
← Cloud security — CNAPP, agents, and the alert-volume problemThe market — consolidation, pricing, and what it means for you →