When Prevention Isn't Detection

The assumption

Security tooling vendors sell "prevention and detection" as a single capability, usually as two checkboxes on the same feature list. It's easy to absorb the implication: buy the platform, and you get both, roughly for free, roughly at the same time. I believed some version of this for longer than I'd like to admit: that a WAF blocking a SQL injection attempt was, functionally, also detecting it, because obviously something logged the block.

The problem

Prevention and detection optimize for opposite failure modes, and that's the part the shared dashboard hides. A prevention control's job is to be conservative: when in doubt, block, because a false positive costs you a legitimate request and a false negative costs you a breach. A detection control's job is closer to the opposite: it needs to surface enough signal to be useful without drowning the person reading it in noise, which means it's tuned to under-alert rather than over-block.

Those are different optimization targets. A system tuned well for one is not automatically tuned well for the other. And because prevention is the flashier, more demoable capability ("look, it stopped the attack live"), it's usually where the engineering investment goes first, with detection treated as a byproduct rather than its own discipline.

This shows up in how products are built, not just how they're marketed. A prevention feature has a natural, satisfying acceptance test: throw the attack at it, watch it get blocked, ship it. A detection feature has no equivalently crisp acceptance test, because "is this useful to a human trying to reconstruct an incident" is a judgment call that depends on who that human is, what they already know, and what they're trying to decide, not a pass/fail assertion a test suite can check in CI. Engineering organizations reliably build more of what's easy to write a clean test for. That's not a moral failing on anyone's part; it's what happens by default when one property of a system is machine-checkable and the other one only shows up when a person, under pressure, tries to use the output for something the test suite never modeled.

Two failure functions, not one

It's worth stating the asymmetry precisely, because it's easy to nod along with "different optimization targets" without registering how differently the two systems actually fail. A prevention control that's too aggressive produces false positives: legitimate traffic gets blocked, someone complains, and the cost is visible almost immediately, which creates fast, self-correcting pressure to loosen it. A prevention control that's too permissive produces false negatives, which are invisible until the breach that resulted from them is discovered, often much later, by someone other than the team that configured the control. A detection control has the mirror problem: too sensitive, and it produces alert fatigue, which quietly trains the humans reading it to skim past real signal; too quiet, and it produces false confidence, a security dashboard with no red on it that's read as "nothing happened" rather than "nothing was noticed." Because the feedback loops for these four failure modes operate on wildly different timescales, from immediate (a blocked legitimate user complains today) to indefinite (a missed attack surfaces months later, if ever), the systems that are cheap to tune well are exactly the ones that get tuned, and the ones with long or absent feedback loops drift.

The experiment

I wanted a number, not just an intuition, so I went back to the admission-control experiment and looked specifically at what changed between the prevention outcome and the detection outcome as I varied one thing: audit verbosity. At the default level, prevention was 100% effective (every disallowed pod spec was rejected), and detection quality, measured as "could I reconstruct what was attempted," was close to zero. At the highest verbosity level, detection quality was excellent, but the audit log volume increased by roughly 40x, which is the actual reason most clusters don't run at that level.

Measuring detection quality, not just log volume

"Detection quality" needed a real definition before it was worth measuring, so I scored it against a fixed rubric applied to a batch of forty submitted violations: could I identify, from the log alone, which resource violated which rule (1 point); could I identify the exact field and value that caused the violation (1 point); could I reconstruct the full submitted spec well enough to tell whether the violation was a single mistake or part of a pattern across multiple submissions (1 point); and could I do all of that without cross-referencing a system other than the audit log itself, such as the cluster's live state or a separate CI log (1 point). A perfect score was 4 out of 4 per incident. At the default audit level, the mean score across the forty incidents was 0.3. At maximum verbosity, it was 3.9.

# Kyverno audit level comparison, same policy, two configurations
# default
spec:
  validationFailureAction: Audit
  background: true
# maximum verbosity (what it takes to reach the 3.9 score)
spec:
  validationFailureAction: Audit
  background: true
  validationFailureActionOverrides: []
  emitEvents: true
  admission: true
  webhookTimeoutSeconds: 30
# plus: cluster audit policy set to log full request/response bodies
# for admission review objects, not just metadata

The gap between 0.3 and 3.9 wasn't produced by a smarter detection algorithm. It was produced entirely by turning on request-body logging that most clusters leave off by default, specifically because it's the setting responsible for most of that 40x volume increase.

The proof

Prevention and detection moved independently

Prevention quality and detection quality didn't move together. They were almost orthogonal: you could hold prevention constant at "always blocks the bad pod spec" while detection quality swung from unusable to excellent, purely as a function of a setting most teams never revisit after initial cluster setup. That's the concrete version of the abstract claim: these are two different systems wearing one name, and tuning one does nothing to the other.

The volume cost wasn't uniform across incident types

The 40x figure is an average, and averages hide something useful here too. Breaking the log volume increase down by event type, the overwhelming majority of the added volume came from routine, compliant submissions gaining full request-body logging alongside the violations, not from the violations themselves, which were a small fraction of total admission traffic in a healthy cluster. In other words, most of the cost of good detection was being paid to log the boring 99% of traffic that never violated anything, in order to be ready for the 1% that did. That reframes the trade-off: it isn't really "cheap detection versus expensive detection," it's "pay a fixed background tax on all traffic, all the time, so the rare incident is fully reconstructable when it happens." Framed that way, the 40x number stops looking like an argument against high verbosity and starts looking like the actual, unavoidable price of detection as a capability, which is closer to insurance than to a tunable feature.

# where the extra log volume actually went (proportion of total increase)
compliant submissions, full body logged:   88%
violating submissions, full body logged:    9%
system/internal admission review traffic:   3%

You might disagree

You could reasonably argue that for a huge share of security events, you don't need detection quality at all: if the WAF blocked the SQL injection attempt, the incident is over, and spending engineering time reconstructing exactly which query was tried is a research exercise, not an operational necessity. For a single blocked request, sure. But attackers rarely try once. The value of detection isn't explaining one blocked attempt. It's noticing the pattern across a hundred blocked attempts that tells you this isn't noise, it's reconnaissance, and it's increasing. Prevention alone can't tell you that a wall is being tested with increasing sophistication. It can only tell you the wall, so far, has held. I'd add, given the volume breakdown above, that this argument gets stronger rather than weaker once you see where the logging cost actually comes from: if 88% of the added log volume is spent on traffic that was never a threat, the marginal cost of also being able to reconstruct the 1% that mattered is much smaller than the headline 40x number implies, because you're already paying almost all of that cost regardless of whether any given day includes an actual attack. The number that looks scary in aggregate is mostly overhead you're paying either way; treating it as the reason to skip detection is treating a sunk, largely fixed cost as if it were the marginal one.

What I think now

I now treat "prevention" and "detection" as two line items in any security design, not one. When I add a control, I ask what it prevents and, separately and explicitly, what it lets me detect, and if the second answer is thin, I treat that as a real gap, not an acceptable trade-off I get to ignore because the first answer was strong.

That's changed how I evaluate new tooling, too. A vendor pitch that leads with a live block demo now gets a specific follow-up question before anything else: show me the incident report your platform would produce for the same event, using default settings, and let me judge whether it's something a person who wasn't in this demo could act on. If the answer is a screenshot of a single "blocked" row with a severity color and nothing else, that's not a detection capability, whatever the feature list says. It's a prevention capability with a detection-shaped label on it, and the two are worth paying for separately, with separate scrutiny, because a platform that's excellent at one has told you nothing about whether it's even adequate at the other.

It's also changed the questions I ask about cost, specifically. Given the volume breakdown from the experiment, "detection is too expensive" almost always deserves a follow-up: expensive relative to what baseline, and is the cost actually the incremental price of catching the rare incident, or is it the fixed price of instrumenting the routine 99% that most teams were already going to want visibility into for reasons that have nothing to do with security (debugging, capacity planning, understanding what a pipeline actually submits). In the cluster I tested, a meaningful share of that "detection cost" would have paid for itself in ordinary operational debugging even if no incident had ever occurred. Bundling it entirely under "the price of catching attackers" overstates the marginal cost of security specifically and makes it an easier line item to cut when budgets tighten, which is exactly backwards from where I'd want that pressure to land.

The lesson learned

A dashboard that shows a blocked request has told you about one moment. It has not told you about the pattern that moment belongs to. Building for detection is a distinct investment from building for prevention, and skipping it doesn't make the risk go away. It just moves the cost from "engineering time now" to "incomplete incident report later," and by the time that incomplete report matters, the fixed cost you declined to pay earlier only became more expensive to pay retroactively.