The assumption
Early on, I thought about security the way most engineers do before they've had to clean up after an incident: as a series of gates. You put a gate in front of the thing you want to protect, the gate has rules, and if the rules are good, bad requests don't get through. A firewall rule. An RBAC policy. An admission webhook that rejects a misconfigured pod. Deploy the gate, watch it reject the bad request, close the ticket. It's a satisfying model because it gives you a clean unit of success: the request either got through or it didn't, and if it didn't, you did your job.
For a long time, "the control worked" and "we're safe" were the same sentence in my head. I don't think I'd ever stated that equivalence out loud, or even to myself, but it showed up in every design review I sat in: the question was always "will this stop the bad thing," and almost never "if this stops the bad thing, what will we know afterward." Nobody was being negligent. The gate model just doesn't have a slot for the second question, because in a gate model, once the gate holds, there's nothing left to ask.
The key idea
A security control that successfully blocks an action has only answered one question: did this specific attempt succeed? It hasn't told you who tried, how often, or what they were actually attempting. Prevention and evidence are two different things a control can give you, and most controls are only built to give you one.
The problem
The sentence started coming apart the first time I had to explain, after the fact, exactly what had been attempted against a system I was responsible for. Not whether something bad had happened; I knew it hadn't, because the control had blocked it. What I couldn't answer was what had been attempted, how many times, from where, and whether it looked like a mistake or a probe. Those aren't exotic questions. They're the first four things anyone asks in any post-incident conversation, security or otherwise, and I had answers to none of them.
The honest answer was: I don't know. The control did its job. Nobody could tell me anything else.
That's a strange position to be in. We talk about security controls as if their job is binary: stop the bad thing, don't stop the good thing. But a control that stops something is also, whether we've designed for it or not, a witness. And most of the controls I'd deployed were bad witnesses. They'd testify that something happened and refuse to say what, the security equivalent of a witness who confirms a crime occurred but can't describe the person, the method, or the time.
The distinction I hadn't been making
Prevention and evidence get treated as a package deal because they're often implemented by the same piece of software. A single admission webhook both decides whether to allow a request and is the only thing positioned to observe the request in detail. It's easy to assume that if the software is doing the first job, it's automatically doing the second, since the data needed for both jobs technically passes through the same code path at the same moment. But "the data passed through" and "the data was recorded in a form anyone could later query" are very different claims, and conflating them is exactly the mistake I'd been making without noticing.
What changed my mind
The moment this became concrete for me was staring at a Kubernetes audit log after a policy correctly rejected a batch of pods requesting hostNetwork: true, a real privilege-escalation vector, not a hypothetical. hostNetwork: true lets a pod share the host's network namespace directly, which is precisely the kind of misconfiguration (or deliberate escalation attempt) an admission policy exists to catch. The policy worked. Twelve times. What I wanted from the audit trail was simple: who tried this, and was it increasing.
$ kubectl logs -n kube-system kube-apiserver-control-plane-1 | grep hostNetwork
# nothing: hostNetwork isn't a field the default audit policy surfaces
# what the default audit log actually recorded, per rejected request
{
"kind": "Event",
"verb": "create",
"objectRef": {"resource": "pods", "namespace": "default"},
"user": {"username": "system:serviceaccount:default:ci-deployer"},
"responseStatus": {"code": 400, "reason": "hostNetwork not allowed"},
"requestObject": null,
"responseObject": null
}
What I got, at the audit verbosity most clusters ship with by default, was a verb, a response code, and a username. No pod spec. requestObject and responseObject are both null at the default Metadata audit level, which means the exact thing I most wanted, the actual field that tripped the policy and the actual value it was set to, was the one piece of the event explicitly omitted. No way to distinguish this denial from any other kind of validation failure; a hostNetwork: true rejection and a completely unrelated schema-validation rejection produce audit entries that are, at this verbosity, indistinguishable from each other. I could prove the wall held. I couldn't describe what hit it.
I wrote up the full version of this as an experiment in The Lab: running the actual attempt, checking the actual audit trail, measuring the actual cost of getting better evidence. The short version: the gap wasn't a bug. It was the default, and the default exists because verbose auditing is expensive at scale, both in raw storage and in the write load on the API server's audit backend under a busy cluster. Someone, somewhere, made a deliberate trade between visibility and cost, and that trade had quietly become mine without my choosing it, the moment I adopted the platform's defaults without re-examining what they actually gave up.
What it cost to close the gap
Raising the audit level from Metadata to RequestResponse for the specific resource and verb combination that mattered (pod creation attempts hitting this policy) restored the missing detail, requestObject included, at a measurable but bounded cost: audit log volume for that narrow slice roughly quadrupled, dominated by the size of full pod specs rather than the terse metadata-only events. That's a manageable cost when scoped to a specific, security-relevant resource and verb. It would not be a manageable cost applied blanket, across every resource and every verb in the cluster, which is presumably close to the reasoning behind why Metadata is the sane default in the first place. The fix isn't "turn up auditing everywhere." It's "know exactly which few controls you actually need full evidence from, and turn it up only there."
The uncomfortable part
The uncomfortable part isn't that the default was wrong. Defaults are compromises; that's what makes them defaults, and a platform that shipped RequestResponse auditing everywhere by default would just be trading one kind of pain (missing evidence) for another (audit infrastructure nobody budgeted for). The uncomfortable part is how long I'd been treating "the control blocked it" as a complete sentence, when it's actually the beginning of a question: blocked it, and told you what, exactly?
Prevention and detection get bundled together in how we talk about security ("we have controls in place"), as if a control that stops something automatically produces the evidence a control that watches something would. It doesn't. They're different instruments, built to answer different questions, and a security posture built entirely out of the first kind is a posture that can be breached in ways you'll never be able to fully describe, even after you've stopped the breach. Worse, a posture like that produces a false sense of completeness precisely because the blocking part is working: nothing is failing, no alerts are firing, the dashboard is green, and green dashboards are exactly what stops people from asking whether the thing behind the green is actually telling them anything.
You might disagree
The obvious pushback here is that prevention is simply more valuable than evidence in the vast majority of real incidents, and demanding rich audit detail for every control is how you end up drowning in log volume nobody ever reads, chasing a completeness that has real infrastructure and attention costs of its own. If a control genuinely prevents the bad outcome every time, does it matter that the accompanying record is thin? Nobody got hurt. I take that seriously, because it's mostly right: I'm not arguing for maximal verbosity on every control, and I said as much above when the fix turned out to be scoped, not blanket. But "prevention worked every time" is a claim you can only make with confidence if you also have the evidence to confirm it, and a control that can't distinguish twelve hostNetwork attempts from twelve unrelated validation failures can't actually support that claim. It can only support a much weaker one: nothing got through. It has nothing to say about whether the twelve attempts were the same actor escalating, a script iterating through payloads, or twelve unrelated developers making twelve unrelated typos. Those are extremely different situations that this control, as configured, could not tell apart. Prevention without evidence isn't wrong to prioritize. It's just a smaller achievement than it feels like at the moment the gate holds.
What I think now
I still think prevention matters more than detection, if I have to rank them: stopping the bad thing is better than a good record of the bad thing happening. But I no longer think of a prevention control as "done" when it prevents. I ask a second question now, for every control I deploy: if this fires ten times tonight, what will I actually be able to reconstruct tomorrow? If the answer is "not much," that's not a detail to fix later. It's a second control I haven't built yet, disguised as a finished one.
In practice that question has changed how I roll out any new admission policy or firewall rule: before I ship the control, I write down, in one sentence, what an analyst investigating a real incident would need from its logs, and I check the default logging configuration against that sentence before assuming it's covered. Most of the time it isn't, not because the platform is badly designed, but because the platform's defaults were tuned for cost and noise reduction across the general case, not for the specific evidentiary question I'd have in a real incident involving this specific control.
It's also changed which controls I bother auditing at all. Not every gate deserves the RequestResponse-level treatment; a rate limiter dropping malformed requests from bots doesn't need forensic-grade evidence, because there's rarely a follow-up investigation where the detail would matter. But a small number of controls in any system are the ones that would actually get investigated if they fired for real: the ones guarding privilege escalation, data exfiltration paths, or anything a postmortem would need to reconstruct precisely. Sorting controls into "needs evidence" and "prevention alone is enough" turned out to be a more useful exercise than trying to raise verbosity everywhere, and it's the exercise I now do explicitly, rather than assuming a platform's defaults already did it for me.
The lesson learned
A control that prevents an action does not necessarily provide evidence that someone attempted it. Treating "it was blocked" as equivalent to "we understand what happened" is the assumption this whole series exists to keep testing, starting with the next one, where the same idea shows up at the scale of an entire architecture, not just one policy.