The Security Control That Worked Too Well

The assumption

If you'd asked me to rank the properties of a good security control, "does it block what it's supposed to block" would have been at the top, by a wide margin. Everything else (explainability, auditability, how noisy the false-positive rate is) felt like polish. Nice to have, not the point.

The problem

The disallow-host-network policy from the admission-control experiment is, by the metric I used to rank controls, a great control. It caught every single violation across every test run. Not one bad pod spec got through. If effectiveness were the whole story, this article wouldn't exist.

But when I sat down to write an incident report using only what the policy itself produced, I had almost nothing. Kyverno's policy reports, at default settings, recorded that a resource was blocked and which rule blocked it: not the offending field, not a diff against what a compliant spec would look like, not enough for someone who wasn't already deeply familiar with the policy to understand, from the report alone, what the submitter did wrong.

That's a specific kind of failure: the control worked, and its own record of working was nearly useless to anyone who wasn't the person who wrote it.

What "blocked" actually looked like on disk

Here's the entire policy-report entry Kyverno produced, at default configuration, for a genuinely dangerous submission (a pod requesting hostNetwork: true in a namespace where that's disallowed):

apiVersion: wgpolicyk8s.io/v1alpha2
kind: PolicyReport
results:
  - policy: disallow-host-network
    rule: host-network
    resources:
      - kind: Pod
        name: worker-7c9f4d
        namespace: payments
    result: fail
    message: "validation error: rule host-network failed"
    timestamp: "2026-08-11T02:14:33Z"

Compare that to what an on-call engineer actually needs at 2 a.m.: which of potentially forty pods in a failed deploy has the problem, which exact field is set, what the field currently contains versus what it needs to contain, and ideally a suggested fix. None of that is in the record above. "validation error: rule host-network failed" is true, and it is not useful to anyone who isn't the rule's author, because it describes the rule's internal failure, not the submitter's actual mistake.

What changed my mind

I started imagining the on-call engineer who isn't me: someone paged at 2 a.m. because a deploy pipeline is failing every pod submission in a namespace, with no context about why. The policy report says: blocked, rule no-host-network. It doesn't say which of forty pod specs in that deploy had the problem, what the field looked like, or how to fix it. The engineer now has two choices: page the person who wrote the policy, or start guessing. Neither is what "working" should feel like from the inside.

Effectiveness had been optimized in isolation. Nobody had asked the second question (effective and understandable to whom, under what pressure) because the first question is the one that shows up in a demo.

Reproducing the on-call scenario deliberately

To make this concrete rather than anecdotal, I ran a small test: I handed the raw policy-report YAML above, with no other context, to three engineers on the team who hadn't written the policy and hadn't been in the room when it was designed. None of them could identify, within two minutes, which field in the pod spec had caused the failure, without opening the policy source itself to read the rule logic. That's the actual cost of a report that names the rule but not the violation: it converts a self-service fix into a page to whoever understands the rule internals, every single time, regardless of how simple the underlying mistake was.

I then rewrote the policy with a structured message field that names the offending field and its value directly, and reran the same test:

# excerpt from the rewritten ClusterPolicy
validate:
  message: >-
    Pod {{request.object.metadata.name}} in namespace
    {{request.object.metadata.namespace}} sets
    spec.hostNetwork={{request.object.spec.hostNetwork}}.
    hostNetwork must be false or omitted. Remove the
    hostNetwork field from the pod spec and resubmit.
  pattern:
    spec:
      =(hostNetwork): "false"

With that message in place, the same three engineers identified the fix in under thirty seconds each, without opening the policy source. The rule's logic didn't change at all. Only its explanation did. That gap, thirty seconds versus a page to the policy author, is the entire difference between a control that's accountable and one that merely works.

The uncomfortable part

The uncomfortable part is that fixing this wasn't a matter of writing a better regex or tuning a threshold. It required deliberately adding verbosity to every policy: custom message fields that explain the violation in terms a non-author can act on, structured output that names the exact field and its offending value. And that verbosity has a real cost: policies take longer to write, longer to review, and are easier to get subtly wrong when the message text drifts from what the rule actually checks. I found this drift myself, twice, while writing the improved messages for this experiment: a message that said "must not exceed 3 replicas" for a rule whose actual pattern checked a different field entirely, left over from an earlier draft of the policy that had since been edited. A wrong message is arguably worse than no message, because it actively misleads the person trying to fix the problem instead of just leaving them uninformed.

Nobody wants to spend that time on a control that's "already working." That's the trap. The control's success at its primary job is exactly what makes its secondary failure invisible: there's no red X anywhere. Just a quietly unhelpful policy report, waiting for the night someone actually needs it.

Where this scales badly

The cost of writing a good explanatory message for one rule is small. The cost compounds once a cluster has the dozens of policies a real production security baseline requires: disallowed host paths, required resource limits, disallowed privilege escalation, required non-root users, image provenance checks, and so on. Each one needs its own field-specific, value-specific message, and each message needs to be kept in sync with its rule as the rule evolves. In the lab cluster used for this experiment, that meant maintaining explanatory text for around twenty active policies. In a real platform team's cluster, with security, compliance, and cost-governance policies layered from multiple sources (some hand-written, some pulled from a shared policy library, some generated by a scanning tool), keeping messages accurate against rules that nobody on the current team wrote becomes its own maintenance burden, one that competes directly with the maintenance burden of the rules themselves. I don't have a clean answer for that scaling problem beyond treating message accuracy as a first-class review item, the same way a code reviewer checks that a comment still matches the code it describes.

There's also a versioning problem that doesn't show up until a policy library gets updated out from under you. A shared library that ships disallow-host-network alongside forty other rules will, sooner or later, tighten a pattern or rename a field, and if the message text was written against the old pattern, an upgrade can silently produce a control that still blocks correctly but explains incorrectly: worse than no message, because it actively points the on-call engineer at the wrong field. I now pin policy library versions the same way I pin any other dependency, and I diff the rule bodies against their messages on every version bump, specifically looking for that mismatch rather than trusting the changelog to mention it.

What I think now

I've started treating "can someone who didn't write this policy understand why it fired, without asking me" as a release criterion, not a nice-to-have. If a policy passes every test but fails that question, it's not done. It's effective and unaccountable, which is a specific, nameable kind of unfinished.

Practically, that means a few concrete habits I didn't have before this experiment. Every new admission policy gets a message field written and tested against someone who didn't write the rule, the same way the rule logic itself gets tested against a violating and a compliant spec. Policy reviews now check the message text against the pattern it describes, not just the pattern against the intended behavior, because a message that drifts from its rule is a liability, not neutral. And when I evaluate a third-party policy library rather than writing rules from scratch, "what does the failure report actually say" is now a question I check before adoption, not something I discover during the first real incident.

I've also started weighting this into how I read a vendor's admission-control pitch. A demo that shows a dangerous pod spec being rejected is showing you the easy half of the product. The harder half, whether the rejection comes with enough context for someone under pressure to fix it without paging the vendor's support line or the internal policy author, almost never makes it into the demo, precisely because it's the half that's less visually satisfying and more work to build well. I now ask for that half explicitly, with a sample report from a real violation, before treating "blocks everything" as the whole evaluation.

The lesson learned

Effectiveness and accountability are different properties, and a control can have one without the other. The dangerous version isn't the control that fails quietly. It's the control that succeeds constantly while staying silent about why, because success is exactly what makes that silence easy to overlook. A policy report that says only "blocked" is a control halfway built, even when the blocking half works perfectly, and the honest way to test for that is to hand your own report to someone who didn't write the rule and see how long it takes them to understand it.