The Castle Was Never the Problem

The assumption

The castle-and-moat model is probably the most durable metaphor in security. Harden the perimeter, control who gets in, trust whoever's already inside. It's intuitive, it's old, and for a long time it mapped reasonably well onto how infrastructure actually looked: a data center with a defined edge, a small number of ways in, and a network you controlled from end to end.

I don't think the metaphor was ever fully accurate. But it was close enough for long enough that a lot of us, myself included, absorbed it as a mental model for how security should work, even after the infrastructure it described had stopped existing.

The key idea

A perimeter that holds tells you where an attacker didn't get in. It tells you nothing about where they're still trying, or what's already inside.

The problem

Modern infrastructure doesn't have a castle. It has a Kubernetes cluster with dozens of namespaces, a service mesh, workloads that talk to each other constantly, third-party APIs, CI pipelines with their own credentials, and a perimeter that is, at best, a fuzzy administrative boundary rather than a network one. There is no single wall. There are hundreds of small, mostly-unexamined trust decisions, made by default, most of them saying yes.

The dangerous part of the castle metaphor isn't the wall. Walls are fine: a well-configured perimeter stops a huge amount of noise before it becomes your problem. The dangerous part is what the metaphor teaches you to stop asking once the wall holds. In a castle, if the gate didn't open, the threat is outside, and outside is not your concern until it's at the gate again. Applied to infrastructure, that becomes: if the request was denied, the incident is over.

It usually isn't. It's just the part you can see.

Why the metaphor survives past its usefulness

Part of why castle-and-moat outlives its accuracy is that it still describes something real: most attack traffic genuinely is stopped at a perimeter, most of the time, and a security team that removed its perimeter controls entirely would see an immediate, measurable increase in noise reaching internal systems. The metaphor isn't a lie about the wall's value. It's a lie about the wall's completeness. It quietly implies that "inside" is one place, uniformly trusted, when in a Kubernetes cluster "inside" is actually dozens of namespaces, hundreds of service accounts, and a mesh of pod-to-pod trust decisions that have nothing to do with the network edge at all. A metaphor built around a single boundary has no vocabulary for a system with dozens of internal boundaries, so it just doesn't ask about them, and the absence of a question is much harder to notice than a wrong answer.

There's a career-stage version of this too, worth naming honestly. Early in a security career, the perimeter is the thing you're handed responsibility for first: firewall rules, ingress rules, WAF configuration. It's also the layer with the most mature tooling and the clearest pass/fail signal. It's natural to generalize from "I have gotten good at this layer" to "this layer is the one that matters most," and the castle metaphor gives that generalization a comforting shape. The layers behind the perimeter, RBAC scope, network policy correctness, admission control, are less mature, less tool-supported, and much harder to get a clean signal on, which makes them easy to under-invest in even by security-conscious engineers who aren't being lazy, just following where the tooling made the feedback loop tightest.

The interactive part: what's actually inside the perimeter

Click through the layers below. Each one is a real Kubernetes control point (the kind that shows up in almost every production cluster), and each one makes a different, mostly implicit trust decision. The "castle wall" people usually picture is layer one. There are at least four more behind it that decide what happens next.

Interactive: click a layer

Ingress / Perimeter

What it does: Terminates external traffic, routes it to the right service, and is usually the only layer with a WAF or rate limiting in front of it.
What can go wrong: Misconfigured routing rules can expose an internal-only service externally without anyone noticing, since the perimeter itself still looks 'up' and healthy.
Security implication: This is the castle wall: the layer most teams invest in first, and the only layer most non-security engineers think about when they hear 'security control.'

Every one of those layers can fail open under some condition (a webhook timeout, a misconfigured selector, an overly broad service account), and every one of those failures looks, from outside the cluster, exactly like the castle wall holding. That's the part the metaphor hides: the wall isn't one thing. It's a chain, and the chain's strength is its weakest silent link, not its most visible one.

What "failing open" actually looks like in practice

It's worth being concrete about what a fail-open condition looks like operationally, because "fails open" can sound like an edge case rather than the default behavior of several common components. A validating admission webhook, by design, has a failurePolicy field, and the Kubernetes documentation itself notes that many cluster operators set it to Ignore rather than Fail, specifically because a webhook outage that also blocks all deployments is considered worse than a temporary lapse in policy enforcement. That's a reasonable operational trade-off in isolation. It also means that during exactly the kind of infrastructure incident when things are already going wrong (a webhook pod OOM-killed, a certificate expired, a network blip between the API server and the webhook service), the admission-time policy enforcement silently stops applying, and every request that would have been rejected is instead admitted, with no error surfaced anywhere that says "policy was skipped this time."

# a webhook configuration that fails open under exactly
# the conditions when enforcement matters most
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
webhooks:
  - name: policy.example.com
    failurePolicy: Ignore  # the quiet part: any webhook outage
                            # means every request is admitted unchecked
    timeoutSeconds: 5

Nothing about this configuration is a mistake in the conventional sense. Ignore is a legitimate, documented, frequently recommended setting. It's also a structural reason why "the policy is enforced" and "the policy was enforced during this specific window" can be two different facts, and only one of them is what a dashboard showing "webhook: healthy" is actually telling you.

What changed my mind

I went looking for a concrete case rather than arguing from the metaphor, and ran an experiment on network policy misconfiguration, deliberately introducing the kind of typo that passes code review because the YAML is syntactically correct. A NetworkPolicy whose selector matched zero pods instead of the intended service. From the outside, from kubectl describe, from the CNI's own metrics: everything looked healthy. Ingress into the namespace it was supposed to protect was, in fact, completely open.

# the policy that looked like protection
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: restrict-payments-ingress
  namespace: payments
spec:
  podSelector:
    matchLabels:
      app: payments-api   # actual pods are labeled app.kubernetes.io/name
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: gateway

kubectl apply accepted it without complaint, because there is nothing invalid about a selector that matches nothing; Kubernetes has no way to know that app: payments-api was supposed to be app.kubernetes.io/name: payments-api, matching a labeling convention the rest of the manifests had already migrated to. kubectl get networkpolicy reported it as present. kubectl describe networkpolicy reported the selector, the rule, everything exactly as written, all of it technically true, none of it doing anything, because a NetworkPolicy that matches zero pods governs zero pods' traffic, which by Kubernetes' flat-networking default means all traffic to that namespace remained unrestricted.

Nothing was denied. No gate closed. No log entry said "policy misconfigured." The castle's wall, in this case, was a picture of a wall.

Confirming it wasn't a fluke

To make sure this wasn't a one-off artifact of a single typo, I ran the same class of mistake three more ways against the same lab cluster: a selector using the wrong label value instead of the wrong key, a namespaceSelector missing where one was needed for cross-namespace traffic, and a policy applied to the wrong namespace entirely due to a copy-paste error in a Helm values file. All four produced the identical outward signature: Applied, Ready, present in kubectl get, and zero actual enforcement. None of the four generated a distinguishable event, warning, or metric that would have separated "policy enforcing as intended" from "policy syntactically present, functionally absent" without someone deliberately testing traffic against it. That consistency is the actual finding here, more than any single misconfigured YAML file: the failure mode isn't rare or exotic, it's the default behavior of a NetworkPolicy object whenever its selector doesn't match what the operator intended, and Kubernetes has no built-in mechanism that flags the mismatch.

You might disagree

The strongest version of the counterargument is this: perimeter controls still stop the overwhelming majority of real attack traffic, and spending equal attention on "what if the wall doesn't actually hold" for every control is a good way to build nothing, because you can second-guess any control indefinitely. Defense in depth is expensive, and not every system needs zero trust's assumption of universal hostility. A blog's static site host doesn't need the same posture as a payments cluster.

That's fair, and I don't think the answer is to distrust every layer equally, all the time. But the conclusion isn't "trust the wall." It's "know which of your walls you've actually tested, and which ones you're trusting because they've never been challenged." Those are very different confidence levels wearing the same badge. The network policy in my experiment had never been meaningfully attacked; it had just never been checked, and an unchecked control isn't evidence of security. It's an absence of a finding, which is a different thing wearing the same clothes.

Worth adding to that counterargument's own defense: the cost of verification here is genuinely small relative to the cost of the failure mode. Testing whether a NetworkPolicy actually enforces what it claims takes one kubectl exec into a disallowed pod and one connection attempt, run once per policy at deploy time as part of CI rather than as a standing, expensive practice. The proportional-effort argument for perimeter trust is much weaker once the verification cost is this close to free; the real reason most teams skip it isn't cost, it's that nobody thought to ask the question, which loops back to exactly what the castle metaphor trains people not to ask.

What I think now

I still build perimeters. I still think the outermost gate matters: it filters an enormous amount of noise before it reaches anything more expensive to defend. What I no longer do is treat a held perimeter as the end of the security question for a given request. The real question was never "did it get past the wall." It's "what did the system decide, silently, at every layer behind the wall, and would I actually know if one of those decisions was wrong?"

Practically, this has turned into a habit rather than a philosophy: for any control I'm responsible for, perimeter or otherwise, I now ask what its fail-open condition is before I ask what its intended behavior is. Every control has one, whether it's a webhook timeout, a selector typo, or a stale certificate, and knowing the failure's shape in advance is the only way to recognize it when the control's dashboard is telling you everything is fine.

The castle was never the problem. The problem was believing the castle was the whole building.

The lesson learned

A perimeter that holds tells you where an attacker didn't get in. It doesn't tell you what's happening at the layers behind it, and modern infrastructure has far more of those layers than the castle metaphor has room for. Zero trust isn't a rejection of perimeters. It's an admission that the interesting security questions live inside them now.