Observability Isn't the Same Thing as Logging

The assumption

"We have good observability" is a sentence I used to say about systems that had, more accurately, good logging. Structured logs, shipped to a searchable store, with a dashboard on top. It felt complete. It had all the visual signals of maturity: retention policies, alerting rules, a runbook that said "check the logs."

The problem

Logging is a system telling you what it already decided was worth saying. Someone, at some point, wrote a log.info() call and made a judgment about what would matter later. That judgment is necessarily incomplete; it's made in advance of the incident it might one day help explain, by someone who couldn't know exactly what that incident would look like.

Observability, as distinct from logging, is what you have when the system's outputs are rich and structured enough that you can compose a new question out of existing data, rather than being limited to the questions the original log lines anticipated. The distinction sounds academic until you're in an incident and the question you actually need answered ("which upstream dependency's p99 correlates with this error spike, broken down by customer tier") has no corresponding log line, anywhere, because nobody wrote one for that exact combination in advance.

Why logging feels like enough until it isn't

Part of what makes this substitution so easy is that logging and observability agree completely on the easy cases. "Did the deploy fail," "is the error rate elevated," "which pod restarted" are all answerable from a log line someone thought to write, because they're exactly the kind of question people anticipate when they instrument a system. The gap only opens on the hard cases: the ones that involve a combination of dimensions nobody predicted together, cache tier crossed with customer tier crossed with a specific upstream version, for instance. Those combinations are rare enough that a team can go a long time, sometimes years, without hitting one, which is exactly long enough to build genuine confidence in a logging setup that has never actually been tested against the kind of question it will eventually need to answer.

There's also a metrics-shaped version of the same mistake. A team with rich dashboards and pre-aggregated metrics can feel even more confident than a team with raw logs, because the dashboards look purpose-built. But a metric is a decision made in advance about which dimensions to keep and which to throw away at aggregation time. If "cache tier" wasn't one of the labels attached to that histogram when it was defined, the dashboard cannot answer a question that needs it, no matter how many other dimensions it tracks beautifully. Pre-aggregation is a form of anticipatory judgment with the same blind spot as a hand-written log line, just wearing a more polished interface.

The experiment

I tested this more directly than the phrasing usually gets tested: on a service with what most teams would call solid logging (structured JSON, correlation IDs, reasonable retention), I picked five questions that came from real past incidents at various jobs, phrased exactly as an on-call engineer would ask them under pressure, and tried to answer each using only the existing logs, with a 10-minute budget per question, the rough length of patience an incident actually affords.

The five questions, in the form they'd actually get asked on a call:

1. "Is this error spike isolated to one upstream dependency, or is it everywhere?"
2. "Did this start right after the last deploy, or was it already trending before?"
3. "Which upstream's p99 correlates with this error spike, broken down by customer tier?"
4. "Is this the same shape of failure we saw during the incident three months ago?"
5. "Which cache tier served the requests that failed?"

To keep the test honest, I didn't allow myself to write new instrumentation mid-test. The rule was: only the data already being collected, queried within the existing tooling, in ten minutes. That constraint matters, because "we could answer this if we added a field" is a true statement about almost any question given enough lead time. The interesting number is what's answerable right now, from what's already there, because that's the actual condition an on-call engineer is in during a live incident.

Question 1: answered in 3 minutes (log-based error rate split by upstream tag, already indexed)
Question 2: answered in 6 minutes (had to manually cross-reference deploy timestamps
            against a separate log stream; no join key existed, so this was done by eye)
Question 3: not answered in 10 minutes (customer tier was logged; upstream p99 was
            in a separate metrics system with no shared trace ID to join on)
Question 4: answered in 4 minutes (past incident's log signature was documented in
            a runbook, made comparison fast; this is really a documentation win, not
            an instrumentation one)
Question 5: not answerable at all (cache tier was never logged as a field)

Two of the five were answerable directly. Two required a slow, manual join across multiple log streams that a dashboard couldn't do and that took most of the 10 minutes just to set up. One wasn't answerable at all: the necessary dimension (which cache tier served the request) had never been logged, because nobody had needed it until exactly this question existed.

The proof

Three out of five is a real number, and it's a worse number than "we have good logging" implies. The gap wasn't volume: this service logged plenty. The gap was dimensionality: the logs recorded events, but not enough of the structured context (cache tier, shard, feature flag state, upstream version) to let a new question be composed after the fact. That's precisely the difference between logging and observability: one is a record of anticipated facts, the other is a substrate flexible enough to answer unanticipated ones.

The join problem specifically

Question 3 is worth dwelling on separately, because it wasn't a missing-data problem the way question 5 was. The customer tier was logged. The upstream latency was tracked. Both pieces of information existed somewhere in the stack. What didn't exist was a shared key that let them be joined after the fact: no trace ID connected the request log line to the metrics sample for the same request, which meant the two data sources were each individually complete and jointly useless for this question. This is a distinct failure mode from missing dimensions, and it's arguably the more common one in systems that have grown incrementally, where logging and metrics were adopted at different times, by different people, with different tools, and nobody was responsible for making sure the two could be correlated later. A trace ID propagated consistently through both would have turned a 10-minute manual cross-reference into a single query.

The documentation adjacency

Question 4 also deserves a caveat, because on reflection it wasn't really testing observability at all. It was fast because a human had written down, in a runbook, what the earlier incident's log signature looked like. That's valuable, and it's worth doing deliberately (a short "if you see this pattern again, here's what it was" note attached to every closed incident pays for itself), but it's closer to institutional memory than to the system's own answerability. A new team member without access to that runbook would have had a much harder time with question 4, which suggests the true "unanticipated question" success rate on this service, absent the shortcut of a well-kept runbook, was closer to two out of five than three.

You might disagree

A reasonable objection: you can't instrument for every possible future question, and trying to is how teams end up with unbounded cardinality, exploding costs, and dashboards nobody can navigate. That's true, and it's the real engineering trade-off underneath this whole piece: dimensionality has a cost, and infinite dimensionality is not the goal. But "we can't log everything" is not the same claim as "our current dimensions were chosen well." Most teams have never actually tested whether their existing dimensions cover the questions that matter; they've just accumulated whatever was easy to add at the time each log line was written. The fix isn't more data. It's periodically testing your system's answerability against real, recently-asked questions (the exercise I ran above) and adding dimensions deliberately, where the gaps actually are.

It's also worth being honest about the cost side of that objection rather than dismissing it. Adding a high-cardinality field like customer ID or cache key to every log line genuinely does increase storage and query cost, sometimes substantially, and a team that adds every dimension anyone can imagine wanting will eventually drown in exactly the way the objection warns about. The discipline this piece argues for isn't "log more." It's "log the dimensions a real, recently-asked question actually needed and didn't have," which is a much smaller and more targeted set than "log everything imaginable," and it has the advantage of being backed by an actual incident rather than a hypothetical one.

What I think now

I now treat "logging" and "observability" as a maturity gradient rather than synonyms, and I test systems the way I tested this one (with real questions from real past incidents, under a real time limit), rather than trusting log volume or dashboard count as a proxy. A system with fewer, better-chosen dimensions consistently outperforms a system with more logs and less structure.

I've also started treating the five-question exercise itself as something worth repeating on a schedule rather than running once. A service's observability posture degrades quietly as new failure modes appear that the original dimension set didn't anticipate, the same way any of the other layers in a hardening program drift without a periodic recheck. The right cadence is roughly: after every incident that took longer than it should have to diagnose, ask whether the missing piece was a missing dimension or a missing join key, and fix that specific gap rather than the observability stack in general. Fixing "our observability" as an abstract goal produces sprawling projects that never finish. Fixing "we couldn't join request logs to upstream latency by trace ID" is a concrete, scoped, one-sprint change.

What actually shipped afterward

The concrete outcome of running this exercise once was small and boring, which is roughly what I'd now expect from a well-targeted fix: a trace ID got propagated from the request logging middleware into the metrics exporter, and cache tier got added as a label on the request log line. Two changes, neither expensive, both aimed at a specific question that had already come up rather than a speculative future one. Re-running the same five questions afterward, all five were answerable inside the ten-minute budget, with question 3's join taking under a minute instead of failing outright. That's the shape I'd expect a healthy observability practice to take over time: not a single large investment, but a series of small, evidence-driven additions, each traceable to a specific question that once took too long or couldn't be answered at all.

The lesson learned

Logs tell you what a system said about itself in advance. Observability is what lets you ask something nobody thought to log for, and still get an answer. The way to find out which one you have isn't to count your log lines: it's to take a real question from your last incident and time how long it takes to answer it from what you've already collected.