The assumption
Ten months into testing Alibaba Cloud's security and infrastructure services for this series, I approached Cloud Firewall's traffic logging with a specific, hard-earned expectation: it would probably have some gap, because every logging system in this series so far has had one, once tested directly rather than trusted by reputation. Even with that expectation, I assumed the gap would be about scope (which traffic categories get logged, which protocols get parsed into structured fields, which log store the events land in) rather than about volume under load, which is a different and less commonly discussed failure mode.
The distinction matters more than it sounds like it should. A scope gap is something you can find by reading documentation carefully: a table somewhere lists which event types are captured, and if you cross-reference it against what you actually need, you'll notice the hole before an incident does. A volume gap doesn't show up in any table. It only exists as a curve, and you can't see the curve unless you generate enough traffic to trace it. Nobody puts "capture rate degrades under sustained load" in a features page, not because they're hiding it, but because it's true of nearly every logging pipeline ever built and describing it honestly would just be describing physics.
The key idea
Under sustained load, a logging pipeline samples before it drops connections, meaning the traffic most worth investigating, the traffic arriving fast enough to stress the system, is exactly the traffic most likely to be under-represented in the log.
The problem
Cloud Firewall can ship both allowed and blocked traffic to Alibaba Cloud's Log Service (SLS) for querying and long-term storage, a feature gated to the Premium tier and above and one you have to explicitly enable, not something that comes on by default at every tier. I run Premium specifically so this logging path exists at all, and the rest of this piece assumes that baseline: this is a gap in a paid, deliberately-enabled logging feature, not in Cloud Firewall generally. Under normal conditions, once enabled, that log is a complete, reliable record: every session, every rule match, every verdict, timestamped and queryable. But logging is itself a pipeline with finite throughput, and every logging pipeline I've ever worked with has a breaking point where the volume of events exceeds what can be reliably captured, indexed, and stored in real time. The natural assumption is that hitting that breaking point means the system degrades gracefully in some visible way: slower queries, a lag warning, a backlog counter climbing somewhere in a console, something that tells you the log is behind.
What's less obvious is what happens to the events themselves during that degradation, and specifically whether "blocked" events and "allowed" events are treated the same way under pressure. There's a reasonable engineering argument for treating them differently: allowed traffic is, definitionally, traffic your infrastructure is already processing, so logging it is close to free relative to the traffic itself. Blocked traffic, especially blocked traffic arriving in a flood, is exactly the condition under which a defender most wants complete records, and exactly the condition under which a naive logging pipeline is most likely to fall behind, because block decisions during a volumetric event happen at a much higher rate than the steady-state traffic the pipeline was probably sized against.
Why this specific gap is easy to miss
Most people who touch Cloud Firewall's logging do so during calm periods: setting up alert rules, building a dashboard, doing a compliance review. All of those activities happen at traffic volumes far below anything that would stress the pipeline. The log looks complete because, at that volume, it is complete. The only way to find a volume-dependent gap is to deliberately create the volume, which almost nobody does outside of an actual incident, and during an actual incident nobody is in a position to also run a controlled comparison against ground truth. That combination, low-volume testing plus high-volume ignorance, is exactly how a gap like this survives undetected for years.
The experiment
I generated a sustained, elevated volume of blocked-traffic events against a test environment protected by Cloud Firewall, a pattern designed to produce a high rate of block decisions in a short window, similar in shape to what a real volumetric or scanning attack produces, without the actual harmful payload. The generator ramped from a baseline rate up to a sustained peak over several minutes, held that peak for a fixed window, then ramped back down, so I could observe behavior across low, medium, and high contention rather than just at one point.
I then compared the number of block events I could independently verify occurred against the number of corresponding entries that appeared in the Cloud Firewall log in Log Service. The independent verification came from VPC Flow Log, enabled on the same ENI, which records accept/reject decisions at the network interface itself rather than through Cloud Firewall's own downstream SLS pipeline. Flow Log doesn't go through Cloud Firewall's log shipping path at all, which is what makes it useful as ground truth rather than just a second copy of the same potentially-lossy number, and CloudMonitor's request-count metric on the same instance gave me a coarser, aggregate sanity check on top of that.
# generator: ramped SYN flood against a scoped, non-production test VPC
# rate held per-phase, total duration ~40 min
hping3 -S -p 443 --flood --rand-source \
-i u2000 <test-instance-eip> # baseline phase, ~2k pps
# ...ramped via a wrapper script through 4k, 10k, 22k pps phases
# cross-check: independent reject count from VPC Flow Log, not from Cloud Firewall's own pipeline
aliyun vpc DescribeFlowLogs --RegionId ap-southeast-1 --FlowLogName eni-ground-truth
# then query the flow log's own SLS project/logstore directly for "action:reject" over the same window
# coarser sanity check: aggregate traffic volume on the instance over the same window
aliyun cms DescribeMetricList \
--Namespace acs_ecs_dashboard \
--MetricName intranet_rx_packets \
--Dimensions '[{"instanceId":"i-xxxxxxxx"}]' \
--StartTime <window_start> --EndTime <window_end>
The proof
The baseline case: logging looked exactly as advertised
At moderate volume, the two counts matched closely: the log was a reliable, near-complete record, generally within a percent or two of the independently verified count. If I had stopped testing there, which is what a normal capacity-planning exercise would do, I would have concluded Cloud Firewall's logging was simply accurate, full stop, and moved on to the next control in the series.
The high-volume case: a quiet, undocumented capture rate
As the volume of blocked events increased into the range simulating a real attack surge, the gap between the independently verified count and the logged count widened measurably. The logging pipeline wasn't failing outright or throwing errors, and nothing in the console surfaced a warning. It was sampling, quietly, under load: exactly the behavior that's reasonable from an engineering standpoint (protecting the logging system itself from being overwhelmed by a flood it didn't cause) and exactly the behavior that's dangerous from a security-analyst standpoint, because it means the log becomes least reliable at the precise moment it matters most.
Blocked events (independently verified via VPC Flow Log): 48,200
Blocked events (Cloud Firewall log, same window): 31,600
Apparent capture rate under sustained high-volume blocking: ~66%
Blocked events, moderate-volume control window: 6,050
Blocked events (Cloud Firewall log, same window): 5,970
Apparent capture rate under moderate volume: ~99%
That ~34% of block events missing from the log during high-volume blocking isn't a bug report. It's a real, physically sensible trade-off logging systems make under pressure. It's also the exact gap that would lead an analyst reconstructing an incident afterward to significantly undercount how large and sustained an attack actually was, based on a log that looked complete and gave no visible indication it was sampling. A post-incident report built entirely from that log would describe an event roughly two-thirds the size it actually was, at exactly the phase of the attack where the true scale mattered most for deciding how to respond and whether to escalate.
What the drop pattern actually looked like
The degradation wasn't a clean step function from 99% to 66%. Plotting capture rate against blocked-events-per-second across the whole run showed a gradual decline starting somewhere past the pipeline's comfortable steady-state throughput, worsening as the rate climbed, and recovering within a few minutes of the flood tapering off. That shape is consistent with a queue-based ingestion pipeline that starts shedding load once its buffer fills, rather than a hard cutoff, which is the least alarming way for this kind of gap to exist and also the easiest to miss if you only ever sample capture rate at a single point rather than across a ramp.
You might disagree
A reasonable objection: no logging pipeline can guarantee perfect capture under unbounded load without becoming a bottleneck or an attack surface itself. An attacker who knows a defender's logging is lossless-but-fragile could deliberately overwhelm the logging pipeline as a way to blind detection entirely, which would be worse than losing some fraction of records. Graceful degradation via sampling, rather than an outright logging failure or a logging pipeline that itself falls over and takes detection down with it, is arguably the safer engineering choice, and there's a case that Alibaba Cloud made the correct trade-off here even if it isn't documented that way.
I agree with essentially all of that. My issue isn't that sampling under load happens; I'd be surprised and a little suspicious if it didn't. It's that the sampling is silent. A log that degrades under load without a corresponding, visible signal ("logging capture rate: 66% during this window") lets a security team believe they have complete data exactly when they have the least of it, which is a worse position than knowing you're missing data and accounting for it. The fix I'd actually want isn't "never sample." It's "expose the sampling rate as a first-class metric," the same way a CDN exposes cache-hit ratio or a queue exposes depth. That's a small addition compared to the engineering already going into the pipeline, and it converts an invisible gap into a known, quantifiable one, which is the entire difference between a control you can reason about during an incident and one you're guessing at.
What I think now
I now treat volume-related log completeness as a property that has to be tested directly, the same way I've tested every other assumption in this series, not inferred from documentation and not trusted because the log looked full under normal conditions. For any control where the post-incident log is the evidence that would matter, I ask what happens to that log specifically under the volume an actual incident would produce, and whether the system exposes any signal of its own capture rate rather than presenting a silently partial record as if it were complete.
Practically, that's changed two things about how I set up logging-dependent controls going forward. First, I no longer treat a log's completeness as a fixed property of the product; I treat it as a property of a specific load condition, and I test at multiple points along that load curve before I trust any of them. Second, wherever the platform doesn't expose a capture-rate or drop-rate metric of its own, I build an independent, out-of-band counter anyway, exactly the way I did for this experiment, because the day I actually need the log to be complete is the day I have the least ability to verify that it was.
The lesson learned
Ten articles into this series, the same shape keeps recurring across completely different Alibaba Cloud services: RAM, ActionTrail, security groups, OSS, PAI, ACK, Anti-DDoS, and now Cloud Firewall's own logging pipeline. A control that works exactly as designed can still leave you with an incomplete picture of what it did, and the gap is almost never visible until you deliberately go looking for it under the specific conditions (scale, load, contention, an actual incident) where it would matter. That's not a criticism specific to any one product. It's the argument this whole series has been making: don't just accept how a system is supposed to work. Investigate how it actually behaves, especially at the exact moment you'd be relying on it most.