The assumption
"We deployed the model" implies, in most conversations I've had, that the thing running behind the inference endpoint is the thing that finished training. Deployment sounds like a copy operation: take the trained weights, put them somewhere that can serve requests, done. On PAI, and on most managed ML platforms, that framing skips several real steps that happen in between.
The key idea
Deployment is not a copy operation. It's a second pipeline (quantization, batching, serialization, runtime differences) with its own opportunities to change model behavior, and most teams validate only the first pipeline.
The problem
Getting a trained model onto a production inference endpoint on PAI typically involves format conversion (to a serving-optimized format), often quantization (reducing numerical precision to cut latency and cost), and always a different runtime than whatever trained the model in the first place. Each of those steps is individually well-understood and individually reasonable. What's less discussed is that each one is also a place where the model's actual outputs can drift from what validation during training measured, and the standard workflow validates accuracy once, at the end of training, on the training-time runtime, and then treats the deployed endpoint as inheriting that validation for free.
It doesn't, automatically. It inherits it only if nothing meaningful changed between the two runtimes, and quantization specifically is designed to change something meaningful: that's the entire point of using it.
The pipeline nobody diagrams
If you ask a team to draw their ML system architecture, you almost always get a training pipeline: data ingestion, feature engineering, the training loop, a validation step, a model registry. Deployment shows up as a single arrow out of that box, usually labeled "deploy" or "serve." That arrow is doing a lot of unlabeled work. On PAI-EAS specifically, the arrow expands into: export the trained model to a serving format (often ONNX or a framework-specific serialized graph), optionally apply post-training quantization (INT8 is the common choice, sometimes INT4 for larger models where cost pressure is higher), package it with a runtime container that is not the same software that ran the training loop, configure batching behavior for the inference server, and attach autoscaling rules that determine how many replicas handle a given request pattern. Every one of those five steps has its own defaults, its own version number, and its own capacity to produce numerically different outputs than the training-time graph did on the same input.
None of this is a criticism of PAI specifically. Every managed serving platform I've used, and most self-hosted ones, has the same shape. The problem is procedural, not architectural: the validation step in the diagram happens before the arrow, and nothing in the standard workflow re-runs it after.
The experiment
I trained a small classification model, validated its accuracy on a held-out test set using the training runtime, then deployed it through PAI-EAS (the platform's model-serving product) with INT8 quantization enabled, a common choice for reducing inference cost, and one PAI actively offers as a one-step configuration during deployment. I then ran the identical held-out test set against the live inference endpoint and compared predictions directly, example by example, rather than only comparing aggregate accuracy.
Setting up a fair comparison
The methodology mattered more than it looks. It's easy to run a sloppy version of this experiment that produces a reassuring, meaningless result: batch the requests differently than training-time batching, let the client library retry silently on transient errors and drop those examples from the comparison, or compare accuracy computed on different random subsamples of the test set rather than the identical ordered set against both runtimes. Any of those shortcuts can hide the exact effect I was trying to measure. So the constraints I held fixed were: the same 2,000 held-out examples, in the same order, submitted one at a time (no batching effects folding multiple examples' numerics together), against both the FP32 training-time model and the INT8 PAI-EAS endpoint, with every single response logged, including latency, so a silently dropped or retried request would show up as a gap in the log rather than a missing row in an aggregate.
The deployment configuration itself was close to what PAI recommends as a default cost-optimization path:
# PAI-EAS deployment, paraphrased for readability
# (the real service definition is JSON, with its own field names;
# this is the shape of the decision, not a literal config dump)
name: fraud-classifier-int8
model_path: oss://ml-artifacts/fraud-clf/v12/model.onnx
quantization: int8, calibrated against a held-out sample from the same distribution
instance_type: ecs.gn6i-c4g1.xlarge
instance_count: 2
runtime: onnxruntime-1.17
source_runtime: pytorch-2.1
Nothing about that file is unusual or aggressive. It's the kind of configuration a cost-conscious team would write on the first pass, calibration dataset and all, following the platform's own quantization guide.
# Comparing predictions, not just aggregate accuracy
mismatches = [
(x, y_true, y_pred_fp32, y_pred_int8)
for x, y_true, y_pred_fp32, y_pred_int8 in zip(X_test, y_test, preds_fp32, preds_int8)
if y_pred_fp32 != y_pred_int8
]
# len(mismatches) / len(X_test) ≈ 0.04
The proof
The aggregate number lied by omission
Aggregate accuracy moved by less than one percentage point: the kind of number that, seen alone in a dashboard, reads as "no meaningful change, ship it." But the example-by-example comparison showed something the aggregate number hid: roughly 4% of individual predictions flipped between the training-time and quantized-serving-time runs: cases the model previously predicted correctly now predicted incorrectly, and a smaller number that flipped the other direction, netting out to a nearly unchanged aggregate score while meaningfully different actual behavior on a real subset of inputs.
The aggregate metric isn't wrong, exactly. It's answering a narrower question than the one people think it's answering. "Is overall accuracy stable" and "does the deployed model behave the same as the validated model" sound like the same question until you have the per-example data in front of you, at which point it's obvious they aren't.
Where the flips clustered
The more useful finding, once I had the mismatch list, was that the flips weren't uniformly distributed across the confidence spectrum. They clustered almost entirely among examples the FP32 model had already scored close to its decision boundary: predictions the training-time model made with, say, 52% confidence rather than 98%. That's intuitive in hindsight (quantization introduces small numerical perturbations, and small perturbations only flip a decision when the decision was already close), but it has a direct operational consequence: a model's error surface after quantization is not a random 4% subsample of all inputs. It's concentrated on exactly the inputs the model was already least confident about, which for a fraud classifier or a moderation model is very often also the subset of inputs that matter most, because "close call" is usually where a human reviewer or a downstream escalation would have wanted the model to be right.
# Confidence distribution of the 4% that flipped (FP32 confidence, pre-quantization)
0.50–0.60: 61%
0.60–0.70: 24%
0.70–0.80: 9%
0.80–0.90: 4%
0.90–1.00: 2%
For a low-stakes classification demo, a 4% prediction flip rate that nets out to a near-identical aggregate score is a curiosity. For a model making decisions with real consequences (a fraud flag, a content moderation call, a medical triage signal), the aggregate number hiding a 4% behavior change is exactly the kind of gap that shows up in production complaints long before it shows up in a dashboard, and it shows up disproportionately on the cases that were already borderline, which are precisely the cases where a human downstream is least equipped to tell, from the outside, whether the model's new answer is a reasonable judgment call or a quantization artifact.
You might disagree
You could reasonably say this is a known, well-documented trade-off of quantization, not a hidden gotcha: anyone doing serious ML engineering knows precision reduction changes outputs at the margin, and PAI doesn't claim otherwise. The quantization documentation is explicit that INT8 is a lossy approximation, and any engineer who has taken a numerical methods course understands that reduced precision perturbs a function's output near decision boundaries. That's fair as far as it goes, and I don't think PAI, or ONNX Runtime, or any serving platform is hiding this. My point isn't that quantization is secretly dangerous; it's that the standard validation workflow (measure accuracy once, on the training runtime, before deployment) doesn't actually measure the thing that matters, which is whether the deployed model's individual predictions match what was validated. Knowing quantization changes things in the abstract is different from measuring, for your specific model and dataset, which specific inputs it changes for, and abstract knowledge doesn't get encoded into a CI check the way a concrete measurement does. I'd also push back gently on the implied conclusion that because this is "known," teams have actually accounted for it: in the accounts and pipelines I've reviewed, "known trade-off" and "measured trade-off" are treated as interchangeable far more often than they should be, and only one of those actually shows up in a deployment gate.
What I think now
I now treat "validated during training" and "validated as deployed" as two separate claims that require two separate tests, and I compare predictions example-by-example against the live endpoint, not just aggregate accuracy against a benchmark. On PAI specifically, that means running the held-out test set through the actual EAS endpoint after deployment, not trusting that the pre-deployment validation number still applies once quantization, format conversion, and a different runtime are in the loop.
This has a second-order effect on how I think about model registries. A registry entry that records "accuracy: 94.2%, validated 2026-01-30" is recording a fact about the training-time artifact. It says nothing about the serving artifact that a consumer of the registry will actually call. I've started pushing for registries to carry a second, distinct field: post-deployment parity, measured against the same held-out set, re-run every time the serving configuration changes (a new quantization scheme, a runtime version bump, a different instance type). That field is more work to maintain than a single accuracy number, and it will occasionally show drift that requires a real decision about whether the deployed behavior is still acceptable. That's the point. The alternative is finding out about the drift from a downstream complaint instead of a dashboard.
It's also changed how I read vendor deployment guides. Any one-click "optimize for serving" option, whether it's PAI's quantization toggle or an equivalent on another platform, is a request to trust that the optimization step preserves behavior. That request deserves the same skepticism as any other unverified claim in a production system: not because the vendor is being dishonest about what the toggle does, but because "what the toggle does in general" and "what the toggle did to your model, on your data" are different facts, and only the second one is the one you're actually responsible for when the endpoint is live.
The lesson learned
The model behind your inference endpoint is not automatically the model you validated. It's the output of a second pipeline (conversion, quantization, a different runtime) applied to that model, and that pipeline has its own failure modes. An aggregate accuracy number can stay flat while a meaningful share of individual predictions change underneath it, and those changes are not evenly distributed: they concentrate on the inputs the model was already least sure about, which are frequently the inputs where being right mattered most. Validate the thing that's actually serving traffic, not the thing that finished training.