Notes on inference architecture: the trade-offs don't transfer

Companion to How I would design an LLM inference API in a system design interview, which works the capacity math. This one is the architecture and the trade-offs.

Notes to myself from The Inference Engineering Masterclass — Philip Kiely and Ali Taha of Baseten, on Latent Space. Quotes come from the published machine transcript, so I’ve bracketed the obvious transcription slips. One episode is the whole evidentiary base here; where a claim is theirs alone and I can’t check it, I say so.

The through-line: almost nothing in this stack is good in the abstract. Nearly every technique is good or bad relative to a bandwidth number, an interconnect, a kernel launcher, or a traffic shape.

Parallelism: read the axis off the interconnect

Three ways to split a model. Each buys one thing and taxes one thing, and the tax is what picks the axis.

TP is the interconnect-hungry one. Kiely: “Tensor parallelism requires that you are able to do this like all gather, all reduce. So you shard the model across the GPUs entirely. And then for each step, you’re combining the results of each of the GPUs, which is why the interconnect matters a lot.” The payoff: “TP is helpful for latency.”

EP is the mixture-of-experts move, and the reason MoE serving economics differ from dense. Kiely: “you put the entire expert on a GPU… And then by moving the generation from expert to expert, with each expert being inside a GPU, they’re not competing for resources. You massively increase the throughput.” Throughput scaling that does not lean on the fabric.

PP is the only one chosen under duress. Kiely: “The only reason you would have to do pipeline parallelism… is if you are forced to do multi-node inference, because a model is bigger than you have the… capacity. Because the interconnect is so slow between the nodes, the only viable way to parallelize there is pipeline, but then you would do expert and tensor within each node.”

That last clause is the part worth keeping. The topology is nested, and it is a direct readout of the bandwidth hierarchy: pipeline across nodes because the fabric between them is slow, tensor and expert within a node because NVLink is not. A B200 carries 1.8 TB/s of NVLink per GPU; nothing between racks is close. You don’t really choose a parallelism strategy. You transcribe it from the interconnect.

TurboQuant: the same optimization inverts sign

The cleanest example of nothing-transfers. TurboQuant is a quantization technique that trades extra compute for less memory traffic. Taha’s account: it “made such huge hype,” was “implemented on local devices because your memory bandwidth is so slow on like a MacBook, for instance.” Then: “try putting the same thing on like an NVIDIA GPU on a B200… NVIDIA made it clear that this is not a good optimization.”

The mechanism is a straight comparison of two costs. Dequantizing costs kernel time; the compressed format saves bytes moved. On a laptop, unified memory bandwidth is one to two orders of magnitude below a data-center part, so bytes saved dominates and the technique wins. On a B200 at 8 TB/s per GPU, the bytes were nearly free to begin with and the dequant kernel is pure overhead. Same technique, same math, opposite verdict.

Kiely frames the two worlds as different questions entirely: “With local AI, it’s how do I fit this model onto my hardware and then make it less dumb? And with data center inf[erence], it’s how do I load this model and then make it less slow?”

Fit versus speed. Once I had that, several of the local-inference tricks I’d been mentally filing as “optimizations” reclassified as “workarounds for not having bandwidth.”

Quantization: more can be better

This one genuinely reset an intuition. I had quantization filed as a monotone dial — quantize more layers, get more throughput, lose more quality. It isn’t.

Taha: “if I have a model that I quantize layers one, five, and 10, and another model where I only quantize layers one and [five]… It is possible that the model in which I quantized more information is going to perform better because the quantization errors have canceled out.”

The commercial consequence: “you end up with a model that’s 20% more quantized than another provider, so you get 20% more throughput of it because there’s more layers than running an NVFP4, and your quality is better than that other quant because the layers that you chose to quantize have their errors cancel out.”

More quantized and better quality. Which means quantization is not a dial at all — it is a layer-selection search problem, where the objective is finding pairs whose errors offset. That reframes it from a compression decision to an optimization problem with a real search space, and it explains why two providers serving “the same model at FP4” can differ in both speed and quality.

Worth flagging honestly: this is one team’s published research direction, not a settled result I can check against a second source.

Mega-kernels: fusion loses to modularity

The intuitive win is fusing an entire forward pass into one kernel — no launch overhead, no intermediate round-trips to HBM. It does not survive contact with production.

Taha: “even the companies that have worked or people that I’ve spoken to who work at companies that do fused mega kernels, they very often don’t end up running those in production because the TensorRT-LLM and modular kernels that launch are faster.”

That is secondhand, from conversations, not a benchmark anyone published. I’m repeating it because the mechanism is checkable even where the survey isn’t. A mega-kernel is one artifact you tune globally; modular kernels are many artifacts each tuned locally, and per-component optimization compounds where global fusion forecloses it. The fused version also has to be re-derived for every architecture and every GPU generation, while the modular one inherits vendor improvements for free.

A pattern I recognize from query engines: the hand-written fused operator that beats the planner on one shape and loses across the workload.

GPUs are becoming ASICs, so control moves up the stack

Taha on where the hardware is going: “the GPU is moving more towards being an ASIC, where… you’re just trying to orchestrate what happens on the GPU, but you’re not controlling it thread by thread level.” And on what that silicon is for: “it has the systolic arrays and tensor cores and TMAs and tensor memory, and it has these things that are almost exclusively useful for loading model weights.”

Two consequences worth sitting with:

The leverage moves up. If you cannot control threads, the wins come from KV-cache placement, routing, and disaggregation rather than from kernel authorship. Inference engineering converges on traditional infrastructure engineering.
The silicon is fitted to today’s model shapes. Structures built around current head dimensions and current weight-loading patterns are a bet that models keep this shape. If architectures move, that specialization ages badly — and unlike a kernel, you cannot recompile a die.

Serving a model can mean modifying it

Day zero on a new open-weights release is not “point the router at it.” Kiely: “You have to redo the quantization work… And then we also have to train the speculator… there’s of course just the process of standing up all the infrastructure behind it, loading all this stuff, testing it.”

Both the layer-selection search and the draft model are per-model work, redone from scratch each time. Taha goes further: “we find it better to like, okay, we’re gonna replace this, we’re gonna replace this layer with a layer from another model.”

Swapping a layer out for one from a different model was not on my list of things an inference provider does. It puts the provider inside the architecture rather than downstream of it, which makes “we serve model X” a looser claim than I had assumed.

The bugs live in the engine, not the model

The most useful part of the episode, and the part I’d have gotten wrong. When output goes bad, the model is usually not the culprit.

Token loops, handled by brute force. Taha: “if a model was to output the same token like four plus times, we just cut the generation. We say like, ‘Oh, sorry, this— Like try again.’” A guardrail at the serving layer, not a fix.
The bug is engine-specific. Same weights, different runtime, different behavior: “it seems to be like an extremely like deterministic software issue… But if you were to switch to vLLM, that isn’t the case.” So TensorRT-LLM, vLLM, and SGLang are not interchangeable backends. They are distinct failure surfaces.
Down at the barrier. Taha: “you’ll have certain threads access data points from registers before they’ve been written to by other threads… because like your barrier is wrong or your synchronization was wrong.” Classic concurrency bugs, in a layer most people serving models never read.
And the hardware itself varies. Reported earlier in the episode: identical software on different clusters diverging, because “the KV cache transfer from a node to node in that one cluster is using a slower interconnect than the node to node in another cluster.”

Put together: a nondeterministic quality regression could be your quantization, your engine version, your kernel, or which rack you landed on. That is a wide differential diagnosis, and it argues for pinning the engine image and recording cluster identity as a variable rather than an implementation detail.

Key takeaways

Find the hardware assumption buried in the technique. Every optimization here encodes a bet about which resource is scarce. TurboQuant’s bet is “bandwidth is the bottleneck” — true on a laptop, false on a B200.
Parallelism is transcribed, not chosen. The nesting — pipeline across nodes, tensor and expert within them — is a readout of the bandwidth hierarchy.
Quantization has a search space. If layer errors can cancel, “how much” is the wrong question and “which layers” is the right one.
Local optimality can beat global. Fusion trades away the per-component tuning that compounds, and the ability to inherit someone else’s improvements.
Debug downward before you debug the model. Engine version, kernel, and interconnect all produce bad output that presents as a model problem.

One practical consequence I did not expect going in: it makes published inference benchmarks close to unreadable. A throughput number without the hardware, the engine build, and the traffic shape attached is not a result, it is an anecdote.