How I would design an LLM inference API in a system design interview

An inference API is a scheduler for scarce memory bandwidth, and the objective function is goodput under a latency SLO — not tokens per second. If you state that in the first two minutes, the rest of the interview is arithmetic. Draw the load balancer and the GPU pool first and you can get thirty minutes in before noticing you never worked out how many requests fit in one GPU, which is the number every other decision depends on.

Latent Space’s The Inference Engineering Masterclass with Baseten’s Philip Kiely and Ali Taha is where several of the production details below come from. What follows is how I would run the 45 minutes, with the numbers I would work on the whiteboard. The architecture and trade-offs from that episode are in a companion note, the trade-offs don’t transfer.

The time budget

The unusual allocation is minutes 8–18. Ten minutes of arithmetic feels like a long time to spend not drawing. It is the part that decides whether the architecture you draw afterward is defensible.

Clarify the traffic shape, not the requirements

Ask four questions, because each one changes the design and nothing else does.

1. What is the prompt-to-output ratio? A 200K-token prompt with a 200-token answer is a prefill system. A 500-token prompt with a 4K answer is a decode system. These have almost nothing in common.
2. How much prefix is shared? If every request carries the same 10K-token system prompt, cache reuse is the highest-leverage thing in the design. If every prompt is unique, it is worth nothing.
3. Streaming or batch? Streaming makes inter-token latency a user-visible SLO. Batch makes it irrelevant.
4. One model or many? Multi-model changes the whole capacity story, because weights are the fixed cost and you cannot amortize them across models on the same GPU.

Do not ask about DAU or QPS first. Request count is a weak signal here; token count and prefix overlap are the strong ones.

Goodput is the objective function

Name the three metrics and pick the one you are optimizing.

• TTFT — time to first token, set by prefill.
• TPOT / ITL — time per output token, set by decode. This is what a user perceives as speed.
• Throughput — total tokens per second across the fleet, which is what the accountant perceives.

The trap is that throughput and TPOT trade off directly. You can raise throughput by batching harder until every individual stream crawls. So the number to maximize is goodput: what the DistServe paper frames as “the maximum rate that can be served within both TTFT and TPOT constraints on each GPU.”

This distinction is live in practice, not just in papers. Kiely and Taha put it bluntly: “there’s tokens per second as the throughput number, and the latency number. Most people only care about tokens per second as the latency number.” Two different quantities, one name. Separate them explicitly before you draw anything.

Commit to numbers. I would propose TTFT p95 under 500 ms for prompts up to 8K, and TPOT p95 under 25 ms — about 40 tokens/sec/stream, roughly reading speed.

The capacity math

Take a 70B-class model on an H200. Weights in FP8, KV cache in FP16.

The KV cache per token:

bytes/token = 2 × layers × kv_heads × head_dim × dtype_bytes
            = 2 × 80 × 8 × 128 × 2
            = 327,680 bytes  ≈ 320 KiB/token

The leading 2 is key plus value. The 8 KV heads rather than 64 is grouped-query attention, which is why this number is tolerable at all.

Now the HBM budget, and it is worth keeping the units honest — NVIDIA quotes capacity in decimal GB while KV cache math is naturally binary. The H200 SXM has 141 GB of HBM3e at 4.8 TB/s, which is 131 GiB. FP8 weights take 70 GB, or 65 GiB. That leaves 66 GiB; reserve ~4 GiB for activations and workspace and call it 62 GiB for KV cache:

62 GiB ÷ 320 KiB/token ≈ 203,000 tokens of KV cache

That single number sizes everything:

• At 8K average context: ~25 concurrent requests per GPU.
• At 32K: ~6.
• At 200K — the long-context case the podcast opens with: one.

One request, occupying 61 GiB — 200,000 × 320 KiB, which is essentially the entire KV budget of the GPU. That is the number I would write largest on the board, because it quietly reprices the product: at 200K context you are not selling tokens, you are renting most of a GPU.

Why decode is bandwidth-bound

Every decode step reads the full weight matrix to produce one token per sequence, plus each batched sequence’s KV cache for attention. Count only the weights first, because that is the version most people carry around:

2 × P × B FLOPs ÷ P bytes = 2B FLOP/byte      (FP8 weights, 1 byte/param)

The H200’s FP8 ridge point — peak dense compute over bandwidth — is 1,979 TFLOPS ÷ 4.8 TB/s ≈ 412 FLOP/byte, taking the dense half of the 3,958 FP8 TFLOPS NVIDIA quotes with sparsity. At batch size 1 you sit at 2 FLOP/byte: 0.5% of the silicon you are paying for. The weight-only model says B ≈ 206 closes the gap.

It doesn’t, and the correction is the more useful half. KV reads scale with the batch too:

intensity = 2·P·B ÷ (P + B × L × 320 KiB)

At 8K context and full occupancy — B = 25, from the budget above — KV traffic is 61 GiB against 65 GiB of weights. The two are comparable, so roughly half the read is something the weight-only model never counted. Push B higher and the denominator grows as fast as the numerator, so intensity converges to a ceiling with no batch term in it at all:

2P ÷ (L × 320 KiB)  =  53 FLOP/byte at 8K,  13 at 32K,  2 at 200K

No batch size saturates decode compute. At 8K context the ceiling sits eight times below the ridge, and longer context makes it worse rather than better. Decode is structurally memory-bound: batching buys throughput until KV traffic dominates the read, then it stops buying anything. That is the real reason a decode pool looks nothing like a training cluster, and the reason the next section exists.

It also bounds a claim worth stating carefully. In the weight-dominated regime — small batch, short context — halving precision doubles compute and halves weight bytes together, so the batch you would need is unchanged at 206. Once KV cache dominates the read, weight precision stops being the lever and KV precision becomes one.

Prefill sits on the other side of the roofline. At ~2P FLOPs per token, one H200 does 1,979 TFLOPS ÷ 140 GFLOPs ≈ 14,000 prefill tokens/sec at FP8 peak, so a 200K prompt is ~14 seconds of pure compute at 100% utilization and closer to 30 at a realistic 50%. Nobody waits 30 seconds for a first token, which is why prefill gets sharded across GPUs.

Prefill and decode belong on different GPUs

This is the architectural claim I would build the whole design around, and it falls straight out of the numbers above: prefill saturates compute, decode saturates bandwidth. Colocate them and a long prefill stalls every decode sharing that GPU.

There are two published answers, and a strong candidate knows both.

Chunked prefill keeps them together but slices the prefill. Sarathi-Serve splits a prefill into near-equal chunks and builds “stall-free schedules that add new requests in a batch without pausing ongoing decodes,” reporting 2.6× serving capacity for Mistral-7B on one A100 and up to 5.6× for Falcon-180B.

Disaggregation separates them physically: one pool of GPUs processes the input and produces the KV cache and the first token, then hands that cache to a separate pool that runs decode. DistServe reports this serves 7.4× more requests, or meets a 12.6× tighter SLO, than colocated systems while holding latency for over 90% of requests.

The warehouse comparison is the one I reach for: this is why you do not run overnight ETL and interactive dashboards on the same Snowflake warehouse. Same data, opposite access patterns, and the bursty job starves the latency-sensitive one. The fix in both worlds is separate compute against shared state — and in both worlds it moves the hard problem to state transfer, which is where the bodies are buried.

The cache hierarchy is the real system

Once prefill and decode are separate, KV cache becomes the object the architecture is organized around. Four tiers, cheapest first:

1. Paged HBM. vLLM’s PagedAttention manages KV cache in fixed blocks like OS virtual memory, achieving “near-zero waste in KV cache memory” and 2–4× the throughput of prior systems at the same latency. This is table stakes.
2. Prefix reuse. SGLang’s RadixAttention automatically reuses KV cache across requests sharing a prefix, reporting up to 6.4× higher throughput. Providers surface the economics directly: Anthropic bills a cache read at ~0.1× base input price against 1.25× for a 5-minute write, so the break-even is two requests.
3. CPU / NVMe offload. Cheaper than recompute for a warm conversation, more expensive than HBM.
4. Recompute. The fallback, and sometimes correct — recomputing a short prefix beats fetching it.

The router is what makes tiers 1 and 2 pay off. It must be cache-aware rather than round-robin: send a request to a replica that already holds its prefix, because a miss here turns a 0.1× cache read back into a full recompute.

So the router needs per-replica prefix-cache state, queue depth tracked separately for prefill and decode, and a scoring function that trades cache-hit value against load. Say out loud that this makes the router stateful, and that a stateful router needs its own failover story.

Decode-side multipliers

The cache tiers reduce work you would otherwise repeat. Two more levers change the cost of the work that remains, and they differ in one way that matters.

Speculative decoding converts spare compute into tokens. A draft model proposes several tokens; the target verifies them in one forward pass. Leviathan et al. report 2×–3× on T5-XXL “without changing the distribution” — genuinely lossless. It works for exactly the reason the roofline gave: verifying several tokens costs one weight read instead of several, and spare compute is the thing decode has in surplus. The size of the win depends on acceptance rate, which Baseten illustrates concretely: “if you’re summarizing Harry Potter books, I can train exclusively that draft model on Harry Potter books, and I can guarantee you that I’m gonna accept the three tokens every single time.” Narrow traffic, high acceptance — which is an argument for per-customer draft models, and therefore for dedicated deployments.

Quantization is the exception. It is the only lossy step in this stack, which is why it belongs behind a quality gate and an eval suite while the rest do not.

Multi-tenancy is a pricing problem

Fair queueing is the wrong frame. Latency is the product, so sell it.

• Admission control. Per-tenant token buckets on requests per minute and tokens per minute, separated into input and output. Return 429 with a retry-after header rather than degrading everyone.
• Priority classes. An interactive tier that holds reserved decode slots, and a batch tier that consumes headroom. Anthropic’s Batch API prices that deferral at 50% of standard rates with a 24-hour completion window — a real number for what latency is worth.
• Shared versus dedicated. Shared endpoints amortize weights across tenants. Dedicated wins when a tenant’s volume justifies a custom draft model, a specific quantization, or a hard SLA — and the cost floor is a dedicated GPU rather than a per-token rate, so the crossover is a volume question you should be able to state.

What actually breaks

Three failure modes, from the operationally boring to the genuinely nasty.

• Queue collapse. Decode slots fill, prefill queues grow, TTFT degrades and clients retry — which adds load. Needs load shedding at the admission layer and a queue-depth signal in autoscaling, not just GPU utilization.
• Autoscaling lag. Cold-starting a 70B replica means loading 70 GB of weights. Your scale-up latency is minutes; your traffic spike is seconds. Warm pools are not optional.
• Heterogeneous hardware. The one I would not have predicted: identical software on different clusters producing different results, because “the KV cache transfer from a node to node in that one cluster is using a slower interconnect than the node to node in another cluster.” That is one team’s reported experience rather than a published result, so I would offer it as an anecdote and not a law — but the structural point stands on its own. Disaggregation turns cache transfer into a correctness surface, not just a latency one.

The probes to expect

• A tenant sends one 500K-token prompt. What happens? It does not fit in one GPU’s KV budget. Answer with prefill sharding and a per-request context cap that is a product decision, not a bug.
• Your p99 TTFT is fine but p99.9 is 30 seconds. Why? Almost certainly head-of-line blocking from an unchunked prefill, or a cache-aware router hot-spotting onto one replica.
• How do you roll out a new model version safely? Weights are the deploy artifact and they are enormous. Shadow traffic, per-replica canaries, and a quality gate — quantization regressions are silent.
• Cut cost 40% without touching the SLO. Quantize, raise the prefix cache hit rate, then move eligible traffic to the batch tier. In that order, because only the first is lossy.

The thing I would not do is present any of this as a stack of techniques. Every item above is downstream of two numbers: 320 KiB of KV cache per token, and a ridge point at 412 FLOP per byte. One says how many requests fit. The other says how many you would need for the GPU to be busy. They disagree by an order of magnitude, and every component in the design is something built in that gap. Get both on the whiteboard in the first fifteen minutes and the architecture argues for itself.