What Happens When You Ask ChatGPT a Question

You type a question into ChatGPT and hit send.

A JSON body leaves your laptop as an ordinary HTTPS POST. The same POST, more or less, whether the box you typed into was ChatGPT, Claude, Cursor, or a curl against OpenAI’s API. A few hundred milliseconds later, a token shows up, then another, then another. Most of us treat that gap as a black box: text went in, tokens came out. Fine, until you care why the first token paused, why a short answer cost a long bill, or why the same prompt is cheap on the second call and expensive on the first.

Let’s follow one request through the stack, the same way those “what happens when you type a URL” posts follow a packet from your keyboard to a server and back.

The request is this:

curl https://api.example.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3-8b-instruct",
    "stream": true,
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Explain the KV cache in one sentence."}
    ]
  }'

Every hop below is that prompt, or a token that grew out of it.

The POST leaves your laptop

The request is encrypted on the way out, lands at a server, and gets checked like any other paid API: is this key valid, have you blown your quota. Then someone has to take "Explain the KV cache in one sentence." and turn it into something a GPU can run.

If you sent the same system prompt a moment ago, the server already has that work sitting around and skips a chunk of what follows. First time through, it does all of it.

Tokenizing the prompt

A byte-pair encoding tokenizer, trained with the model and frozen at inference, maps text onto a fixed vocabulary of integer IDs. Llama 3’s vocabulary is 128,256 entries. Many production models sit in that 100k–200k range.

"KV" might be one token. " cache" might be another, with a leading space baked in. "Hello world" is [9906, 1917] on one model and [15496, 995] on another. Tokenization is cheap. Context limits, prefill cost, decode cost, and the KV cache all scale with the token count.

Your system prompt and user prompt are concatenated (plus special tokens for roles) and become a sequence of IDs. Say the whole thing is 40 tokens. Those 40 integers are now the request.

Turning tokens into vectors

The embedding matrix is what makes that jump: one of the model’s own weight matrices, learned during training just like every other weight, with one row per token ID. Llama 3 8B’s version has 128,256 rows, one for every entry in the vocabulary, and each row is a list of 4096 numbers. Token 857 becomes a vector by fetching row 857 from that matrix. No math, just a fetch.

After this hop you have a matrix of shape [40 × 4096]. That matrix is what the transformer eats.

Attention and the FFN, 32 times over

Llama 3 8B has 32 transformer layers. Every layer does the same two things to your [40 × 4096] matrix: attention lets each position pull in context from the other positions, then a feed-forward network transforms each position on its own. Stack 32 of those and the vector for “cache” has picked up context from “KV,” “Explain,” and everything else in the prompt. I will explain attention and FFN more in depth in a future post.

Attention projects the hidden states into Q, K, and V, then computes softmax(QK^T / sqrt(d_head)) * V. Each position gathers from every other position, weighted by relevance. Production kernels (FlashAttention-3, FlashInfer) never materialize the full N×N score matrix; they tile the work in on-chip SRAM. Some models also use sliding-window attention on some layers, so attention does not always look at the full prompt.

Llama 3 8B doesn’t give every query head its own key and value, the way older models did. It uses grouped-query attention: 32 query heads, but only 8 key/value heads, shared in groups of 4. Fewer KV heads means a smaller KV cache, which is why the old “a 7B’s cache is about 1GB” rule of thumb is off by roughly 4× for this model.

On a dense model like Llama, every FFN parameter fires for every token. Mixture of Experts models swap that single FFN for many smaller ones, called experts, plus a router that looks at each token and sends it to only a handful of them. DeepSeek-V3 has 671B parameters total but only about 37B of them fire for any given token, because the rest belong to experts the router didn’t pick. That’s a different serving problem than Llama 70B, where every parameter is always in use. More on how the routing actually works in an upcoming post.

After 32 layers, each of your 40 positions holds a representation that has looked at every other position.

Prefill vs. decode

No output token exists yet. The server is in prefill: one forward pass over all 40 known tokens, in parallel. That is why a 1,000-token prompt can still take milliseconds on a modern GPU. The work is large (full attention is O(n²) in sequence length) but it maps onto matrix multiplies, and GPUs are built for those.

When prefill finishes, two things exist: a final hidden state for the last prompt token, and a KV cache for every token in the prompt. Time-to-first-token, on a normal chat model, is “how long did prefill take, plus one decode step.”

Decode is the other phase. Each new token is its own forward pass. Only the last position’s output matters. For an 8B dense model on an A100, a well-optimized step is roughly 5-10ms. Prefill keeps the tensor cores busy. Decode waits on HBM: the step is reading the weights, then a relatively cheap multiply.

Those two phases fight if they share a GPU. A newly arrived 10,000-token prefill wants a huge compute burst; requests already decoding want tiny, steady steps. Chunked prefill is the intra-GPU compromise most engines use: the new prompt is broken into chunks and interleaved with decode so one whale does not stall every in-flight stream. That’s a fix within a single GPU. Some serving stacks (vLLM, SGLang, Mooncake, NVIDIA Dynamo) go further and give prefill and decode their own separate pools of GPUs entirely, shipping the KV cache between them over NVLink or RDMA once prefill is done. That transfer costs something, so it’s only worth it past a certain scale. More on that trade-off in an upcoming post.

The KV cache is why the next token is affordable

Without a cache, generating token N would recompute K and V for every previous token. A 500-token reply would do O(500²) attention work instead of 500 steps. The cache stores those K and V tensors and reuses them. Token N computes K and V for itself, reads the rest.

Count KV heads in the formula, not query heads:

KV cache = 2 × layers × kv_heads × d_head × context × batch × bytes

Llama 3 8B, 2,000 tokens, one request, FP16:

2 × 32 × 8 × 128 × 2000 × 2 bytes ≈ 250 MB

Full multi-head attention on the same skeleton is ~1GB, which matches an older MHA 7B. Llama 3 8B with GQA is 250MB at 2k tokens and about 12.5GB at 100k tokens.

DeepSeek took a different path to the same problem, called multi-head latent attention. Instead of storing the full K and V for every attention head, it first compresses them down into one small, shared vector per token, called a latent, and caches that instead. When the model needs to do attention, it reconstructs the full K and V from that latent on the fly. You are trading a bit of extra compute per token for a much smaller thing to store and read from memory. On DeepSeek-V3 that latent is 576 numbers per token per layer. Across the model’s 61 layers, in FP16, that works out to about 70KB per token, or roughly 7GB of cache at 100k tokens of context. Store the full K and V the normal way and the equivalent cache would run into the hundreds of gigabytes.

There’s another way to shrink the cache that doesn’t touch the model’s architecture at all: reuse. If twenty different requests all start with the same system prompt, that shared prefix produces the exact same K and V values every time, so there’s no reason to recompute it twenty times. vLLM’s automatic prefix caching and SGLang’s RadixAttention both keep a lookup structure of previously computed KV blocks, so a new request that starts with a prompt you’ve already seen can reuse that cache instead of running prefill on it again. APIs pass the savings on as a cache-hit discount on your bill. This only works if the same prompt tends to land on the same server, which is a routing decision, not a cache one, and it’s easy to get wrong.

The model draws a token

The last prompt position’s hidden state goes through the language-model head: a linear layer from d_model to vocabulary size. You get 128,256 logits. Softmax turns them into a distribution. Sampling picks one.

Temperature, top-p, and top-k reshape the distribution before the draw. The common Chat Completions default is temperature 1 with top-p left uncapped; nucleus 0.9 is an override. Reasoning models often use a lower temperature, and some APIs drop the knobs entirely in favor of a thinking budget.

Suppose the draw is The. That ID is appended to the sequence. Your cache grows by one column. Decode runs again.

That token comes back to you

The server writes The onto the HTTP response as a server-sent event (or a WebSocket frame) the moment it exists. You see a pause, then a tick, then a tick.

The pause is prefill. There is nothing to stream until the first decode step finishes. After that, tokens arrive at the model’s decode rate, typically 20–80 per second depending on size and hardware.

A 10,000-token system prompt still produces a long pause unless the prefix is already cached. Reasoning models can finish prefill in milliseconds and still make you wait, because they are decoding thinking tokens you may never see. Time-to-first-visible-token is no longer a clean proxy for prefill. A 200-token answer can easily cost a few thousand billed decode steps.

Then it happens again. And again.

Token 4 cannot exist before tokens 1–3. There is no intra-response parallel generation. That sequential tax is the constraint every inference trick is working around. The loop continues until a stop token or a max-output limit.

Your original prompt, “Explain the KV cache in one sentence,” might come back as: The KV cache stores precomputed keys and values so each new token does not redo attention over the whole prompt. Maybe 20 tokens. Twenty sequential GPU passes after that one parallel prefill.

Your request is not alone on the GPU

Running one request at a time is wasteful. The weights get loaded from HBM whether the batch is 1 or 32. Extra requests amortize that read.

Static batching waits for the longest request. If yours is 20 tokens and a neighbor is 500, you sit in padding. Continuous batching lets a finished request free its slot immediately. A new prefill slides in while yours is still decoding.

PagedAttention stores KV in fixed-size pages, like virtual memory, so you do not reserve a contiguous max-length slab per request. Finished pages go back in the pool. Every serious engine now has some version of this. If a stack still does static batching, that is the first thing to change.

If the weights do not fit

An 8B model in FP16 is about 16GB. A 70B is 140GB. An H100 still has 80GB, so 70B FP16 still needs more than one of them. Production serving usually does not stay in FP16: FP8, and 4-bit on Blackwell, is how a 70B fits with cache to spare. An H200 has 141GB; a B200 has 180GB. Hardware moved the dense-70B line. The models people want to serve got sparse and much larger, so you still split them.

There are a few ways to split one model across multiple GPUs, and they cut it up differently.

Tensor parallelism splits a single layer across GPUs, the way four people might carry one heavy table, each holding a corner. Every GPU works on the same layer, on the same token, at the same time, each doing its slice of the matrix multiply. Because they’re all touching the same computation simultaneously, they have to sync up after every layer to combine their partial results before moving to the next one. That sync happens on every layer, for every token, so it only pays off when the GPUs are connected fast enough (NVLink, not a network cable) that the chatter doesn’t eat the savings.

Pipeline parallelism is an assembly line instead of a shared table: GPU 0 owns layers 1–20, GPU 1 owns layers 21–40, and a token is fully done with GPU 0 before it ever touches GPU 1. No mid-layer syncing. The catch is the same as any assembly line: if there’s only one token in the pipeline, GPU 1 just sits there waiting for GPU 0 to finish, doing nothing. You only get the benefit once you have enough requests flowing through that GPU 0 is starting request 2 while GPU 1 is finishing request 1, keeping every stage busy.

Expert parallelism is specific to Mixture of Experts models. Instead of splitting a layer or the model, you split up the experts themselves: different experts live on different GPUs, and the router sends each token only to the GPUs holding the experts it picked, not to every GPU. If you’re serving a big MoE model, you’re doing this whether you meant to or not.

A shortcut: guess several tokens, check them in one pass

Speculative decoding is how production stacks dodge that sequential tax, and it works by loading two models instead of one. Alongside the big model actually serving your request, there’s a much smaller, much faster “draft” model sitting on the same GPU. The draft model guesses ahead: it generates N candidate tokens on its own, cheaply, without waiting for the big model at all. Then the big model checks all N guesses at once, in a single forward pass, instead of generating them one at a time itself, because checking a token (does this fit, given everything before it) is cheap compared to generating one from scratch. It accepts the prefix of guesses it agrees with and throws out everything after the first one it doesn’t. The tokens you keep are exactly as good as if the big model had generated them itself, one at a time, checking N candidates in the time it would have taken to generate one.

On code and factual text you might keep 3-4 of 5 drafts (3-4x). On open-ended generation the draft disagrees more and the gain shrinks. Production stacks usually do not ship a second LLM. EAGLE-3 trains a lightweight head on the verifier’s hidden states. DeepSeek-V3 and GLM-class MoEs bake multi-token prediction into the base model, so the draft is a native next-N head.

Back at your terminal

The stream ends. The KV cache stores precomputed keys and values so each new token does not redo attention over the whole prompt. That’s the whole response: 20-ish tokens, a couple hundred milliseconds, and to you it just looked like text appearing on screen a word at a time.

Here’s everything that actually happened between your curl and that sentence: the prompt got tokenized into integers, those integers became vectors, 32 transformer layers turned that matrix into something that understood its own context, one parallel prefill pass read the whole prompt at once, a KV cache got built so the next 20 tokens didn’t have to redo that work, and then a sequential decode loop ground out one token at a time, sharing the GPU with whoever else’s requests were in flight, maybe with a smaller model guessing ahead of it, maybe with a reasoning loop thinking through tokens you never saw. All of that, for one sentence explaining a KV cache. Which is a decent way to end this: the thing that made your answer possible is also the thing your answer described.

If you found this interesting, I’d love to hear your thoughts. Share it on Twitter, LinkedIn, or reach out at guptaamanthan01[at]gmail[dot]com.