What is your GPU
waiting for?

In a previous blogpost [see “How many devs fit on a GPU?”], we studied how many developers can use a common LLM inference engine effectively without slowing down the GPU system to a crawl… and of course, the answer is “it depends”. The basic mechanics at play are worth an article on their own. In this article, our goal is to give you a back-of-the-napkin understanding of why a single-user chat session is barely noticed by your GPU, but a few long-context agentic coding sessions can make it break a sweat.

Autoregressive Large Language Models (LLMs) do exactly one thing: generating text. An LLM does this probabilistically, and exactly one word at a time: given the preceding sequence of words, the context, it produces a probability distribution over what it believes the next word should be. Feed it “Once upon a midnight” as context and it will — remembering Poe [1] — output a distribution with a high probability for “dreary”. Sample a word from that distribution (big chance it will be “dreary”), append it to the context, and run everything again to generate the word after “dreary”. Repeat until done.

To be precise, LLMs don’t operate on words, but on tokens. Tokens are chunks that may be any combination of (sub)-words, spaces, punctuation, symbols, etc. Sequences of these tokens can represent text from many natural and programming languages using a single unified vocabulary. Everything we provide as context is first converted into tokens, everything the LLM generates needs to be converted back into text before we read it. This is why many of the metrics we use (e.g., tokens per second) measure tokens instead of words.

As explained, longer texts are generated by repeatedly invoking the model, one token at a time. Naively, every generation step would process the entire (growing) context again from scratch. Nobody does that, because it would saturate your GPUs just to keep repeating the same computations on the same input. For this reason, intermediate results for processed tokens in the context are stored for reuse in the KV cache. Each generation step simply reads from that KV cache instead of churning through the same context repeatedly.

Initially, there is no KV cache yet to read from. The model has to process the whole input prompt at once to build up the KV cache and generate a single output token. This is called prefill. From there on, every subsequent step, called decode, just feeds the newest token as input to the model. Only this input token is processed, while the information associated with previous tokens is read from the KV cache. As in prefill, every decode step produces a single output token. Additionally, the KV cache is updated with the intermediate results of every added token.

In short. Prefill: process the whole prompt once, build the KV cache, and produce a single output token. Decode: process one token, read KV cache, append token to the KV cache, and produce a single output token. Rinse and repeat.

LLMs often receive multiple requests at once, each with its own context. The number of such concurrent sequences is the batch size. In each decode step, the model then generates one new token per sequence. Each sequence still drags along its own KV cache — an important detail for later.

AI is mostly just linear algebra, so most of the time goes into matrix multiplications. To reason about them, we use a well-established tool: the roofline model. It tells us that the runtime of any operation is determined by one of two ceilings: how fast you can move data around (memory bandwidth) and how fast you can crunch it (peak FLOPs/s).

Take a simple vector addition as an example. Let’s say both vectors are \(10^{9}\) elements long, with each element taking up 2 bytes. We would need to — at least — read both vectors (\(2\times2\times10^{9}\) bytes), do the additions (= \(10^{9}\) FLOPs) and write back the result (another \(2\times10^{9}\) bytes). On an NVIDIA B200, with a peak computation speed of \(C_{peak}\approx2.25\times10^{15}\) FLOPs/s for bfloat16, the compute would take a minimum of \(T_{comp}=10^{9}/C_{peak}\approx0.44\,\mu\text{s}\). With its HBM3e bandwidth of \(B_{mem}\approx8\times10^{12}\) bytes/s, moving the data would take \(T_{mem}=6\times10^{9}/B_{mem}\approx0.75\,\text{ms}\).

You may think those act sequentially: data is moved, processed and moved again. But modern hardware is optimized to overlap data transfers and compute. A better — although still idealized — estimate of the total run time is to model it as whichever of the two takes the longest: \(T_{op}=\max(T_{mem},T_{comp})\). In this example, \(T_{mem}\gg T_{comp}\), i.e., we are memory-bandwidth-bound. Being memory-bandwidth-bound is usually bad news: it means your expensive accelerator’s compute units are idling, waiting for data to arrive. Thus we’d rather be compute-bound instead; this is when \(T_{comp}\) dominates. Only then are we getting out of the hardware what we paid for.

To make this happen, we need our operations to perform a lot of FLOPs for every byte of data transferred: \(C_{peak}/B_{mem}\) FLOPs per byte, to be specific. This is approximately 280 FLOPs per byte on a B200. In other words, run time is bottlenecked by memory bandwidth unless we use operations that can heavily reuse each byte they load into memory.

The table below compiles the FLOPs per byte for some example operations (at 2 bytes per element). The takeaway here is that big matrix-matrix multiplies give you lots of computational bang for your memory bandwidth buck — operations on vectors don’t. The former are compute-bound; working with the latter will be bottlenecked by memory bandwidth.

Their matrix shapes depend on whether the model is performing prefill or decode, on the number of concurrent users (batch size), and on the context length. These shapes, in turn, determine whether each operation is limited by memory bandwidth or by compute performance. To estimate the total runtime of LLM inference, we must therefore identify the relevant operations and their dimensions.

Most modern LLMs are Transformers: a stack of identical layers, each doing two different things. An Attention Mechanism lets every token pull in information from the tokens before it. This helps with grammar, long-range reference, and context in general. A Feed-Forward Network (FFN) then processes each token independently and contributes much of the model’s linguistic and factual knowledge. Every token travels through the stack as a vector of numbers until the last layer. There, the final token’s vector becomes a probability distribution over the token vocabulary. From this distribution, the next token is sampled.

We now discuss both the Feed-Forward Network and the Attention Mechanism in more detail. We focus solely on whether the computations involved are memory-bandwidth-bound or compute-bound. If you’d like more details on the inner workings of the FFN and the Attention Mechanism, Sebastian Raschka’s work [2] is an excellent resource.

The Feed-Forward Network

There exist many different FFN architectures. Nowadays, a popular option is the SwiGLU-FFN (Swish-Gated Linear Unit). As you can see from the diagram, it looks quite complex.

From a roofline perspective, all that matters are the three matrix-matrix multiplications, indicated by \(\ast\) in the formula. The element-wise operations (\(\text{SiLU}, \odot\)) are cheap in comparison and can be ignored. To estimate the runtime of the FFN using the roofline model, we just need to understand the dimensions of the matrices involved.

The dimensions of the weight matrices \(W\) are dictated by the model architecture and thus fixed. They are relatively large: for the Llama 3 8B model, the dimensions are 4096 by 14336 for \(W_{gate}\) and \(W_{up}\) and 14336 by 4096 for \(W_{down}\). The \(X\) matrix is a stack of the \(b\) input token vectors. In the context of the FFN, it does not matter whether these \(b\) tokens belong to the same sequence, or to different users. For Llama 3 8B, the \(X\) matrix has dimensions \(b\) by 4096.

Therefore, the FFN’s fate is solely determined by \(b\): in the case of a single-user decode scenario \((b=1)\), \(X\) is a vector. All operations are then memory-bandwidth-bound matrix-vector multiplications. In contrast, if \(b\) is large enough (prefill or batched decode), the FFN’s main computations are matrix-matrix multiplications, which run at near-peak compute performance.

Great! The FFN is almost always using our shiny, expensive GPUs effectively! But now let’s turn our attention to the other half, the Attention Mechanism.

The Attention Mechanism

This is the place in the Transformer where tokens within the same sequence talk to each other. As our working example, we’ll examine a vanilla multi-head Attention Mechanism. The architecture is unquestionably even more complex than that of the FFN. But again, looking through the lens of the roofline model, we can abstract much of it. It is also where prefill and decode part ways, and we therefore discuss them separately.

The Attention Mechanism: Prefill

In the case of prefill, the input consists of \(b\) prompt tokens that belong to the same sequence. As with the FFN, their vector representations are stacked to form input matrix \(X\).

Again, we spend a considerable part of our compute on multiplying the input matrix \(X\) with weight matrices \(W^{Q}\), \(W^{K}\), and \(W^{V}\) for the input projections (bottom of the diagram). Those operations are easily compute-bound if the input prompt is not too small. By similar reasoning, the output projection with weight matrix \(W^{O}\) is compute-bound (top of the diagram).

Apart from those projections, we also have multiple attention heads (middle part of the diagram), each computing an attention operation \(O = \text{softmax}(Q \ast K^{T}) \ast V\). This is where tokens exchange information with each other. But those are still genuine matrix-matrix multiplications with a great FLOPs-per-byte ratio.

Single-user prefill can be extended to multiple users with a little bit of extra bookkeeping, such that tokens of one user do not interact with tokens of other users. But regardless of the number of users, the Attention Mechanism is typically compute-bound for prefill.

The Attention Mechanism: Single-user Decode

Prefill is about preprocessing a large number of prompt tokens, making it easily compute-bound. Decode is a very different beast. First, we focus on the single-user case.

During decode, the Attention Mechanism applies the same formulas as during prefill. The main differences are that we are now processing only a single input token, and that we rely on the KV cache to retrieve information of the past tokens.

These differences result in our matrices having different shapes. Where we previously had a large batch of \(b\) tokens, we now have only a single vector \((b=1)\) to multiply by our weight matrices, both for the input and output projections. Those matrix-vector multiplications are memory-bandwidth-bound.

It doesn’t get any better for the attention heads. Per head, we need to load the matrices (\(K\) and \(V\)) from the KV cache to multiply with our single vector (\(Q\)). It is first multiplied by the \(K\) matrix and after applying the softmax operation, the resulting vector is again multiplied by the \(V\) matrix. All matrix-vector operations.

The \(K\) and \(V\) matrices are then updated with the entry of the currently processed token. The \(K\) and \(V\) matrices in the KV cache therefore grow linearly with the number of context tokens, so each decode step becomes slower with every new token added to the sequence. This also explains why long-context decode requires high memory bandwidth: for every new token that is generated, the entire KV cache of that sequence has to be loaded from memory.

A solo session will typically leave a GPU severely underutilized because both the FFN and the Attention Mechanism are memory-bandwidth-bound during decode. As we saw previously, we can bring the FFN layers into a more compute-bound regime by batching requests from multiple users. Can we do the same for the Attention Mechanism?

The Attention Mechanism: Multi-user Decode

When decoding for multiple users, the Attention Mechanism receives multiple input tokens. This may appear similar to prefill, but there is an important difference: during decode, the \(b\) input tokens belong to \(b\) different users, whereas during prefill, the input tokens belong to the same sequence. Recall that each user comes with their own KV cache. Tokens from different users do not interact. Instead, each token gets information from the past tokens in its own sequence, whose keys and values are stored in that user’s KV cache. The core attention head computations therefore remain separate for each user and they remain memory-bandwidth-bound matrix-vector operations.

However, there is some benefit to batching: the input projections, involving weight matrices \(W^{Q}\), \(W^{K}\), and \(W^{V}\) as well as the output projection, involving \(W^{O}\), can be expressed as matrix-matrix multiplications by stacking the input tokens from the different users.

As a result, batched decoding improves performance through model weight reuse, but attention will always be bottlenecked by the memory bandwidth required to read the KV cache. Especially so for long contexts. This also explains why so many research efforts have gone into compressing the KV cache: grouped-query attention (GQA), sliding-window attention (SWA), multi-head latent attention (MLA), linear attention, etc., all aim to reduce the memory footprint of the KV cache.

In summary, the Attention Mechanism can operate in different regimes, depending on the number of users, the context length, and whether we are doing prefill or decode.

Well, there is a big downside. Users pay for our efficiency gains by having to wait longer for their tokens to pop up on their screen. The more other users we add to their batch, the more processing needs to be done before their next token arrives. Sure, that token may have been produced more efficiently, but ultimately still slower, and that’s the only thing an individual end user cares about.

With the roofline methodology we can put everything together and estimate the performance of an LLM inference engine. This helps us to understand the trade-off and find the best configuration for every situation. Given a certain number of users b¹, given the context length per user, and given the model parameters, we can infer all matrix and vector dimensions involved. As explained, by plugging in the hardware specs \(C_{peak}\) and \(B_{mem}\), we can compute \(T_{op}=\max(T_{mem},T_{comp})\) for each operation. Simply summing the runtime of all required operations gives us the total runtime \(T_{total}\) for one inference engine step.

For example, the table below lists the model parameters² for Llama 3 8B[3].

¹ We overload the usage of the symbol \(b\). Throughout, b counts the token vectors stacked into the input matrix X, regardless of whom they belong to. During prefill these are the prompt tokens, so b is large even for a single user. During decode, each user contributes exactly one token per step, so b equals the number of concurrent users.

² Llama 3 actually uses grouped-query attention (GQA), where groups of 4 query heads share a single K/V head. That is why the table lists 32 attention heads but only 8 KV heads. We describe vanilla MHA because the prefill and decode mechanics are the same and the diagrams stay simpler.

Remember that one LLM decode step provides us with one token per user. From the perspective of an individual user, their number of tokens per second is thus simply \(1/T_{total}\). Each individual user doesn’t care about other users, as long as this number of tokens/s/user is high enough, they are happy. This means \(T_{total}\) should not be too high. For example, for a \(T_{total}\) of 20 ms, each individual user gets 50 tokens/s, which is likely faster than they can read. The number of tokens/s/user is also called the interactivity.

An inference provider, who typically charges per generated token, has a different agenda. They want to maximize throughput, that is, they want a GPU system to generate as many tokens per second as possible across all users. Again, because one LLM decode step provides one token per user, the throughput is given by \(b/T_{total}\). This is a factor of \(b\) higher than the interactivity.

The plot below shows the trade-off between interactivity and throughput. The dashed lines are obtained through roofline modeling, the solid lines are actual measurements.

The horizontal axis shows the interactivity, expressed as tokens per second per user. This is the performance metric perceived by each individual user, who would like to be as far to the right on the graph as possible.

The vertical axis shows the total throughput, expressed as the total number of tokens generated per second across all users. An inference provider or any hardware owner would like to operate as high as possible on the graph.

The two most important variables that control the trade-off between interactivity and throughput are the average context length (different curves on the plot correspond to different context sizes) and the number of concurrent users (different dots on the same curve). The context length is largely dictated by the intended application: a chatbot typically requires a shorter context than a multi-turn agentic application that keeps track of a longer interaction history. The number of concurrent users, on the other hand, can be controlled by the inference provider.

Back to the two different regimes. For a single user, interactivity is high, but throughput is low. This corresponds to the rightmost dot on each curve. In this regime, the workload is severely memory-bandwidth-bound because most operations are matrix–vector operations: model weights have to be streamed from memory to the compute units to generate only a single output token. For an inference provider, this is an inefficient operating point. On the other hand, from a user’s perspective, all the compute is being spent on serving their tokens, not anyone else’s.

As the number of concurrent users increases, the FFN and some parts of the Attention Mechanism become less memory-bandwidth-bound. The model generates \(b\) new tokens at once, one per user, and the same model weights are reused across these tokens. Several operations shift from matrix–vector to matrix–matrix form. As a result, throughput increases rapidly, while interactivity decreases only modestly. This is the preferred operating region for inference providers, but users will need to wait a bit longer on while compute is done for their peers.

When the number of users keeps growing, the cost of the KV cache becomes increasingly important. Each user request has its own KV cache, and reading the cached keys and values remains memory-bandwidth-bound. At large batch sizes and/or large context sizes, the system therefore spends a relatively increasing fraction of time streaming KV cache data and again becomes more memory-bandwidth-bound. In this region, interactivity decreases rapidly with additional users, while throughput improves only marginally. Even though the total throughput is high, the individual user experience becomes poor, making this again an undesirable operating point.

The plot shows that roofline modeling is a useful indicator of real inference performance. It gives insight into how a workload maps onto the underlying hardware and where it hits a bottleneck. This is especially powerful combined with real measurements. When a measurement drifts away from the estimate, that gap helps diagnose performance issues and improve the system’s latency or throughput. We’ll deep-dive into the most relevant bottlenecks for LLM inference in follow-up posts so stay tuned!

1. Poetry Foundation (Edgar Allan Poe), “The Raven” poetryfoundation.org
2. Sebastian Raschka, “How to build an LLM from scratch” sebastianraschka.com. We also highly recommend his blog magazine.sebastianraschka.com
3. Meta, “Llama 3 Model Card” github.com
4. Further reading: for an excellent deep-dive into these topics, “How to Scale Your Model” jax-ml.github.io