LLM Batching Explained: Static vs Dynamic vs Continuous Batching

Imagine four people booking a ride to the same destination. If each person takes a separate taxi, four vehicles make the same trip. But if they travel together in a bus, everyone reaches the destination in a single trip. The bus carries more people at once and makes better use of the resources.

Batching in AI works in a similar way. Instead of processing one request at a time, a server groups multiple requests together so the GPU can process them more efficiently. This helps serve more users while making better use of expensive computing resources. But how those requests are grouped and scheduled also affects how quickly users get responses and how much it costs to serve them.

There are three common approaches to batching: static batching, dynamic batching, and continuous batching. Let’s understand what batching means technically and how these approaches differ.

What is Batching?

Batching is the process of running multiple inputs through a model in a single forward pass instead of processing each input individually. In an LLM, each input is a sequence of tokens, so the model processes a batch of sequences together. Conceptually, instead of an input tensor shaped [1, seq_len], the model processes a batch shaped [batch_size, seq_len].

This allows the GPU to perform computations across multiple requests in parallel, improving hardware utilization and throughput. However, LLMs generate responses token by token, and different requests can take different amounts of time to finish. How the server groups requests, handles incoming traffic, and frees up resources as responses finish determines which batching approach works best.

Why LLMs Need Batching?

The answer is simple, during generation, the GPU is limited by how fast it can read memory, not how fast it can do the math. Batching lets one memory read serve many requests. 

LLM inference has two phases: 

1. Prefill: The model reads your whole prompt in one forward pass. All prompt tokens are processed in parallel, so this phase keeps the GPU’s compute units busy. 
2. Decode: The model generates one token at a time. Each new token needs a full forward pass, which means reading every model weight from GPU memory again. 

Decode is where the waste happens. Some rough numbers for a 7B model in FP16 on an H100: 

• The weights take about 14 GB. 
• H100 memory bandwidth is about 3.35 TB/s, so reading 14 GB takes roughly 4 ms. 
• That caps a single request at around 240 tokens per second. 
• Each token needs about 14 GFLOPs (2 × parameters). Doing that in 4 ms is around 3.5 TFLOPS, while the H100 offers close to 1,000 TFLOPS of dense BF16. 

So at batch size 1, the GPU’s compute sits more than 99% idle. It spends almost all its time loading the weights. If you run 32 sequences in the same steps, the weights are loaded once and you get 32 tokens instead of 1, nearly the same step time. That’s the whole case of batching higher throughput and lower cost per token, without buying more GPUs. 

There is a limit. Every sequence in the batch keeps its own KV cache in GPU memory, and that cache grows with every token. Batch size ends up bounded by memory, which becomes important when we get to continuous batching.

Also Read: The Four Caches in LLM Serving

Static Batching

Static batching groups a fixed set of requests, runs them together, and returns results only when every request in the group is finished. Nothing new can join until the whole batch is done.

The model.generate() call from earlier is static batching. It’s the default behavior of most hand-written inference loops.

The problem is that LLM outputs vary wildly in length. Say a batch of 4 requests ends up generating 20, 50, 100 and 500 tokens. Then the batch runs for 500 decode steps, because it waits for the longest one. The 20-token request is done at step 20, but its slot sits idle (or computes padding) for 480 more steps. Useful work is 20 + 50 + 100 + 500 = 670 token-steps out of 4 × 500 = 2,000 possible. That’s about 33% utilization.A request that arrives at step 1 has to wait 499 steps before it can even start. 

This is called head-of-line blocking: one long request holds up everyone behind it.

To compare the three strategies fairly, let’s build a small simulator. It skips the model and only tracks decode steps, which is enough to see how scheduling affects GPU usage and latency. A new request arrives every 10 steps, and the GPU can run 4 sequences at once.

Code:

from collections import deque 
 
# Each request: (id, arrival_step, output_tokens) 
# A new request arrives every 10 decode steps, with very different output lengths. 
lengths = [120, 30, 60, 300, 40, 90, 25, 150, 50, 70, 35, 80] 
REQUESTS = [(f"R{i}", i * 10, n) for i, n in enumerate(lengths)] 
SLOTS = 4  # max sequences the GPU runs at once 
 
def summarize(name, total_steps, useful, finish): 
    latency = [finish[rid] - arr for rid, arr, _ in REQUESTS] 
    util = useful / (total_steps * SLOTS) 
    print(f"{name:<11} steps={total_steps:<4} slot_util={util:.0%}  " 
          f"avg_latency={sum(latency) / len(latency):.0f}  first_two={latency[:2]}") 
 
def static_batching(requests): 
    queue = deque(requests) 
    step, useful, finish = 0, 0, {} 
    while queue: 
        batch = [queue.popleft() for _ in range(min(SLOTS, len(queue)))] 
        step = max(step, max(arr for _, arr, _ in batch))  # wait until the batch is full 
        longest = max(n for _, _, n in batch) 
        for rid, _, _ in batch: 
            finish[rid] = step + longest                   # results only when the batch ends 
        useful += sum(n for _, _, n in batch) 
        step += longest 
    return step, useful, finish 

Two lines carry the whole story. The batch waits until it has 4 requests, and every request gets its result at step + longest, no matter how short it was.

When static batching is fine: offline jobs where you control the inputs and outputs are similar in length, like summarizing 10,000 documents with a fixed output cap.

Dynamic Batching

Dynamic batching changes how a batch is formed, not how it runs. Requests wait in a queue, and the server launches a batch when either of two things happens first: 

• The batch reaches its maximum size. 
• The oldest request in the queue has waited longer than a set timeout. 

This fixes one problem with static batching. Under light traffic, a request no longer waits forever for the batch to fill up. Under heavy traffic, batches still fill quickly and the GPU stays busy. 

NVIDIA Triton Inference Server is the classic example. You turn it on in the model’s config.pbtxt:

Code:

max_batch_size: 8 
 
dynamic_batching { 
  preferred_batch_size: [ 4, 8 ] 
  max_queue_delay_microseconds: 100 
} 

This tells Triton to aim for batches of 4 or 8, and to never hold a request more than 100 microseconds waiting for others to join.

Here is the same idea added to our simulator:

Code:

def dynamic_batching(requests, max_wait=15): 
    pending = deque(requests) 
    step, useful, finish = 0, 0, {} 
    while pending: 
        step = max(step, pending[0][1]) 
        deadline = pending[0][1] + max_wait 
        # launch when the batch is full OR the oldest request has waited max_wait 
        while sum(r[1] <= step for r in pending) < SLOTS and step < deadline: 
            step += 1 
        batch = [r for r in pending if r[1] <= step][:SLOTS] 
        for r in batch: 
            pending.remove(r) 
        longest = max(n for _, _, n in batch) 
        for rid, _, _ in batch: 
            finish[rid] = step + longest                   # still runs to completion 
        useful += sum(n for _, _, n in batch) 
        step += longest 
    return step, useful, finish 

Look at the line finish[rid] = step + longest. It’s the same as in static batching. Once a batch starts, it still runs until its longest request is done.

That’s why dynamic batching works so well for models where every request costs about the same: BERT classifiers, embedding models, image models. One forward pass, done. For LLM generation, the straggler problem is still there. Worse, launching a half-full batch early can lock the GPU into running 2 sequences for 120 steps while new requests pile up outside. We’ll see this in the numbers shortly.

Continuous Batching 

Continuous batching makes scheduling decisions at every decode step instead of once per batch. When a sequence emits its end-of-sequence token, it’s returned right away and its slot is freed. A waiting request takes that slot on the very next step. The batch never drains; it keeps refilling.

You’ll also see it called iteration-level scheduling (from the Orca paper, OSDI 2022, which introduced the idea) or in-flight batching (NVIDIA TensorRT-LLM’s name for it).

Back to the 4-request example with 20, 50, 100 and 500 tokens. With continuous batching, the 20-token request returns at step 20, and a new request starts in that slot at step 21. The GPU stays close to full, and new requests start almost immediately.

Here is the last piece of the simulator:

Code:

def continuous_batching(requests): 
    waiting = deque(requests) 
    running = {}                                           # id -> tokens left 
    step, useful, finish = 0, 0, {} 
    while waiting or running: 
        # every step: fill free slots with requests that have arrived 
        while waiting and len(running) < SLOTS and waiting[0][1] <= step: 
            rid, _, n = waiting.popleft() 
            running[rid] = n 
        useful += len(running) 
        step += 1 
        for rid in list(running): 
            running[rid] -= 1 
            if running[rid] == 0:                          # done: return now, free the slot 
                finish[rid] = step 
                del running[rid] 
    return step, useful, finish 
 
if __name__ == "__main__": 
    summarize("Static", *static_batching(REQUESTS)) 
    summarize("Dynamic", *dynamic_batching(REQUESTS)) 
    summarize("Continuous", *continuous_batching(REQUESTS)) 

The difference is that the scheduling loop runs inside the decode loop. Each request’s finish time is its own, not the batch’s.

What makes it work in practice:

The idea is simple, but real systems need a few extra pieces:

• Mixing prefill and decode: New requests need prefill while existing ones generate tokens. Orca uses selective batching to handle both efficiently.
• KV cache management: vLLM’s PagedAttention stores KV caches in fixed-size blocks, reducing memory fragmentation and allowing more sequences to fit in a batch.
• Chunked prefill: Systems like Sarathi-Serve split long prompts into smaller chunks and interleave them with decode steps to reduce delays for running requests.

Static vs Dynamic vs Continuous Batching

Running the full simulator on the same 12 requests gives this:

Static      steps=560  slot_util=47%  avg_latency=402  first_two=[330, 320] 
Dynamic     steps=665  slot_util=39%  avg_latency=418  first_two=[135, 125] 
Continuous  steps=330  slot_util=80%  avg_latency=118  first_two=[120, 30] 

A few things stand out: 

1. Continuous batching finishes the same work in 41% fewer steps (330 vs 560) and cuts average latency from 402 steps to 118. The GPU slots are busy 80% of the time instead of 47%. 
2. Dynamic batching helps the first requests get results about 2.5x sooner than static, since it doesn’t wait for a full batch. But it does worse overall, because it launched half-empty batches that then had to run to completion. That’s the trade-off you get when a request-level batcher meets variable-length generation. 
3. R1 asked for only 30 tokens. With static batching it waited 320 steps for its answer. With continuous batching it waited exactly 30. 

The simulator ignores prefill cost and memory limits, so real numbers will differ, but the pattern holds on real serving engines.

	Static	Dynamic	Continuous
When a batch starts	Once a fixed group is ready	When full or the timeout hits	Every decode step
New requests join mid-run?	No	No	Yes
Finished sequences	Wait for the whole batch	Wait for the whole batch	Returned at once, slot reused
Scheduling level	Per batch	Per batch	Per iteration (token)
Best fit	Offline jobs, similar output lengths	Fixed-cost models (BERT, embeddings, vision)	LLM serving with variable outputs
Where you’ll see it	Plain model.generate() loops	Triton dynamic batcher	vLLM, TGI, TensorRT-LLM, SGLang

If you remember one line: static batching waits for everyone, dynamic batching waits less to start but still waits for everyone to finish, and continuous batching doesn’t wait for anyone. That’s why every serious LLM serving stack today is built on continuous batching, and why swapping a hand-rolled generate() server for vLLM is often the cheapest speedup you’ll ever get.

Conclusion  

Batching exists because LLM decode is memory-bound. Reading the weights once and using them for many sequences is the single biggest lever you have on cost per token. The three strategies are really a story of moving the scheduling decision to a finer level. Static batching decides once per group. Dynamic batching decides once per batch but stops waiting for a full group. Continuous batching decides on every token, so short requests never get stuck behind long ones. For most teams, the practical takeaway is simple: don’t build your own batching loop for LLM serving. Use an engine like vLLM, TGI or TensorRT-LLM, then spend your time tuning the knobs that matter, such as maximum concurrent sequences, tokens per step, and how much GPU memory goes to the KV cache.