Definition
Continuous batching is an inference scheduling technique for autoregressive generation. The scheduler can rebuild the active batch at token-iteration boundaries, removing requests that have finished and admitting waiting requests while longer generations continue.
This matters because requests rarely produce the same number of tokens. In a static batch, a short request may wait for the longest request before its slot can be reused. Continuous batching gives the server repeated opportunities to fill freed capacity, which can raise accelerator utilization and throughput.
Origin and naming
The 2022 Orca paper introduced the foundational mechanism under the name "iteration-level scheduling." Orca scheduled one model iteration at a time and used selective batching to accommodate transformer operations with different batching behavior. The later label "continuous batching" describes this family of schedulers, but the paper does not establish who coined that label.
Tradeoffs
Continuous admission is a scheduling policy, not free capacity. The server still has to account for KV-cache memory, prefill work, request priorities, cancellation, and fairness. A policy tuned only for total throughput can increase tail latency or delay large requests. Implementations also differ in when they admit work and how they combine prefill with decoding, so the label does not specify one exact algorithm.
Distinguish it from nearby terms
- Static batching holds a fixed group of requests together for a larger portion of their lifetime.
- Dynamic batching may collect requests that arrive within a short window before starting a batch. Continuous batching goes further by changing membership between generation iterations.
- Paged attention manages KV-cache memory. It complements continuous batching but is not the scheduler itself.
Check your understanding
Two requests start together. One needs 20 output tokens and the other needs 2,000. Describe when a continuous scheduler can reuse the short request's slot and why that does not guarantee low latency for every waiting request.