Definition
An inference scheduling technique that adds and removes generation requests at iteration boundaries as capacity becomes available.
Distinguish it from nearby terms
A static batch holds a fixed group of requests together. Continuous batching can admit a new request when another finishes, even while longer requests continue generating.
Check your understanding
Continuous batching can improve accelerator use and throughput, but the scheduler still has to manage latency, fairness, and memory pressure.