Definition
A rate limit constrains how many requests, tokens, compute units, or actions may be consumed in a defined time window. Providers use limits to protect capacity, control abuse, enforce service tiers, and preserve fairness. Applications use them to prevent one user, agent, or workflow from exhausting a shared dependency.
Common implementations include fixed windows, sliding windows, and token buckets that allow a controlled burst before replenishing. Limits may apply by account, organization, model, endpoint, region, credential, or operation. A system needs to know which boundary it is sharing before it can schedule concurrent work correctly.
Design the response, not only the threshold
When a limit is reached, agents should not all retry immediately. Respect provider retry signals, add bounded exponential backoff and jitter, queue fairly, preserve idempotency, and stop when delay would violate the run contract. Reserve capacity for recovery or high-priority work where the risk justifies it.
Distinguish it from nearby terms
A quota or budget limits total consumption over a larger scope. A concurrency limit caps simultaneous work. A rate limit controls pace over time. Backpressure is the wider mechanism by which downstream capacity slows upstream production.
Check your understanding
Twenty agents share one provider key and all receive a 429 response. Should each retry after one second? No. Coordinate the shared limit, honor the reset or retry interval, add jitter, prioritize work, and avoid duplicating side effects after uncertain responses.