Rate limiting

Enables controlled burst tolerance while enforcing a long-term average rate cap on incoming requests or messages.

API rate limiting: throttling per-client requests per second at a gateway; Network traffic shaping: smoothing bursty ingress before a constrained link; Resource protection: shielding a database or worker pool from request floods

API gateways, CDN edge nodes, distributed systems, cloud rate-limit services, network QoS pipelines

define capacity and refill rate → on each request, compute elapsed time and top up tokens → if tokens ≥ 1, decrement and allow; otherwise reject or queue

tokens = min(capacity, tokens + rate × elapsed); if tokens >= 1: tokens -= 1; allow

• rate limiter
• burst size
• token rate
• leaky bucket
• traffic shaping

• Leaky bucket (similar smoothing but different algorithm)
• Fixed window counter
• Sliding window log

Setting token refill rate too low causing unnecessary request drops – caused by misunderstanding burst capacity vs sustained rate – leads to degraded user experience Using a shared token bucket across unrelated services causing unintended throttling – caused by sharing state across boundaries – leads to cross‑service interference Failing to reset token count after a period causing burst accumulation – caused by forgetting to reset on interval – leads to burst bursts exceeding intended limits Using integer token counts with fractional request costs causing rounding errors – caused by mismatched units – leads to inaccurate throttling Ignoring network jitter and treating token arrival as perfectly periodic – caused by assuming ideal clock – leads to bursty traffic under load

Leaky bucket: smooths traffic by leaking at constant rate, unlike token bucket which allows bursts up to bucket size Fixed window counter: resets counter at fixed intervals, can allow bursts at window boundaries unlike token bucket's smooth rate Sliding window log: tracks timestamps of each request for precise rate limiting, more memory intensive than token bucket

Coming from Java's Semaphore: may confuse permits with tokens, but tokens are replenished continuously rather than acquired/released discretely Coming from Go's rate limiter (golang.org/x/time/rate): may assume token bucket automatically handles burst size, but must explicitly set burst parameter Coming from AWS API Gateway throttling: may assume token bucket is per‑method, but it can be shared across methods leading to unexpected throttling

• Rate limiting
• Traffic shaping
• Congestion control

Do not use token bucket when you need strict per‑request latency guarantees; it allows bursts that can cause latency spikes. Implementation must handle token overflow correctly to avoid unbounded memory usage. Token bucket works best when traffic is bursty but average rate is bounded.

Provides smooth traffic shaping that absorbs short bursts while enforcing long‑term rate limits, preventing service overload and ensuring fair resource usage.

Imagine a bucket that catches water droplets (requests) at a steady drip rate; you can pour a handful in at once (burst) but it slowly leaks out, keeping the flow steady.

Token bucket algorithm is often combined with a leaky bucket for both burst shaping and average rate limiting.

Consider moving to a hierarchical token bucket or adaptive rate limiter for dynamic traffic patterns.