request hedging

Resilience Patterns

Meaning

Request hedging is a latency‑optimization technique where the same request is sent to multiple replicas of a service, and the first response is used while the others are cancelled. It reduces tail latency by exploiting variability in service response times. Use it when services are idempotent and can tolerate the extra load.

Primary Function

Latency optimization

Communicative Purpose

Reduce tail latency by issuing redundant requests

Pattern

send request to multiple replicas → use first response → cancel others

Core Structure

latency = min(response_times)

Função primária

Latency optimization

Propósito comunicativo

Reduce tail latency by issuing redundant requests

Situações de gatilho

High‑variance latency services, tight SLOs, idle replica capacity

Contextos

Distributed microservices, cloud‑native systems, load‑balanced backends

Padrão

send request to multiple replicas → use first response → cancel others

Estrutura central

latency = min(response_times)

Colocados típicos

load balancing
retry policies
circuit breaker
tail‑latency budgeting

Substituições comuns

request replication
backup requests
speculative execution

Erros comuns

over‑hedging causing excess load, forgetting to cancel redundant requests, applying to non‑idempotent endpoints

Similar / contraste

request retry – re‑sending after a failure; hedging – sending duplicates proactively before any failure

Interferências

Coming from synchronous RPC backgrounds: may assume duplicate requests cause errors; ensure idempotency and proper cancellation.

Família do chunk

request replication
speculative execution
tail‑latency optimization

Nuance

Only effective when the service can handle the extra load and is idempotent; otherwise hedging can increase load and worsen latency.

Efeito pragmático

Cuts down the 99th‑percentile response time and improves user‑perceived performance.

Dica de memória

Think of placing a hedge bet: you make several wagers to capture the fastest win.

Nota

Hedging improves tail latency but adds extra load; it should only be used when the service is idempotent and can absorb the additional traffic.

Upgrade path

Adaptive hedging that dynamically adjusts the number of replicas based on observed latency

Frequência: MediumFormulaicidade: FlexibleTipo de construção: conceptPrioridade de aquisição: Recognition firstPrioridade de output: BothTag de espaçamento: Medium-term

Examples

import asyncio, aiohttp async def fetch(session, url): async with session.get(url) as resp: return await resp.text() async def hedged_request(urls): async with aiohttp.ClientSession() as session: tasks = [asyncio.create_task(fetch(session, u)) for u in urls] done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED) for p in pending: p.cancel() return list(done)[0].result() # usage example # asyncio.run(hedged_request(["https://service-a/endpoint", "https://service-b/endpoint"]))

import concurrent.futures, requests def hedged_request(urls, timeout=2): with concurrent.futures.ThreadPoolExecutor(max_workers=len(urls)) as executor: future_to_url = {executor.submit(requests.get, u, timeout=timeout): u for u in urls} for future in concurrent.futures.as_completed(future_to_url): try: response = future.result() # cancel remaining futures by shutting down the executor executor.shutdown(wait=False, cancel_futures=True) return response.text except Exception: continue raise RuntimeError("All hedged requests failed") # usage example # result = hedged_request(["https://svc1/api", "https://svc2/api"])

import threading, queue, requests def fetch(url, q): try: r = requests.get(url, timeout=3) q.put(r.text) except Exception: pass def hedged_request(urls): q = queue.Queue() threads = [threading.Thread(target=fetch, args=(u, q)) for u in urls] for t in threads: t.start() result = q.get() # blocks until first response arrives # stop other threads for t in threads: if t.is_alive(): # threads will exit after request timeout; no explicit kill needed t.join(0) return result # usage example # data = hedged_request(["https://api1.example.com", "https://api2.example.com"])