log sampling

Observability

Meaning

Log sampling is a technique that records only a subset of log events based on a defined probability or rate. It reduces log volume and storage costs while preserving a representative view of system behavior. It is typically used when high‑frequency events would overwhelm logging infrastructure.

Primary Function

Logging

Communicative Purpose

Enables reduction of log volume by selectively recording events, preventing storage overload and performance degradation.

Pattern

initialize logger → set sampling_rate → log(event) if random() < sampling_rate

Core Structure

sample_rate = N / total_events

Função primária

Logging

Propósito comunicativo

Enables reduction of log volume by selectively recording events, preventing storage overload and performance degradation.

Situações de gatilho

High-throughput services: logging millions of requests per second; Embedded systems: limited flash storage for logs; Distributed microservices: network bandwidth constraints for log shipping

Contextos

Backend services, cloud functions, IoT firmware, high‑performance APIs, data pipelines

Padrão

initialize logger → set sampling_rate → log(event) if random() < sampling_rate

Estrutura central

sample_rate = N / total_events

Colocados típicos

sampling_rate
logger
log_event
probability
sample_rate
log_message

Substituições comuns

fixed interval sampling (log every Nth event)
probabilistic sampling (log with probability p)
rate limiting via token bucket

Erros comuns

Using a constant probability that is too low → important events may be missed; Applying sampling after expensive log message construction → defeats performance gain; Forgetting to include error logs in sampling → critical failures go unnoticed

Similar / contraste

Log throttling (rate limits log calls) vs log sampling (chooses subset of events); Log aggregation (collects all logs) vs log sampling (reduces volume)

Interferências

Coming from JavaScript: using setInterval to decide when to log – may cause drift and miss bursts; Coming from syslog: assuming logs are always line‑oriented – sampled logs may break parsers expecting continuous streams

Família do chunk

log rotation
log level filtering
structured logging
log aggregation

Nuance

Do not use when every error must be recorded for compliance; Sampling introduces statistical variance, so monitoring thresholds must account for reduced sample size; Edge cases where low‑probability events are critical require explicit handling outside sampling

Efeito pragmático

Reduces storage costs and I/O overhead, allowing systems to maintain observability under high load without sacrificing overall insight

Dica de memória

Log sampling is like a photographer taking a quick snapshot of a bustling crowd instead of filming the whole scene.

Nota

Choose a sampling rate that balances observability with resource constraints; consider dynamic adjustment based on recent error rates

Upgrade path

adaptive log sampling with dynamic rate adjustment based on error frequency

Frequência: HighFormulaicidade: Semi-fixedTipo de construção: conceptPrioridade de aquisição: Active recallPrioridade de output: BothTag de espaçamento: Short-term

Examples

import random import logging logger = logging.getLogger("app") logger.setLevel(logging.INFO) SAMPLING_RATE = 0.01 # log 1% of events def log_event(event): if random.random() < SAMPLING_RATE: logger.info("Event: %s", event) for i in range(1000): log_event(f"request_{i}")

import time import logging logger = logging.getLogger("app") logger.setLevel(logging.INFO) INTERVAL = 5.0 # seconds between logged events _last_logged = 0.0 def log_event(event): global _last_logged now = time.time() if now - _last_logged >= INTERVAL: logger.info("Event: %s", event) _last_logged = now for i in range(20): log_event(f"heartbeat_{i}") time.sleep(1)

import random import logging from collections import deque logger = logging.getLogger("app") logger.setLevel(logging.INFO) BASE_RATE = 0.05 ERROR_WINDOW = deque(maxlen=100) # track last 100 events def adjust_rate(): error_ratio = sum(ERROR_WINDOW) / len(ERROR_WINDOW) if ERROR_WINDOW else 0 return min(1.0, BASE_RATE * (1 + error_ratio * 5)) def log_event(event, is_error=False): ERROR_WINDOW.append(1 if is_error else 0) rate = adjust_rate() if random.random() < rate: logger.info("Event: %s (error=%s)", event, is_error) for i in range(200): log_event(f"op_{i}", is_error=(i % 50 == 0))