log retention policy

Observability

Meaning

A log retention policy defines how long log entries are kept before being deleted or archived, helping manage storage usage and comply with regulations.

Primary Function

Data management

Communicative Purpose

Specify the lifecycle of log data to balance diagnostic usefulness with storage costs.

Pattern

{ "retention_days": days, "compress": compress_flag }

Core Structure

{ "retention_days": ..., "compress": ... }

Função primária

Data management

Propósito comunicativo

Specify the lifecycle of log data to balance diagnostic usefulness with storage costs.

Situações de gatilho

Setting up a new service; auditing compliance requirements; experiencing log storage exhaustion.

Contextos

Cloud platforms (AWS, GCP, Azure), logging frameworks (ELK, Fluentd, Splunk), containerized environments (Kubernetes).

Padrão

{ "retention_days": days, "compress": compress_flag }

Estrutura central

{ "retention_days": ..., "compress": ... }

Slots de substituição

retention_days: integer number of days to retain logs, compress: boolean flag indicating whether to compress old logs

Colocados típicos

log rotation
log archiving
storage cost monitoring

Substituições comuns

Using size-based retention (max size in GB) instead of time-based
relying on external log management services

Erros comuns

Setting retention too short causing loss of diagnostic data; forgetting to apply policy to all log sources; not testing policy in staging

Similar / contraste

Log rotation (rotates logs based on size/time but may delete vs archive); Log sampling (reduces volume by sampling entries)

Interferências

Coming from sysadmin background: may confuse retention with rotation; Coming from development: may overlook compliance implications

Família do chunk

log rotation
log archiving
log sampling

Nuance

Policy must consider legal hold requirements; performance impact of deletion/compression; need to ensure atomic deletion to avoid partial logs

Efeito pragmático

Prevents uncontrolled log growth, reduces storage costs, aids compliance

Dica de memória

Keep logs fresh, not stale

Nota

When deleting or compressing old logs, ensure the operation is atomic to avoid partial files; also consider the impact of compression CPU usage on high‑throughput systems

Upgrade path

Implement tiered retention (e.g., hot/warm/cold storage) or integrate with SIEM for real-time alerting

Frequência: MediumFormulaicidade: Semi-fixedTipo de construção: policy configurationPrioridade de aquisição: Recognition firstPrioridade de output: BothTag de espaçamento: Medium-term

Examples

# Simple retention configuration dictionary retention_policy = { "retention_days": 30, "compress": True, } def should_delete(log_timestamp, now): age_days = (now - log_timestamp).days return age_days > retention_policy["retention_days"]

import logging import shutil from logging.handlers import TimedRotatingFileHandler handler = TimedRotatingFileHandler( filename="app.log", when="midnight", interval=1, backupCount=30, # keep 30 days of logs encoding="utf-8", ) # Optional compression of rotated logs handler.rotator = lambda src, dst: shutil.move(src, dst + ".gz") logger = logging.getLogger("myapp") logger.setLevel(logging.INFO) logger.addHandler(handler) logger.info("Service started")

from elasticsearch import Elasticsearch es = Elasticsearch() # Define an Index Lifecycle Management (ILM) policy for log indices policy = { "policy": { "phases": { "hot": {"actions": {"rollover": {"max_age": "7d"}}}, "warm": {"actions": {"forcemerge": {"max_num_segments": 1}}}, "delete": {"min_age": "30d", "actions": {"delete": {}}} } } } es.ilm.put_lifecycle(name="log_retention_policy", body=policy) # Apply the policy to an index template es.indices.put_template( name="logs_template", body={ "index_patterns": ["logs-*"], "settings": {"index.lifecycle.name": "log_retention_policy"} }, )