log rotation

Observability

Meaning

Log rotation is a technique that automatically renames or archives a log file when it reaches a certain size or age, and starts writing to a fresh file. It solves the problem of unbounded log growth that can exhaust disk space and make log analysis difficult. It is typically used when an application produces continuous logging output over long periods.

Primary Function

Log management

Communicative Purpose

Prevents uncontrolled log file growth by rotating logs based on size or time thresholds.

Pattern

configure logger → set rotation handler (size or time) → define backup count → logger writes → handler rotates file automatically

Função primária

Log management

Propósito comunicativo

Prevents uncontrolled log file growth by rotating logs based on size or time thresholds.

Situações de gatilho

Web server: daily log files exceed 100 MB, causing disk pressure; Background worker: log file reaches 10 MB during continuous processing; Embedded device: limited storage requires periodic log truncation.

Contextos

Server-side applications, background services, containerized microservices, embedded systems, any long-running process that writes to files.

Padrão

configure logger → set rotation handler (size or time) → define backup count → logger writes → handler rotates file automatically

Colocados típicos

RotatingFileHandler
TimedRotatingFileHandler
backupCount
maxBytes
when='midnight'
encoding

Substituições comuns

use external logrotate utility → less code but requires separate configuration
use manual file rename in code → more control but higher error risk

Erros comuns

Setting maxBytes too low → rotates too frequently, fragmenting logs.; Omitting backupCount → old logs accumulate and fill disk.; Using a non‑thread‑safe handler in multithreaded code → occasional loss of log entries.; Rotating logs without flushing handlers → last entries may be lost in the old file.

Similar / contraste

Log rotation vs log archiving: rotation swaps active file, archiving moves completed files to long‑term storage.; Size‑based rotation vs time‑based rotation: one triggers on byte count, the other on elapsed time.; Python's RotatingFileHandler vs external logrotate: built‑in vs OS‑level tool.

Interferências

Coming from Windows batch scripts: assume you can delete an open log file – in Python the file handle must be closed before rotation, otherwise rotation may fail.; Coming from Java's Log4j: expect automatic compression flag – Python's handlers need explicit handler for compression.

Família do chunk

log handling
log compression
log archiving
log retention policies

Nuance

1. Do not use rotation for extremely low‑volume logs where a single file suffices, as it adds unnecessary complexity.; 2. Rotation introduces a small I/O overhead each time a file is renamed; on high‑throughput systems consider asynchronous logging.; 3. When maxBytes is an exact multiple of log entry size, the last entry may be split across files, which can affect parsers expecting whole lines.

Efeito pragmático

Ensures that applications continue to log without running out of disk space, simplifies log retention policies, and enables automated cleanup of old logs.

Dica de memória

Think of log rotation like a revolving door: when the room fills up, the door swings and a fresh room opens while the old one is sent out.

Nota

Python's logging module provides both RotatingFileHandler (size‑based) and TimedRotatingFileHandler (time‑based); choose based on the most relevant metric for your deployment.

Upgrade path

After mastering built‑in rotation, adopt external log management solutions such as logrotate or centralized logging platforms (e.g., ELK, Fluentd).

Frequência: HighFormulaicidade: FlexibleTipo de construção: conceptPrioridade de aquisição: Active recallPrioridade de output: BothTag de espaçamento: Medium-term

Examples

import logging from logging.handlers import RotatingFileHandler logger = logging.getLogger("app") logger.setLevel(logging.INFO) handler = RotatingFileHandler( "app.log", maxBytes=5 * 1024 * 1024, # 5 MB backupCount=3, encoding="utf-8" ) logger.addHandler(handler) for i in range(10000): logger.info("Event number %d", i)

import logging from logging.handlers import TimedRotatingFileHandler logger = logging.getLogger("service") logger.setLevel(logging.DEBUG) handler = TimedRotatingFileHandler( "service.log", when="midnight", interval=1, backupCount=7, encoding="utf-8" ) logger.addHandler(handler) logger.debug("Service started")

import os import logging LOG_PATH = "worker.log" MAX_SIZE = 2 * 1024 * 1024 # 2 MB logger = logging.getLogger("worker") logger.setLevel(logging.INFO) handler = logging.FileHandler(LOG_PATH, encoding="utf-8") logger.addHandler(handler) def rotate_if_needed(): if os.path.getsize(LOG_PATH) >= MAX_SIZE: logger.handlers[0].close() os.rename(LOG_PATH, f"{LOG_PATH}.1") open(LOG_PATH, "w").close() logger.addHandler(logging.FileHandler(LOG_PATH, encoding="utf-8")) for i in range(5000): logger.info("Task %d completed", i) rotate_if_needed()