memory-mapped file

Data & Storage

Meaning

A memory‑mapped file maps a region of a file directly into the process’s address space, allowing the file to be accessed through normal memory operations. It eliminates the need for explicit read/write system calls and reduces data copying between kernel and user space. It is most useful when working with very large files that need random access or when performance‑critical code must treat file data as an in‑memory array.

Primary Function

File I/O

Communicative Purpose

Enables efficient random access to large files without loading the entire content into memory.

Pattern

open file → create mmap object → access data as bytearray

Função primária

File I/O

Propósito comunicativo

Enables efficient random access to large files without loading the entire content into memory.

Situações de gatilho

Data processing: analyzing multi‑gigabyte log files; Multimedia: streaming video frames from a large video file; Database: accessing on‑disk B‑tree nodes directly.

Contextos

Systems programming, high‑performance data pipelines, scientific computing, database engines, multimedia processing.

Padrão

open file → create mmap object → access data as bytearray

Colocados típicos

mmap module
os.open
file descriptor
seek
read
write
flush
close

Substituições comuns

read() into a buffer (simpler but copies data) – uses more memory and CPU
streaming I/O (lower memory usage) – may incur more system calls and less random access speed.

Erros comuns

Failing to close the mmap object – leads to file descriptor leaks; Assuming the mmap is thread‑safe without synchronization – can cause race conditions; Mapping a file larger than available address space – results in mmap failure.

Similar / contraste

Standard read/write loops – copy data on each call; Direct I/O (O_DIRECT) – bypasses cache but requires aligned buffers; Memory‑mapped I/O vs. streaming APIs – trade‑off between random access speed and memory usage.

Interferências

Coming from Java: assuming the garbage collector will automatically release the mmap – you must explicitly call close() on the mmap object.

Família do chunk

file read/write
streaming I/O
shared memory
memoryview
direct I/O

Nuance

Do not use mmap for tiny files where the overhead outweighs benefits; Mapping large files can increase virtual memory pressure and may trigger swapping if accessed heavily; On Windows, the file size must be a multiple of the allocation granularity, otherwise mapping may fail.

Efeito pragmático

Using memory‑mapped files can dramatically reduce I/O latency and simplify code that requires random access, enabling real‑time analytics on massive datasets.

Dica de memória

A memory‑mapped file is like a window into a massive book, letting you read any page instantly without turning the whole book.

Nota

On Windows, the mapping length must be a multiple of the system allocation granularity, and the file must be opened with appropriate sharing flags.

Upgrade path

After mastering mmap, move to asynchronous memory‑mapped I/O or combine mmap with OS‑specific advanced flags like MAP_POPULATE for pre‑faulting pages.

Frequência: MediumFormulaicidade: FixedTipo de construção: conceptPrioridade de aquisição: Active recallPrioridade de output: BothTag de espaçamento: Medium-term

Examples

import mmap, os with open('large_log.txt', 'r') as f: with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm: for line in iter(mm.readline, b''): process_line(line.decode())

import mmap, os # Open file for read/write with open('data.bin', 'r+b') as f: size = os.path.getsize('data.bin') with mmap.mmap(f.fileno(), size) as mm: # Modify first 4 bytes mm[0:4] = (1234).to_bytes(4, byteorder='little') mm.flush()

import csv, mmap, io with open('big_dataset.csv', 'r') as f: with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm: text = io.TextIOWrapper(io.BytesIO(mm[:]), encoding='utf-8') reader = csv.reader(text) for row in reader: handle_row(row)