Meaning
Using numpy.memmap creates a memory-mapped array that allows reading and writing large binary files on disk without loading the entire dataset into memory, providing a NumPy-like interface for out-of-core data.
Primary Function
Memory management / Data I/O
Communicative Purpose
Efficiently handle large arrays stored on disk by mapping file regions into memory.
Pattern
import numpy as np arr = np.memmap(filename, dtype=dtype, mode=mode, shape=shape) arr[start_row:end_row, start_col:end_col] = np.zeros((num_rows, num_cols))
Core Structure
np.memmap(..., dtype=..., mode=..., shape=...)
Função primária
Memory management / Data I/O
Propósito comunicativo
Efficiently handle large arrays stored on disk by mapping file regions into memory.
Situações de gatilho
Scientific computing: processing datasets larger than available RAM; Machine learning: accessing slices of large training arrays without full load; Bioinformatics: reading genome-scale binary matrices from disk
Contextos
Scientific computing, data analysis, machine learning, high-performance computing, bioinformatics.
Padrão
import numpy as np arr = np.memmap(filename, dtype=dtype, mode=mode, shape=shape) arr[start_row:end_row, start_col:end_col] = np.zeros((num_rows, num_cols))
Estrutura central
np.memmap(..., dtype=..., mode=..., shape=...)
Slots de substituição
filename: str or path-like; dtype: numpy dtype (e.g., 'float32', np.float64); mode: file mode str ('r', 'r+', 'w+', 'c'); shape: tuple of ints for array dimensions; slice start/end ints for row and column ranges; zeros shape ints for fill array dimensions
Colocados típicos
- numpy.ndarray
- slicing
- np.flush
- np.del
- h5py
- dask.array
Substituições comuns
- np.load with mmap_mode
- scipy.io.mmread
- memoryview
- manual file read/write
Erros comuns
Forgetting to call flush() or del before program exit → changes may not be written to disk; Specifying mode='r' when writing → raises TypeError on assignment; Shape mismatch between memmap and assigned array → ValueError due to broadcasting failure; Assuming data is fully loaded into RAM → unexpected swapping or OOM on very large files
Similar / contraste
numpy.load(mmap_mode='r') vs memmap; h5py Dataset for chunked storage; pandas read_csv with chunksize
Interferências
Coming from MATLAB: expecting memmap writes to auto-flush like MATLAB's memmapfile → must explicitly call flush() or del; Coming from C: assuming manual mmap/munmap calls are needed → numpy.memmap handles OS-level mapping internally
Família do chunk
- numpy memmap
- memory-mapped file
- out-of-core computation
Nuance
Changes are persisted only after flush() or del; performance depends on OS paging; file must exist for write modes; not ideal for tiny arrays due to overhead
Efeito pragmático
Enables out-of-core processing of large datasets, reducing memory usage while retaining a NumPy-like interface
Dica de memória
Think of a file-backed numpy array you can slice like normal memory
Nota
Remember to flush or delete the memmap object to persist changes to disk.
Upgrade path
Use dask.array or zarr for chunked, parallel out-of-core workflows
Log in to save chunks.