Text decoding / I/O

Convert byte data possibly prefixed with a UTF-8 BOM into a clean Unicode string.

File I/O: reading files that may contain a UTF-8 BOM (e.g., CSV exported from Excel); Network: processing network payloads that may contain a UTF-8 BOM; Web: handling user-uploaded files that may contain a UTF-8 BOM

Data processing scripts, file I/O operations, web API endpoints, any code that receives raw bytes and needs Unicode text.

codecs.decode(bytes_data, 'utf-8-sig')

codecs.decode(..., 'utf-8-sig')

bytes_data: bytes-like object containing UTF-8 encoded text possibly with BOM

• open with encoding='utf-8-sig'
• codecs.encode
• io.BytesIO
• pandas.read_csv

• bytes_data.decode('utf-8-sig') (if bytes_data is bytes)
• using io.TextIOWrapper with encoding='utf-8-sig'

Using 'utf-8' instead of 'utf-8-sig' leaves BOM in string; applying to already decoded str raises TypeError; forgetting to import codecs.

codecs.encode(string, 'utf-8-sig') for encoding with BOM; open(file, encoding='utf-8-sig') for automatic decoding.

Coming from Java: may forget to handle BOM when decoding bytes to string → must use 'utf-8-sig' to strip BOM; Coming from C#: may assume Encoding.UTF8.GetString strips BOM → it does not; use Encoding.UTF8.GetString and then remove BOM if present.

• codecs.decode
• codecs.encode
• bytes.decode
• str.encode

The BOM is only stripped at the start of the data; if BOM appears mid-stream it is treated as a zero-width space character. The function raises UnicodeDecodeError if data is not valid UTF-8.

Ensures clean Unicode strings without invisible BOM characters that can interfere with parsing or comparison.

Think 'decode with sig' to strip the BOM.

For bytes objects, bytes_data.decode('utf-8-sig') is equivalent and often preferred; codecs.decode also works with memoryview or bytearray.

Use open(..., encoding='utf-8-sig') for file handling, or io.TextIOWrapper for streams.