Text processing / Unicode normalization

Ensure Unicode strings have a consistent representation for reliable comparison, storage, or display.

Comparing user‑provided strings; preparing text for storage in databases; normalizing filenames or URLs.

Python web APIs, data cleaning pipelines, NLP tools, any application handling international text.

unicodedata.normalize(form, string)

unicodedata.normalize(..., ...)

form: str – one of 'NFC', 'NFD', 'NFKC', 'NFKD', string: str

• str.strip()
• str.lower()
• hashing functions
• encoding/decoding routines

• Other normalization forms: 'NFC'
• 'NFD'
• 'NFKD'
• using a variable form argument.

Assuming the function modifies the string in place; forgetting to reuse the returned value; applying it to bytes instead of str; using an invalid form name.

unicodedata.normalize with other forms (NFC, NFD, NFKD) – differs in decomposition/composition rules; or using str.encode('utf-8').decode('utf-8') for ASCII‑only cleanup.

Coming from C where strings are byte arrays: may think Unicode normalization is unnecessary; from Java where String is Unicode but no built‑in normalization, leading to missing the step.

• unicodedata.normalize
• str.encode
• str.decode
• text normalization routines

NFKC applies compatibility decomposition, which can change semantics (e.g., turning ligatures into separate characters); it is not reversible and may not be appropriate when you need to preserve original formatting.

Provides a canonical representation that guarantees reliable equality checks and safe storage or transmission of text.

Normalize to NFKC for safe string comparison.

NFKC applies compatibility decomposition, which may alter semantics (e.g., ligatures become separate characters); it is not reversible and may not be suitable when original formatting must be preserved.

Using locale‑aware libraries like PyICU for language‑specific collation, or applying custom regex to strip diacritics after normalization.