Data transformation

Ensures data conforms to a canonical format, preventing inconsistencies across processing stages.

Data pipelines: ingesting CSV files with varying column orders; Machine learning: feeding feature vectors that need scaling to a common range; APIs: receiving JSON payloads with mixed date formats

ETL jobs, data cleaning scripts, machine learning preprocessing pipelines, API integration layers

raw_input → normalize → standardized_output

standardized = (value - min) / (max - min)

• scaling
• standardization
• z-score
• min-max
• data cleaning

• standardization (z-score) – centers data with unit variance
• min-max scaling – bounds data to [0
• 1]
• log transformation – handles skewed distributions

Applying min-max scaling after one-hot encoding → distorts categorical features; fitting scaler on full dataset including test set → data leakage; using integer division in Python 2 → loss of precision

standardization vs normalization; scaling vs bucketing; data cleaning vs data validation

Coming from JavaScript: assuming arrays are automatically normalized – in Python you must explicitly apply a scaling function; Coming from SQL: treating NULL as a normal value – normalization should handle missing data explicitly

• scaling
• standardization
• data cleaning
• feature engineering

Do not use normalization when absolute values carry meaning (e.g., timestamps); Min-max scaling adds negligible overhead but can be affected by outliers; For constant-valued columns, division by zero occurs unless handled

Enables reliable downstream analytics, reduces bugs caused by unexpected value ranges, and simplifies model training.

Normalization is like converting all measurements to the same unit before comparing them, just as a chef converts ingredients to grams to follow a recipe accurately.

Fit the normalizer on training data only and reuse the same parameters on validation/test data to avoid data leakage.