Data Leakage
Definition
Data leakage is when a model is trained on information that won’t be available at prediction time. It makes a model look accurate in validation, then fail badly once used for real decisions — a subtle, high-cost mistake.
Core Ideas
Two types
- Leaky predictors — features that include data not available when you actually make a prediction (effectively using the outcome to predict the outcome). They often correlate strongly with the target; a model that looks extremely accurate is a red flag.
- Leaky validation strategy — letting validation data influence preprocessing, e.g. fitting an imputer or scaler before
train_test_split. Validation is supposed to measure performance on unseen data; contaminating it inflates the score.
Prevention
- Leaky predictors — no universal fix; requires domain knowledge, case-by-case inspection, and common sense. Screen for features statistically correlated with the target.
- Leaky validation — exclude validation data from all fitting, including preprocessing. Use scikit-learn Pipelines so preprocessing happens inside cross-validation folds.
“Careful separation of training and validation data is a first step” — data leakage can be a multi-million-dollar error in production.
Relationships
- Machine Learning — a core validation pitfall
- Gradient Boosting / Neural Network — any model is vulnerable; suspiciously high accuracy is the tell
- Portfolio Risk Management — analogous to backtest overfitting in quant trading