tensorturnWhat Is Data Leakage in Machine Learning?
Data leakage in machine learning is when information that would not be available at prediction time leaks into training, making a model look far more accurate in testing than it will ever be in production. It typically shows up as suspiciously high validation scores that collapse on real, unseen data. Leakage is one of the most common and expensive mistakes in applied ML, and it can trap even experienced practitioners.
The main types of data leakage
Leakage comes in a few recognizable forms. The unifying theme is that the model gets a peek at information it will not actually have when it makes real predictions.
- Train/test leakage: the same or near-duplicate rows appear in both the training and test sets, so the model is effectively graded on examples it already memorized.
- Target leakage: a feature is derived from, or is a proxy for, the value you are trying to predict (for example, including a 'payment received' flag when predicting whether an invoice will be paid).
- Temporal leakage: future information is used to predict the past, which happens when time-ordered data is shuffled randomly instead of split by time.
- Preprocessing leakage: scalers, encoders, or imputers are fit on the entire dataset before splitting, letting statistics from the test set influence training.
Common causes of data leakage
Most leakage traces back to doing something to the whole dataset that should have been done only on the training portion. Fitting a StandardScaler or computing target-mean encodings before the split, deduplicating carelessly, or engineering a feature that quietly encodes the label are the usual culprits. Aggregated features such as rolling averages or group means are especially risky, because they can pull in values from rows that belong to the test set.
How to detect and prevent leakage
The strongest signal is a model that scores near-perfectly in validation but performs poorly in production, or a single feature with implausibly high importance. Prevention is mostly discipline about the order of operations.
- Split first, then fit every preprocessing step on the training set only and apply it to the test set.
- Wrap preprocessing and the model in a single pipeline so transforms are refit correctly inside cross-validation folds.
- De-duplicate and check for near-duplicates across splits before training.
- For time series, split by time and never shuffle across the cutoff date.
- Audit high-importance features and ask whether each would truly be known at prediction time.
How TensorTurn catches leakage for you
TensorTurn runs an automated dataset health check on every tabular upload that specifically probes for train/test leakage, duplicate and near-duplicate rows, and suspicious feature-to-target correlations, then returns a quality score and a preprocessing playbook. For image datasets it flags cross-split leakage using perceptual hashing so the same picture never lands in both train and test. Many of the leakage traps above are surfaced before you ever kick off a training run.
Frequently asked questions
What is the difference between data leakage and overfitting?
Overfitting is a model memorizing noise in an otherwise clean training set; it still generalizes poorly for legitimate reasons. Data leakage is a flaw in how the data was prepared, where test information contaminates training and produces scores that are simply not real. Leakage often looks like great performance, whereas overfitting usually shows a gap between train and validation scores.
Does removing duplicate rows fix data leakage?
It fixes one common form, train/test leakage from duplicates, but not the others. Target leakage and temporal leakage can exist in a dataset with zero duplicate rows, so deduplication is necessary but not sufficient.
Can cross-validation cause data leakage?
Yes, if preprocessing is done before the folds are created. Any scaling, encoding, or feature selection fit on the full dataset leaks information into every fold. The fix is to fit those steps inside each fold, which pipelines handle automatically.
How much does data leakage inflate accuracy?
It varies, but leakage can push reported accuracy from realistic values into the high 90s or even 100 percent. The tell is that the inflated score does not survive contact with genuinely unseen data.