Free Duplicate Image Finder for ML Datasets

TensorTurn's duplicate image finder is a free tool that scans an image dataset and groups exact duplicates and near-duplicates, using perceptual hashing for pixel-level matches and DINOv2 embeddings for images that are visually the same but not byte-identical. It also flags duplicates that straddle your train and test splits, a hidden source of inflated accuracy. Upload a zip, folder, or URL and get the duplicate clusters in minutes, on the free tier with no code.

Find exact and near-duplicate images automatically

Real image datasets are full of repeats: the same photo saved twice, a resized or re-compressed copy, a lightly cropped or filtered variant, or frames pulled from the same video. Scrolling thousands of thumbnails to spot them by eye is hopeless. The duplicate image finder does it automatically, clustering copies together so you can see how much redundancy is actually in your data.

Exact vs near-duplicate: what's the difference

• Exact duplicates: the same image content, caught by perceptual hashing even if the files were renamed or re-saved.
• Near-duplicates: images that are visually near-identical but not pixel-identical, such as resized, re-compressed, cropped, or lightly edited versions, caught with DINOv2 embeddings that compare semantic content rather than raw bytes.
• Cross-split duplicates: copies or near-copies that appear in both training and validation/test, which leak the answer and inflate your scores.

How the duplicate finder works

• Import your images as a zip/rar/7z archive or from a URL.
• The tool computes a perceptual hash for every image to catch exact and lightly-altered copies.
• It embeds each image with DINOv2 on a GPU to find semantic near-duplicates that hashing alone would miss.
• Duplicates are grouped into clusters, and any that span your train and test splits are flagged separately as leakage.

Why duplicate images hurt your model

Duplicates cause three concrete problems. First, when a duplicate lands in both train and test, the model is graded on images it already memorized, so accuracy looks better than it is. Second, over-represented images skew your effective class balance, biasing the model toward whatever is duplicated most. Third, you pay GPU time to train on the same content repeatedly for no benefit. Removing duplicates makes evaluation honest and training more efficient.

Supported dataset layouts

The finder auto-detects common layouts including ImageFolder, YOLO, COCO, and VOC, so you do not have to reshape your data first. It is part of TensorTurn's broader image health check, which in the same pass also reports blur, exposure problems, corrupt files, unusual resolutions, and likely-mislabeled images, giving you a complete picture of dataset quality.

Review the clusters, then clean

The tool shows you the duplicate clusters and cross-split conflicts so you stay in control; it does not silently delete files. You decide which copies to keep and which to remove, then re-split so duplicates no longer straddle train and test, and continue straight into training on cloud GPUs.

Find the duplicates in your image dataset free at https://www.tensorturn.com/login. No credit card, free tier included.