Finding Wrong Labels with Geometry, Not a Model

Almost every real dataset has wrong labels in it. A crowd worker mis-clicks, an auto-labeller guesses badly, a scraped caption is nonsense. A few percent of a dataset being mislabelled is normal; a few tens of percent is common in crowdsourced or weakly-supervised data. Those bad labels quietly drag down every model you train on them, so it is worth finding and removing them.

geosift is a small Python library that finds them. What makes it interesting is how it decides a label is wrong: it never trains a model to do the judging. It looks only at the geometry of the data, and that turns out to matter most exactly where the usual tools break.

The auditor problem

The standard way to catch bad labels is a method called Confident Learning (the cleanlab library). The recipe is sensible: train a classifier on your labelled data, then look for places where the trained model is confident the label is wrong. If the model is sure a picture is a dog but the label says cat, that label is suspicious.

There is a catch, and it is circular. The model doing the checking was itself trained on the same possibly-wrong labels. When only a few labels are bad, the model mostly learns the true pattern and its complaints are trustworthy. But as the noise climbs, the model increasingly learns the mistakes themselves, and its "confident disagreements" start reflecting its own corruption. The memorable line from the paper: the auditor has read the same record it is auditing.

So the tool you most need under heavy noise is the tool that degrades fastest under heavy noise.

Birds of a feather

geosift sidesteps the circle by throwing the model away. The idea rests on one everyday observation: things of the same kind tend to sit near each other. If you place every data point in "feature space" (dogs near dogs, cats near cats), then to guess a point's true class you can simply ask what are its neighbours? No training, no model to corrupt. A wrong label is then just a point whose neighbourhood disagrees with it: a lone "cat" sitting deep in a crowd of dogs.

Crucially, the neighbourhoods themselves are built from positions only, never from the labels. Labels cannot corrupt something they were never used to make. That is the whole trick, and it is why the signal survives when a trained model would not.

This isn't a brand-new idea. An earlier method, SimiFeat, already does training-free detection this way. geosift's actual contribution is narrower and more interesting: it changes how big a neighbourhood you look at, and it maps out exactly when that helps.

Two ways to look at a neighbourhood

There are two ways to read a point's surroundings, and they trade off against each other:

• Local (SimiFeat's way): look at the handful of points nearest to your target, maybe its 10 nearest neighbours. Sharp and detailed, but it is a vote of only 10 possibly-wrong labels, so it is jumpy — high variance.
• Pooled (geosift's way, the "forest of means"): group the data into broad regions of hundreds of points each, and read the class proportions of the whole region. Coarser, but the vote is now over hundreds of labels, so it barely wobbles — low variance.

The prediction that falls out of this: local wins when noise is light (detail matters, and 10 mostly-correct neighbours are plenty), while pooled wins when noise is heavy or clustered near class boundaries (a small neighbourhood is easily swamped with errors, but a region of hundreds is barely moved). The demo below lets you feel exactly that.

Demo 1: watch the small neighbourhood get fooled

Two classes, pink and blue, running live in your browser. Drag the noise slider to flip more and more labels at random. Click any point to inspect it. For the point you pick we draw two lenses: its local neighbourhood (the 10 nearest points) and its pooled region (its whole cluster). Each lens votes on what the point's label should be. Watch the local vote start flipping at high noise while the pooled vote holds steady.

Try this: pick the highlighted pink point near the boundary and slide the noise up past 40%. The green local vote — only 10 labels — swings around and eventually calls a genuinely-pink point blue. The purple pooled vote, counting hundreds of labels, keeps pointing at pink long after. That is the whole argument in one picture: a small sample is easy to poison; a big one is not.

(Two honest notes. This toy uses random symmetric noise so you can see the mechanism; the real advantage is largest under boundary-concentrated noise, which is messier to animate. And at low noise you will see the local lens is perfectly reliable too — that is exactly the regime where it is actually the better choice.)

Demo 2: the crossover, with the real numbers

So who actually wins? The paper measures both detectors on ten datasets across three kinds of label noise, scoring each by F1 against the known list of flipped labels (higher is better — it balances catching real errors against false accusations). The chart below shows the measured F1 for geosift (pooled) versus SimiFeat (local). Toggle between 20% and 40% noise and watch the bars swap places.

At 20% noise the local method wins across the board — its finer resolution pays off and there aren't enough errors to swamp a small neighbourhood. At 40% the picture flips: on asymmetric (structured) and boundary noise, the pooled forest pulls ahead, because those are exactly the conditions that poison a small neighbourhood while leaving a big region intact. ("Boundary" noise is the realistic hard case — the errors pile up right where two classes meet.)

Averaged over everything, SimiFeat is still the marginally stronger single method. geosift is not a replacement; it is a specialist for the heavy-noise regime, and it knows it.

Does it hold on real mistakes?

Synthetic noise is a fair test bench, but the real prize is genuine human error. CIFAR-10N is a version of the classic CIFAR-10 image set re-labelled by real crowd workers; its "worst" split carries about 40% real mislabels — squarely in the regime where pooling should win. Running every detector on CLIP image embeddings (where similar images naturally cluster together), the head-to-head F1:

On genuine 40% human noise the pooled forest is the best detector — beating both the local method and the model-based baseline. The synthetic crossover reproduces on real mistakes. (One caveat worth stating: geometry needs a representation where classes actually cluster. On raw pixels, every method here is near chance — the CLIP embedding is doing real work.)

A detail that surprised the author: the detector matters as much as the score

There are really two steps: an estimator produces a suspiciousness score for each point, and a detector turns those scores into actual accusations. It turns out the second step is a first-order choice on its own. cleanlab's native pruning rule was tuned for a trained model's smooth probabilities and fits geometry's spikier scores badly; a dead-simple voting rule — "flag a point if its neighbourhood's favourite class isn't its label" — beats it substantially for geometric scores. The practical upshot: any fair "method A vs method B" comparison has to hold the detector fixed, which is why every number above uses the same voting rule for both sides.

What you actually get

geosift ships as a small, numpy-only library. The core object cleans a dataset in three lines:

from geosift import GeoCleaner

gc = GeoCleaner(clusters_per_class=3, n_estimators=25, random_state=0).fit(X, y)

issues            = gc.label_issues(X, y)         # boolean mask of suspected mislabels
X_clean, y_clean  = gc.clean(X, y)                # the data with issues dropped
quality           = gc.label_quality_scores(X, y) # 0..1, higher = more trustworthy
hardness          = gc.hardness(X_new)            # label-free ambiguity, even at inference

Beyond detection, that last line is a quiet bonus. Because the geometry needs no labels, geosift can hand you a per-point hardness score — how ambiguous a point is — computed on brand new, unlabelled inputs at prediction time. It is available before you have trained anything, it is reusable across every model you later build, and it doesn't suffer the confidence-collapse a heavily-noise-trained model does. Handy for triage: send the hardest cases to a human.

The three defaults worth knowing: clusters_per_class=3 is the robust sweet spot (one cluster per class can't capture sub-groups; past three the gains flatten), the clustering is bagged over random subsets of both rows and features to stabilise it, and everything label-dependent is computed out-of-fold so a point never helps judge itself.

When to reach for it (and when not to)

• Use it when you suspect substantial label noise (tens of percent), or noise concentrated near class boundaries, or you just want a second, model-independent opinion on which labels to trust.
• Don't bother at near-clean labels, where a trained model is just as good and SimiFeat's local view is a touch sharper. geosift is honest about being a specialist.
• Either way it needs a feature space where the classes cluster. If your raw features don't separate the classes, embed them first (as with CLIP above).

The code, a full cookbook (GUIDE.md), and the complete experimental writeup live on GitHub (hoofay/geosift).

Citation

If you build on geosift or its findings, the paper is archived on Zenodo (doi:10.5281/zenodo.21627002):

@misc{hough2026geosift,
  title     = {Model-independent label-noise detection by feature geometry: pooled vs. local estimators},
  author    = {Hough, Daniel},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.21627002},
  url       = {https://doi.org/10.5281/zenodo.21627002}
}