Fairness under Label Bias in Segmentation

Label bias in segmentation

Two groups, annotated in two different ways

Group g_c gets careful, tight boundaries.
Group g_b gets masks that are always a bit too big.
The objects are equally hard. Only the annotation style differs, and it follows the group. That is label bias.

Unlike a wrong class label, a wrong mask has a shape. The mask can be too small (deficit), too big (excess), or irregular, with both at once.

Under-segmentation

The mask is shrunk. Missed object pixels in red.

Over-segmentation

The mask is grown. Extra background pixels in orange.

Irregular boundary

The boundary wobbles. Some pixels missed, some added.

Erosion radius sweep on a CelebAMask-HQ face mask — CelebAMask-HQ: we shrink the face mask by a growing radius to create controlled under-segmentation.

PhC-U373 fine vs coarse annotations — PhC-U373: two real human annotations of the same cell, one fine (green) and one coarse and too big (red).

The bent ruler

Scores on biased labels do not tell you the real performance

Normally we score a model against the labels we have. Here those labels are the biased ones.
The real gap between the groups, which only clean labels can show, can be much larger, much smaller, or even point the other way.

Scoring against biased labels hides the bias instead of showing it. We need a way to check the labels that does not need clean labels.

Confident Learning as an audit

Check the training labels using the training set itself

No clean labels needed. Everything here uses only the biased training set and a model trained on it.
Cross-fitting. Split the training set into K folds. Each fold is predicted by a model that never trained on it, so the prediction is an independent second opinion.
Confident joint Q. For pixels where the model is confident, count how often the given label and the predicted label agree or disagree in a 2×2 matrix. The off-diagonal cells are the suspected label errors.
Three numbers. Adding excess and deficit gives the error rate (LER). Subtracting them gives the direction (DR). Comparing groups gives the ratio (RR).

CL decomposition of annotation disagreement — The given mask is compared with the confident prediction, the disagreement is split into excess and deficit, and the three metrics LER, DR and RR are built from them.

Audit results on three datasets — Top: observed versus true performance on all three datasets. Bottom: the audit, computed from the given masks only. LER rises in g_b as bias grows, RR moves away from the unbiased baseline, and DR gives the direction: negative for shrunk masks (CelebAMask-HQ), positive for too-big masks (PhC-U373).

What it gives

How much error (LER), in which direction (DR), and in which group (RR).
The thresholds adapt to the model's calibration, so temperature scaling does not change the result.
We checked the audit afterwards against clean masks, and it matches.

Limits

It measures disagreement, not truth. When almost all labels are biased (β = 100 %), the model learns the biased style too and the audit reports less than there is.
It needs a reasonably good segmentation model.
It cannot tell you which style is the correct one. That remains a domain decision.

Subgroup separability

Biased labels teach the network to tell the groups apart

To reproduce two annotation styles with one decoder, the network has to know which group an image belongs to. So it learns that.
In the encoder's feature space, g_c and g_b move further apart as bias grows: the silhouette score rises 3.9× and the Fisher ratio 8.8× from no bias to full bias.

Decision boundary shift and PCA/t-SNE/LDA of encoder features — Left: biased labels move the learned decision boundary. Right: PCA, t-SNE and LDA of CelebAMask-HQ encoder features. Red is g_b, blue is g_c.

Separability is a symptom, not the problem itself. Removing group information from the features (adversarial training, MMD, CORAL) does not close the real gap, because the labels are still wrong. See our follow-up preprint: Is invariance all you need for algorithmic fairness? Removing demographic information can create new bias. In this paper we use the group signal instead of removing it.

Mitigation

Learn every annotation style, then predict in the clean one

Subgroup-conditioned decoder (SCD). The decoder receives the group as input through FiLM layers, so each annotation style gets its own setting. At test time every image is decoded with the clean group's setting.
Auto-SCD. When we do not know which group is clean, we watch the training loss per group during warm-up. Clean labels are easier to fit, so the group with the lowest loss becomes the reference.

Citation

Cite the preprint

Preprint: arXiv:2605.06891. Code, splits, configs and seeds: github.com/ADE-17/label-bias-segmentation.

@misc{parikh2026fairnesslabelbiasimage,
      title={Towards Fairness under Label Bias in Image Segmentation: Impact, Measurement and Mitigation},
      author={Aditya Parikh and Stella Frank and Sneha Das and Aasa Feragen},
      year={2026},
      eprint={2605.06891},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2605.06891},
}

Towards Fairness under Label Bias in Image Segmentation

What this paper does, in three points