07 — Gender bias in job title matching, measured

The problem in one example

In Spanish, a lawyer is abogado if male and abogada if female. In German, an engineer is Ingenieur or Ingenieurin. In French, a farmer is agriculteur or agricultrice. These are the same occupation. A job title matcher that treats them differently is broken in a way that has direct real-world consequences: if a recruiter's search for "abogado" surfaces different or fewer candidates than "abogada", the system is filtering people by grammatical gender while appearing to filter by profession.

English mostly hides this problem, which is exactly why it gets missed. Most job-title benchmarks are English-first.

What this notebook measures

Three separate questions, because "is it biased?" is too vague to answer:

1. Symmetry — does searching with the masculine form return the same ranked results as the feminine form? Measured with RBO (explained and implemented below).
2. Retrieval parity — is the model equally good at finding the correct occupation for a feminine title as for a masculine one?
3. Stereotype association — do feminine forms drift toward occupations stereotypically coded as female?

Then one mitigation is tried and measured, rather than merely recommended.

Where the test data comes from — and why it isn't invented

ESCO publishes many occupation labels in explicit dual form, e.g. the German Medizinisch-technischer Laborassistent/Medizinisch-technische Laborassistentin. That gives us real, official, human-curated gendered pairs for the same occupation URI — we are not fabricating word pairs or relying on a hand-written list of guesses.

import os
import re
import warnings

warnings.filterwarnings("ignore")
os.environ["TOKENIZERS_PARALLELISM"] = "false"

import numpy as np
import polars as pl
import matplotlib.pyplot as plt

import common

RESULTS = common.RESULTS_DIR
np.random.seed(0)

esco_occ, esco_titles = common.load_esco()
print(f"ESCO titles loaded: {esco_titles.height:,} rows")

ESCO titles loaded: 213,621 rows

1. Extracting real gendered pairs

German is the cleanest source: ESCO's preferred label often literally contains both forms separated by /. We split on that and keep pairs where the feminine side ends in the -in suffix, which is the reliable German feminine occupational marker.

Spanish / Italian / French encode gender morphologically instead, so we pair up two different labels belonging to the same ESCO occupation URI that differ only by a known masculine→feminine ending transformation. Requiring both forms to carry the same URI is what stops us pairing unrelated words that happen to look similar.

def german_pairs(esco_titles):
    """German ESCO labels of the form 'Masculine.../Feminine...in'."""
    de = esco_titles.filter((pl.col("language") == "de") & pl.col("title").str.contains("/"))
    out = []
    for uri, title in zip(de["uri"].to_list(), de["title"].to_list()):
        parts = [p.strip() for p in title.split("/")]
        if len(parts) != 2:
            continue
        masc, fem = parts
        # The feminine form must end in -in (or -innen) and must not be identical.
        if not re.search(r"in(nen)?$", fem) or masc == fem or len(masc) < 4:
            continue
        out.append({"language": "de", "uri": uri, "masculine": masc, "feminine": fem})
    return out


# Masculine -> feminine ending rules. Deliberately conservative: a small set of
# high-confidence transformations, applied only when BOTH resulting strings exist
# as real ESCO labels for the SAME occupation.
ROMANCE_RULES = {
    "es": [("o", "a"), ("or", "ora"), ("dor", "dora"), ("ero", "era")],
    "it": [("o", "a"), ("ore", "rice"), ("iere", "iera")],
    "fr": [("eur", "euse"), ("teur", "trice"), ("ier", "ière"), ("é", "ée")],
}


def romance_pairs(esco_titles, lang):
    """Pair two ESCO labels of the same occupation that differ only by a known
    masculine->feminine ending."""
    sub = esco_titles.filter(pl.col("language") == lang)
    by_uri = {}
    for uri, title in zip(sub["uri"].to_list(), sub["title"].to_list()):
        by_uri.setdefault(uri, set()).add(title.strip())

    out, seen = [], set()
    for uri, titles in by_uri.items():
        for masc in titles:
            for suf_m, suf_f in ROMANCE_RULES[lang]:
                if not masc.endswith(suf_m):
                    continue
                fem = masc[: -len(suf_m)] + suf_f
                if fem in titles and fem != masc and (uri, masc, fem) not in seen:
                    seen.add((uri, masc, fem))
                    out.append({"language": lang, "uri": uri, "masculine": masc, "feminine": fem})
    return out


pairs = german_pairs(esco_titles)
for lang in ("es", "it", "fr"):
    pairs += romance_pairs(esco_titles, lang)

pairs_df = pl.DataFrame(pairs).unique(subset=["language", "masculine", "feminine"]).sort(["language", "masculine"])
print(f"Extracted {pairs_df.height:,} real gendered title pairs from ESCO\n")
print(pairs_df.group_by("language").len().sort("language"))
print("\nExamples:")
for lang in ("de", "es", "it", "fr"):
    sub = pairs_df.filter(pl.col("language") == lang)
    if sub.height:
        for r in sub.head(3).iter_rows(named=True):
            print(f"  [{lang}] {r['masculine']}  <->  {r['feminine']}")

Extracted 2,181 real gendered title pairs from ESCO

shape: (4, 2)
┌──────────┬──────┐
│ language ┆ len  │
│ ---      ┆ ---  │
│ str      ┆ u32  │
╞══════════╪══════╡
│ de       ┆ 1238 │
│ es       ┆ 285  │
│ fr       ┆ 232  │
│ it       ┆ 426  │
└──────────┴──────┘

Examples:
  [de] 3D-Artist  <->  3D-Artistin
  [de] 3D-Druck-Techniker  <->  3D-Druck-Technikerin
  [de] AHS-Lehrer  <->  in
  [es] abogado  <->  abogada
  [es] acomodador  <->  acomodadora
  [es] acompañante remunerado  <->  acompañante remunerada
  [it] Intervistatore  <->  Intervistatrice
  [it] accompagnatore  <->  accompagnatrice
  [it] acconciatore  <->  acconciatrice
  [fr] abatteur  <->  abatteuse
  [fr] accompagnateur  <->  accompagnatrice
  [fr] accrocheur  <->  accrocheuse

2. RBO — Rank-Biased Overlap, explained and implemented

To ask "did the masculine and feminine queries return the same thing?" we need to compare two ranked lists. Plain set overlap is the wrong tool: it treats a disagreement at position 1 the same as one at position 50, when the top of the list is what anyone actually sees.

RBO compares two ranked lists with more weight on the top. In words:

1. Walk down both lists together, one position at a time.
2. At each depth d, compute the agreement: what fraction of the top-d items appear in both lists.
3. Take a weighted average of those agreements, where deeper positions get geometrically less weight.

The weighting is controlled by p, between 0 and 1:

• small p (e.g. 0.5) → almost all the weight on the very top few results
• large p (e.g. 0.98) → weight spread deep down the list

RBO ranges from 0 (no overlap at all) to 1 (identical rankings). For a perfectly fair system, masculine and feminine forms of the same occupation should give RBO = 1.0.

def rbo(list1, list2, p=0.9):
    """Rank-Biased Overlap between two ranked lists.

    Implements the finite-depth formulation:
        RBO = (1 - p) * sum_{d=1..k} p^(d-1) * A_d
    where A_d is the size of the intersection of the two top-d prefixes, divided
    by d. Because we stop at finite depth k rather than extrapolating to infinity,
    this is a *lower bound* on the full RBO — which is the conservative direction
    for a fairness audit (it cannot make a biased system look fair).
    """
    k = min(len(list1), len(list2))
    if k == 0:
        return 0.0
    s1, s2, total = set(), set(), 0.0
    for d in range(1, k + 1):
        s1.add(list1[d - 1])
        s2.add(list2[d - 1])
        total += (p ** (d - 1)) * (len(s1 & s2) / d)
    return (1 - p) * total


# Validate the implementation on cases where the answer is known by inspection.
identical = rbo(["a", "b", "c", "d"], ["a", "b", "c", "d"], p=0.9)
disjoint = rbo(["a", "b", "c", "d"], ["w", "x", "y", "z"], p=0.9)
swapped_top = rbo(["a", "b", "c", "d"], ["b", "a", "c", "d"], p=0.9)
swapped_deep = rbo(["a", "b", "c", "d"], ["a", "b", "d", "c"], p=0.9)

print(f"identical lists            RBO = {identical:.4f}   (should be close to 1)")
print(f"completely disjoint lists  RBO = {disjoint:.4f}   (should be 0)")
print(f"top two swapped            RBO = {swapped_top:.4f}")
print(f"positions 3&4 swapped      RBO = {swapped_deep:.4f}")
print(f"\nSwapping at the TOP hurts more than swapping DEEPER: {swapped_top:.4f} < {swapped_deep:.4f}"
      f"  -> {swapped_top < swapped_deep}")
print("That ordering is the entire property we want from a rank-aware metric.")

identical lists            RBO = 0.3439   (should be close to 1)
completely disjoint lists  RBO = 0.0000   (should be 0)
top two swapped            RBO = 0.2439
positions 3&4 swapped      RBO = 0.3169

Swapping at the TOP hurts more than swapping DEEPER: 0.2439 < 0.3169  -> True
That ordering is the entire property we want from a rank-aware metric.

Note the finite-depth caveat above: with p=0.9 and only 4 items, even identical lists score below 1.0 because the un-examined tail is treated as unknown. Since we compare every system at the same depth with the same p, the comparison between systems remains fair — but an absolute RBO should not be read as a percentage.

3. Test 1 — Symmetry: do masculine and feminine queries return the same results?

For each gendered pair, we search the same corpus with the masculine form and then the feminine form, and compare the two ranked result lists using RBO.

The corpus is the full set of ESCO preferred labels in that language — a realistic "find the matching occupation" index.

from sentence_transformers import SentenceTransformer

MODELS = [
    ("MiniLM (repo baseline)", "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"),
    ("JobBERT-v3", "TechWolf/JobBERT-v3"),
]
TOP_K = 20
RBO_P = 0.9

pref = esco_titles.filter(pl.col("label_type") == "preferred")
symmetry_rows, per_pair_records = [], []

for model_name, hf_id in MODELS:
    model = SentenceTransformer(hf_id, device="cpu")
    for lang in ("de", "es", "it", "fr"):
        sub = pairs_df.filter(pl.col("language") == lang)
        if sub.height < 30:
            continue
        corpus = (pref.filter(pl.col("language") == lang)
                  .group_by("uri").agg(pl.col("title").first()))
        c_titles = corpus["title"].to_list()
        c_uris = corpus["uri"].to_list()
        c_emb = common.encode_texts(model, c_titles, show_progress_bar=False)

        m_emb = common.encode_texts(model, sub["masculine"].to_list(), show_progress_bar=False)
        f_emb = common.encode_texts(model, sub["feminine"].to_list(), show_progress_bar=False)
        m_idx, _ = common.batched_topk(m_emb, c_emb, k=TOP_K)
        f_idx, _ = common.batched_topk(f_emb, c_emb, k=TOP_K)

        scores = []
        for i in range(sub.height):
            r = rbo(list(m_idx[i]), list(f_idx[i]), p=RBO_P)
            scores.append(r)
            per_pair_records.append({
                "model": model_name, "language": lang,
                "masculine": sub["masculine"][i], "feminine": sub["feminine"][i],
                "rbo": round(r, 4),
                "same_top1": bool(m_idx[i][0] == f_idx[i][0]),
            })
        same_top1 = float(np.mean([m_idx[i][0] == f_idx[i][0] for i in range(sub.height)]))
        symmetry_rows.append({
            "model": model_name, "language": lang, "n_pairs": sub.height,
            f"mean_RBO@{TOP_K}": round(float(np.mean(scores)), 4),
            "identical_top1_rate": round(same_top1, 4),
        })
        print(f"  {model_name:24s} {lang}: n={sub.height:4d}  mean RBO={np.mean(scores):.4f}  "
              f"identical top-1={same_top1:.1%}")
    del model

symmetry_df = pl.DataFrame(symmetry_rows).sort(["model", "language"])
per_pair_df = pl.DataFrame(per_pair_records)
symmetry_df

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

  MiniLM (repo baseline)   de: n=1238  mean RBO=0.6555  identical top-1=82.2%

  MiniLM (repo baseline)   es: n= 285  mean RBO=0.6431  identical top-1=72.3%

  MiniLM (repo baseline)   it: n= 426  mean RBO=0.6553  identical top-1=73.0%

  MiniLM (repo baseline)   fr: n= 232  mean RBO=0.5393  identical top-1=50.0%

  JobBERT-v3               de: n=1238  mean RBO=0.7567  identical top-1=94.9%

  JobBERT-v3               es: n= 285  mean RBO=0.6731  identical top-1=84.9%

  JobBERT-v3               it: n= 426  mean RBO=0.6606  identical top-1=79.8%

  JobBERT-v3               fr: n= 232  mean RBO=0.5636  identical top-1=75.9%

shape: (8, 5)

model	language	n_pairs	mean_RBO@20	identical_top1_rate
str	str	i64	f64	f64
"JobBERT-v3"	"de"	1238	0.7567	0.9491
"JobBERT-v3"	"es"	285	0.6731	0.8491
"JobBERT-v3"	"fr"	232	0.5636	0.7586
"JobBERT-v3"	"it"	426	0.6606	0.7981
"MiniLM (repo baseline)"	"de"	1238	0.6555	0.8223
"MiniLM (repo baseline)"	"es"	285	0.6431	0.7228
"MiniLM (repo baseline)"	"fr"	232	0.5393	0.5
"MiniLM (repo baseline)"	"it"	426	0.6553	0.73

How to read identical_top1_rate: the share of occupations where searching the masculine and feminine form returned the same best match. This is the most concrete, least statistical way to state the problem: when it is below 1.0, the system is literally giving different top answers depending on the grammatical gender of an otherwise identical query.

A reference point for RBO: two identical 20-item lists score rbo(x, x, p=0.9) at this depth, computed below, so compare the measured values against that ceiling rather than against 1.0.

ceiling = rbo(list(range(TOP_K)), list(range(TOP_K)), p=RBO_P)
print(f"RBO ceiling at depth {TOP_K}, p={RBO_P} (two identical lists): {ceiling:.4f}")
print("Measured values above should be compared against this, not against 1.0.\n")

worst = per_pair_df.sort("rbo").head(8)
print("Pairs where the two genders diverge MOST:")
for r in worst.iter_rows(named=True):
    print(f"  [{r['language']}] {r['masculine']!r} vs {r['feminine']!r}  RBO={r['rbo']:.3f}  same top-1: {r['same_top1']}")

RBO ceiling at depth 20, p=0.9 (two identical lists): 0.8784
Measured values above should be compared against this, not against 1.0.

Pairs where the two genders diverge MOST:
  [de] 'AHS-Lehrer' vs 'in'  RBO=0.000  same top-1: False
  [de] 'Arbeitsanalyst' vs 'in'  RBO=0.000  same top-1: False
  [de] 'Artistiklehrer' vs 'in'  RBO=0.000  same top-1: False
  [de] 'Ausbildungsberater' vs 'in'  RBO=0.000  same top-1: False
  [de] 'Berufs- und Bildungsberater' vs 'in'  RBO=0.000  same top-1: False
  [de] 'Berufsschullehrer' vs 'in'  RBO=0.000  same top-1: False
  [de] 'Business-Intelligence-Manager' vs 'in'  RBO=0.000  same top-1: False
  [de] 'Customer Relations Manager' vs 'in'  RBO=0.000  same top-1: False

4. Test 2 — Retrieval parity: is the model equally accurate for each gender?

Symmetry says the two queries disagree; it does not say which one is right. Here we ask the sharper question: querying with each form, does the model retrieve the correct occupation (the URI the pair came from)? A gap between the masculine and feminine accuracy is a direct, interpretable fairness harm.

parity_rows = []
for model_name, hf_id in MODELS:
    model = SentenceTransformer(hf_id, device="cpu")
    for lang in ("de", "es", "it", "fr"):
        sub = pairs_df.filter(pl.col("language") == lang)
        if sub.height < 30:
            continue
        corpus = pref.filter(pl.col("language") == lang).group_by("uri").agg(pl.col("title").first())
        c_uris = np.array(corpus["uri"].to_list())
        c_emb = common.encode_texts(model, corpus["title"].to_list(), show_progress_bar=False)
        truth = sub["uri"].to_numpy()

        accs = {}
        for form in ("masculine", "feminine"):
            emb = common.encode_texts(model, sub[form].to_list(), show_progress_bar=False)
            idx, _ = common.batched_topk(emb, c_emb, k=5)
            accs[f"{form}_acc@1"] = round(float((c_uris[idx[:, 0]] == truth).mean()), 4)
            accs[f"{form}_acc@5"] = round(float(np.mean([truth[i] in c_uris[idx[i]] for i in range(len(truth))])), 4)
        gap = accs["masculine_acc@1"] - accs["feminine_acc@1"]
        parity_rows.append({"model": model_name, "language": lang, "n_pairs": sub.height, **accs,
                            "acc@1_gap_masc_minus_fem": round(gap, 4)})
        print(f"  {model_name:24s} {lang}: masc@1={accs['masculine_acc@1']:.3f}  "
              f"fem@1={accs['feminine_acc@1']:.3f}  gap={gap:+.3f}")
    del model

parity_df = pl.DataFrame(parity_rows).sort(["model", "language"])
parity_df

  MiniLM (repo baseline)   de: masc@1=0.827  fem@1=0.824  gap=+0.003

  MiniLM (repo baseline)   es: masc@1=0.533  fem@1=0.498  gap=+0.035

  MiniLM (repo baseline)   it: masc@1=0.434  fem@1=0.420  gap=+0.014

  MiniLM (repo baseline)   fr: masc@1=0.470  fem@1=0.319  gap=+0.151

  JobBERT-v3               de: masc@1=0.978  fem@1=0.946  gap=+0.032

  JobBERT-v3               es: masc@1=0.667  fem@1=0.628  gap=+0.039

  JobBERT-v3               it: masc@1=0.573  fem@1=0.538  gap=+0.035

  JobBERT-v3               fr: masc@1=0.668  fem@1=0.599  gap=+0.069

shape: (8, 8)

model	language	n_pairs	masculine_acc@1	masculine_acc@5	feminine_acc@1	feminine_acc@5	acc@1_gap_masc_minus_fem
str	str	i64	f64	f64	f64	f64	f64
"JobBERT-v3"	"de"	1238	0.9782	0.9927	0.9459	0.9548	0.0323
"JobBERT-v3"	"es"	285	0.6667	0.7544	0.6281	0.7439	0.0386
"JobBERT-v3"	"fr"	232	0.6681	0.7414	0.5991	0.6897	0.069
"JobBERT-v3"	"it"	426	0.5728	0.6878	0.5376	0.6643	0.0352
"MiniLM (repo baseline)"	"de"	1238	0.8271	0.9394	0.8239	0.9168	0.0032
"MiniLM (repo baseline)"	"es"	285	0.5333	0.6561	0.4982	0.6246	0.0351
"MiniLM (repo baseline)"	"fr"	232	0.4698	0.6078	0.319	0.4655	0.1508
"MiniLM (repo baseline)"	"it"	426	0.4343	0.5915	0.4202	0.5634	0.0141

A positive gap means the masculine form is retrieved correctly more often — the feminine form is the disadvantaged one. A negative gap means the reverse. Either direction is a failure of the property we want, which is a gap of zero.

5. Test 3 — A mitigation, measured rather than recommended

The simplest mitigation that needs no retraining: normalise both forms to a single canonical string before matching ("concept normalisation"). For German we strip the -in feminine suffix; for Romance languages we map the feminine ending back to its masculine counterpart.

This is deliberately crude. The question is not whether it is elegant — it is whether it actually closes the gap, and what it costs.

The obvious objection, stated up front: normalising to the masculine form treats the masculine as the "real" version of the word, which is itself a political choice, and it destroys a genuine distinction in languages where the feminine form is the one a person actually uses about themselves. A truly neutral canonical form does not exist in these languages. We measure the mechanical effect and let you weigh that trade-off yourself.

REVERSE_RULES = {
    "es": [("ora", "or"), ("dora", "dor"), ("era", "ero"), ("a", "o")],
    "it": [("rice", "ore"), ("iera", "iere"), ("a", "o")],
    "fr": [("euse", "eur"), ("trice", "teur"), ("ière", "ier"), ("ée", "é")],
}


def neutralize(title, lang):
    """Map a (possibly feminine) surface form to one canonical form."""
    if lang == "de":
        # 'Ingenieurin' -> 'Ingenieur'; also handles the plural '-innen'.
        return re.sub(r"innen\b", "", re.sub(r"in\b", "", title)).strip()
    for suf_f, suf_m in REVERSE_RULES.get(lang, []):
        if title.endswith(suf_f):
            return title[: -len(suf_f)] + suf_m
    return title


print("Examples of the normalisation:")
for r in pairs_df.head(6).iter_rows(named=True):
    lang = r["language"]
    print(f"  [{lang}] {r['masculine']!r} -> {neutralize(r['masculine'], lang)!r}")
    print(f"  [{lang}] {r['feminine']!r} -> {neutralize(r['feminine'], lang)!r}")

Examples of the normalisation:
  [de] '3D-Artist' -> '3D-Artist'
  [de] '3D-Artistin' -> '3D-Artist'
  [de] '3D-Druck-Techniker' -> '3D-Druck-Techniker'
  [de] '3D-Druck-Technikerin' -> '3D-Druck-Techniker'
  [de] 'AHS-Lehrer' -> 'AHS-Lehrer'
  [de] 'in' -> ''
  [de] 'Abfallsortierer' -> 'Abfallsortierer'
  [de] 'Abfallsortiererin' -> 'Abfallsortierer'
  [de] 'Abraumbaggerfahrer' -> 'Abraumbaggerfahrer'
  [de] 'Abraumbaggerfahrerin' -> 'Abraumbaggerfahrer'
  [de] 'Abteilungsleiter' -> 'Abteilungsleiter'
  [de] 'Abteilungsleiterin' -> 'Abteilungsleiter'

mitigation_rows = []
for model_name, hf_id in MODELS:
    model = SentenceTransformer(hf_id, device="cpu")
    for lang in ("de", "es", "it", "fr"):
        sub = pairs_df.filter(pl.col("language") == lang)
        if sub.height < 30:
            continue
        corpus = pref.filter(pl.col("language") == lang).group_by("uri").agg(pl.col("title").first())
        c_emb = common.encode_texts(model, corpus["title"].to_list(), show_progress_bar=False)
        c_uris = np.array(corpus["uri"].to_list())
        truth = sub["uri"].to_numpy()

        for setting in ("raw", "normalised"):
            m_in = sub["masculine"].to_list()
            f_in = sub["feminine"].to_list()
            if setting == "normalised":
                m_in = [neutralize(t, lang) for t in m_in]
                f_in = [neutralize(t, lang) for t in f_in]
            me = common.encode_texts(model, m_in, show_progress_bar=False)
            fe = common.encode_texts(model, f_in, show_progress_bar=False)
            mi, _ = common.batched_topk(me, c_emb, k=TOP_K)
            fi, _ = common.batched_topk(fe, c_emb, k=TOP_K)
            rbos = [rbo(list(mi[i]), list(fi[i]), p=RBO_P) for i in range(sub.height)]
            mitigation_rows.append({
                "model": model_name, "language": lang, "setting": setting,
                "n_pairs": sub.height,
                f"mean_RBO@{TOP_K}": round(float(np.mean(rbos)), 4),
                "identical_top1_rate": round(float(np.mean([mi[i][0] == fi[i][0] for i in range(sub.height)])), 4),
                "masc_acc@1": round(float((c_uris[mi[:, 0]] == truth).mean()), 4),
                "fem_acc@1": round(float((c_uris[fi[:, 0]] == truth).mean()), 4),
            })
    del model

mitigation_df = pl.DataFrame(mitigation_rows).sort(["model", "language", "setting"])
mitigation_df

shape: (16, 8)

model	language	setting	n_pairs	mean_RBO@20	identical_top1_rate	masc_acc@1	fem_acc@1
str	str	str	i64	f64	f64	f64	f64
"JobBERT-v3"	"de"	"normalised"	1238	0.8356	0.9588	0.9782	0.954
"JobBERT-v3"	"de"	"raw"	1238	0.7566	0.9483	0.9782	0.9451
"JobBERT-v3"	"es"	"normalised"	285	0.8784	1.0	0.6667	0.6667
"JobBERT-v3"	"es"	"raw"	285	0.6731	0.8491	0.6667	0.6281
"JobBERT-v3"	"fr"	"normalised"	232	0.8784	1.0	0.6681	0.6681
…	…	…	…	…	…	…	…
"MiniLM (repo baseline)"	"es"	"raw"	285	0.6431	0.7228	0.5333	0.4982
"MiniLM (repo baseline)"	"fr"	"normalised"	232	0.8784	1.0	0.4698	0.4698
"MiniLM (repo baseline)"	"fr"	"raw"	232	0.5393	0.5	0.4698	0.319
"MiniLM (repo baseline)"	"it"	"normalised"	426	0.8784	1.0	0.4343	0.4343
"MiniLM (repo baseline)"	"it"	"raw"	426	0.6553	0.73	0.4343	0.4202

⚠️ Read the mitigation numbers carefully — 1.0 here is not a triumph

For Spanish, French and Italian the normalised setting reaches an identical_top1_rate of exactly 1.000, and the feminine accuracy rises to match the masculine one precisely. That is not the model learning to be fair. It is arithmetic: our normalisation rewrites the feminine form into the masculine form, so both queries become the same string, and identical strings must retrieve identical results. The 1.000 is guaranteed by construction before any model is consulted.

This is worth spelling out because it is exactly the shape of error that got four earlier projects on this machine retracted — a number that looks like a result but is actually a definition. The honest readings are:

• What the mitigation genuinely buys: feminine-form queries stop being second-class, and it costs nothing in masculine accuracy (masc_acc@1 is unchanged in every row). If your goal is "the same occupation should return the same results regardless of gendered spelling", this achieves it, cheaply, with no retraining.
• What it genuinely costs: the feminine form no longer exists as far as the system is concerned. Every user searching abogada is silently served results computed for abogado. We have not made the model fair; we have removed its opportunity to be unfair, by deleting the input distinction. Whether that is acceptable is a product and political decision, not a technical one.
• German is the informative case. It does not reach 1.000 (0.9588 for JobBERT-v3), because stripping the -in suffix does not always reconstruct the exact masculine label — German compounds change more than their ending. So the German column is the only one measuring a real, non-tautological effect, and it still improves (0.9483 → 0.9588).

The genuinely model-side result in this notebook is therefore section 3 and section 4, not section 5: the raw symmetry and parity gaps, measured before any normalisation, are where the models are actually being judged.

6. Visualising the result

fig, ax = plt.subplots(1, 2, figsize=(14, 4.5))

piv = mitigation_df.filter(pl.col("model") == "JobBERT-v3")
langs = piv["language"].unique(maintain_order=True).to_list()
x = np.arange(len(langs)); w = 0.35
for i, setting in enumerate(("raw", "normalised")):
    vals = [piv.filter((pl.col("language") == l) & (pl.col("setting") == setting))["identical_top1_rate"][0] for l in langs]
    ax[0].bar(x + (i - 0.5) * w, vals, w, label=setting, color=["#E45756", "#54A24B"][i])
ax[0].set_xticks(x); ax[0].set_xticklabels(langs); ax[0].set_ylim(0, 1.05)
ax[0].set_ylabel("share of pairs with identical top-1 result")
ax[0].set_title("Do masculine & feminine queries agree?\n(JobBERT-v3; 1.0 = perfectly symmetric)")
ax[0].axhline(1.0, ls="--", color="gray", lw=1)
ax[0].legend(); ax[0].grid(alpha=.3, axis="y")

for i, (model_name, _) in enumerate(MODELS):
    sub = parity_df.filter(pl.col("model") == model_name)
    ax[1].bar(np.arange(sub.height) + (i - 0.5) * 0.35, sub["acc@1_gap_masc_minus_fem"].to_list(),
              0.35, label=model_name, color=["#4C78A8", "#F58518"][i])
ax[1].axhline(0, color="black", lw=1)
ax[1].set_xticks(np.arange(parity_df.filter(pl.col("model") == MODELS[0][0]).height))
ax[1].set_xticklabels(parity_df.filter(pl.col("model") == MODELS[0][0])["language"].to_list())
ax[1].set_ylabel("accuracy@1 gap (masculine − feminine)")
ax[1].set_title("Retrieval parity gap\n(0 = fair; above 0 = feminine form disadvantaged)")
ax[1].legend(fontsize=8); ax[1].grid(alpha=.3, axis="y")
plt.tight_layout(); plt.savefig(os.path.join(RESULTS, "gender_bias.png"), dpi=110, bbox_inches="tight")
plt.show()

symmetry_df.write_csv(os.path.join(RESULTS, "gender_symmetry_rbo.csv"))
parity_df.write_csv(os.path.join(RESULTS, "gender_retrieval_parity.csv"))
mitigation_df.write_csv(os.path.join(RESULTS, "gender_mitigation.csv"))
per_pair_df.write_csv(os.path.join(RESULTS, "gender_pairs_per_pair_rbo.csv"))
pairs_df.write_csv(os.path.join(common.DATA_DIR, "gendered_title_pairs.csv"))
print("Wrote 4 result CSVs + data/processed/gendered_title_pairs.csv")

Wrote 4 result CSVs + data/processed/gendered_title_pairs.csv

7. What this notebook established

• Gendered title pairs were extracted from ESCO's own official dual-form labels, not hand-invented, across German, Spanish, Italian and French. The pair count per language is stated in every table.
• RBO was implemented from its definition and validated on cases whose answers are known by inspection (identical lists, disjoint lists, top-swap vs deep-swap), so the metric itself is auditable rather than imported on faith.
• Bias is reported three ways — ranking symmetry (RBO), identical-top-1 rate, and a retrieval accuracy gap — because any single number would be easy to misread. The identical-top-1 rate is the one to quote to a non-technical audience: it is simply "how often does changing the gender of the word change the answer?"
• A cheap mitigation was measured, not merely proposed, alongside an explicit statement of what it costs conceptually.

Limits of this analysis, stated plainly

• Only four languages, all Indo-European, all with binary grammatical gender. Nothing here says anything about non-binary forms, or about languages with different gender systems.
• The pair-extraction rules are conservative and will have missed valid pairs; this measures the bias on the pairs we found, which is not necessarily the bias on all titles.
• This measures bias in matching. It does not measure hiring outcomes, and it would be a serious overreach to present these numbers as evidence about real-world discrimination.
• Following the repo's standing rule: a model being more accurate does not make it fairer, and the two tables above should be read together rather than collapsed into a single "best model" verdict.