03 — Multilingual Embedding Matching

A frozen (not fine-tuned) multilingual sentence-embedding retrieval matcher: encode a query title and every ESCO candidate title with the same model, rank candidates by cosine similarity. No task-specific training happens in this notebook at all — see RESEARCH.md for why (fine-tuning bi-encoders needs a GPU and a large labeled title-pair dataset; this machine has neither). This is a deliberate, disclosed downgrade from the literature's dominant approach, not an attempt to match published SOTA numbers.

Model: paraphrase-multilingual-MiniLM-L12-v2 (384-dim, ~470MB, already cached locally — see environment/README.md). Chosen because it's explicitly trained for cross-lingual paraphrase similarity across 50+ languages, which is exactly this task's shape.

Two things happen in this notebook:

1. Build + cache embeddings for the same English ESCO candidate lexicon notebook 02 used, so notebook 04 can compare the two matchers head-to-head on identical ground. Embeddings are cached to data/processed/ as .npy so notebook 04 doesn't have to re-encode ~17K titles.
2. A genuinely cross-lingual, ground-truth-backed retrieval check: can we query with an English ESCO preferred label and retrieve the correct translation out of another language's full ESCO preferred-label set? This is the one thing the lexical baseline structurally cannot do at all — worth measuring on its own, separately from the English-only O*NET eval in notebook 04.

import time

import matplotlib.pyplot as plt
import numpy as np
import polars as pl
from sentence_transformers import SentenceTransformer

from common import DATA_DIR, RESULTS_DIR, batched_topk, build_esco_en_candidates, encode_texts, load_esco, normalize

BLUE = "#2a78d6"
ORANGE = "#eb6834"
INK = "#0b0b0b"
SECONDARY_INK = "#52514e"
MUTED = "#898781"
GRID = "#e1e0d9"
SURFACE = "#fcfcfb"

plt.rcParams.update({
    "figure.facecolor": SURFACE, "axes.facecolor": SURFACE, "axes.edgecolor": GRID,
    "axes.labelcolor": SECONDARY_INK, "text.color": INK, "xtick.color": SECONDARY_INK,
    "ytick.color": SECONDARY_INK, "axes.grid": True, "grid.color": GRID,
    "grid.linewidth": 0.8, "font.size": 10,
})

MODEL_NAME = "paraphrase-multilingual-MiniLM-L12-v2"
t0 = time.time()
model = SentenceTransformer(MODEL_NAME, device="cpu")
print(f"Loaded {MODEL_NAME} in {time.time() - t0:.1f}s")

esco_occ, esco_titles = load_esco()

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

Loaded paraphrase-multilingual-MiniLM-L12-v2 in 5.3s

1. Encode the ESCO EN candidate lexicon (same one notebook 02 used)

normalize_embeddings=True in common.encode_texts means cosine similarity is just a dot product — cheaper and numerically identical.

candidates = build_esco_en_candidates(esco_titles)
n_dropped = candidates["n_dropped_ambiguous_norms"][0]
print(f"Candidate lexicon: {candidates.height} unique normalized EN titles "
      f"({candidates['uri'].n_unique()} distinct ESCO occupations); "
      f"{n_dropped} ambiguous normalized titles dropped (same rule as notebook 02).")

t0 = time.time()
cand_emb = encode_texts(model, candidates["title"].to_list(), batch_size=64)
elapsed = time.time() - t0
print(f"Encoded {cand_emb.shape[0]} candidate titles -> {cand_emb.shape} in {elapsed:.1f}s "
      f"({elapsed / cand_emb.shape[0] * 1000:.2f} ms/title)")

emb_path = f"{DATA_DIR}/esco_en_candidate_embeddings.npy"
csv_path = f"{DATA_DIR}/esco_en_candidates.csv"
np.save(emb_path, cand_emb)
candidates.write_csv(csv_path)
print(f"Cached embeddings to {emb_path} ({cand_emb.nbytes / 1e6:.1f} MB) "
      f"and row-aligned metadata to {csv_path}, for notebook 04 to reuse without re-encoding.")

Candidate lexicon: 16919 unique normalized EN titles (1695 distinct ESCO occupations); 490 ambiguous normalized titles dropped (same rule as notebook 02).

Encoded 16919 candidate titles -> (16919, 384) in 45.3s (2.68 ms/title)
Cached embeddings to /home/ubuntu/claude_/multilingual-job-title-matching/notebooks/../data/processed/esco_en_candidate_embeddings.npy (26.0 MB) and row-aligned metadata to /home/ubuntu/claude_/multilingual-job-title-matching/notebooks/../data/processed/esco_en_candidates.csv, for notebook 04 to reuse without re-encoding.

2. Demo: queries the lexical baseline missed

The same examples from notebook 02, several of which the fuzzy matcher rejected outright (below the 85/100 threshold, or zero token overlap). Here we embed the query and retrieve the top-5 nearest ESCO candidates by cosine similarity.

demo_queries = ["Software Developer", "Marketing Manager, Digital", "Death Investigator",
                 "Certified Wellness Program Coordinator", "MDI (Medicolegal Death Investigator)",
                 "Penetration Testers", "Coder"]
query_emb = encode_texts(model, demo_queries, batch_size=32, show_progress_bar=False)
top_idx, top_score = batched_topk(query_emb, cand_emb, k=5)

titles = candidates["title"].to_list()
for i, q in enumerate(demo_queries):
    print(f"\nQUERY: {q!r}")
    for rank in range(5):
        j = top_idx[i, rank]
        print(f"  #{rank+1}  {top_score[i, rank]:.3f}  {titles[j]!r}")

QUERY: 'Software Developer'
  #1  0.987  'software developer'
  #2  0.966  'software developers'
  #3  0.964  'developer of software'
  #4  0.908  'system software developer'
  #5  0.907  'application software developers'

QUERY: 'Marketing Manager, Digital'
  #1  0.903  'digital marketing strategist'
  #2  0.825  'product marketing manager'
  #3  0.820  'marketing e-catalogue manager'
  #4  0.815  'marketing manager (brand development)'
  #5  0.814  'marketing director'

QUERY: 'Death Investigator'
  #1  0.739  'researcher in the psychology of death and dying'
  #2  0.709  'autopsy assistant'
  #3  0.706  'mortician'
  #4  0.689  'coroner'
  #5  0.658  'funeral director'

QUERY: 'Certified Wellness Program Coordinator'
  #1  0.842  'wellness consultant'
  #2  0.813  'health and wellness consultant'
  #3  0.703  'health and fitness consultant'
  #4  0.694  'sustainable health practitioner'
  #5  0.656  'fitness and nutrition consultant'

QUERY: 'MDI (Medicolegal Death Investigator)'
  #1  0.634  'autopsy assistant'
  #2  0.618  'forensic medical examiner'
  #3  0.580  'health & safety inspector'
  #4  0.579  'researcher in the psychology of death and dying'
  #5  0.578  'life insurance representative'

QUERY: 'Penetration Testers'
  #1  0.836  'tester'
  #2  0.788  'gauge tester'
  #3  0.766  'test engineer'
  #4  0.748  'controls tester'
  #5  0.742  'veneer tester'

QUERY: 'Coder'
  #1  0.678  'database coder'
  #2  0.675  'programmer'
  #3  0.642  'industrial mobile devices software coder'
  #4  0.596  'software programmer'
  #5  0.577  'fabricator/welder'

Compare this to notebook 02's output for the same queries: "Death Investigator", "Certified Wellness Program Coordinator", and "MDI (Medicolegal Death Investigator)" all had no lexical match at all (zero token overlap with anything in the ESCO lexicon above the threshold). The embedding matcher retrieves semantically related ESCO occupations for all of them — whether the top-1 result is the "textbook correct" ESCO occupation is exactly what notebook 04 checks systematically, not just by eyeballing these seven examples.

3. Cross-lingual retrieval: the thing lexical matching structurally can't do

Ground truth here doesn't depend on O*NET at all: ESCO's preferred label for a given occupation URI in language X is, by construction, the correct translation of that occupation's English preferred label. So we can measure, honestly and without any external crosswalk: query with the English preferred label of every ESCO occupation, search only within another language's full preferred-label set, and check whether the correct translation (same occupation URI) comes back at rank 1 / within the top 5.

We do this for every ESCO occupation (no subsampling — it's cheap, ~1700 queries x ~13 languages) and for a spread of languages: large EU languages, a non-Latin-script language (Greek), and a right-to-left non-Latin-script language (Arabic) as a real stress test for this specific model.

target_langs = ["fr", "de", "es", "it", "pt", "nl", "pl", "sv", "cs", "ro", "hu", "el", "ar"]
en_pref = (
    esco_titles.filter((pl.col("language").is_in(["en", "en-us"])) & (pl.col("label_type") == "preferred"))
    .group_by("uri").agg(pl.col("title").first().alias("title_en"))
)
print(f"English preferred-label queries: {en_pref.height} (one per ESCO occupation)")

en_query_emb = encode_texts(model, en_pref.sort("uri")["title_en"].to_list(), batch_size=64, show_progress_bar=False)
en_pref_sorted = en_pref.sort("uri")

results = []
for lang in target_langs:
    lang_pref = (
        esco_titles.filter((pl.col("language") == lang) & (pl.col("label_type") == "preferred"))
        .group_by("uri").agg(pl.col("title").first().alias("title_lang"))
        .sort("uri")
    )
    # inner-join on uri to align query i <-> correct-answer i by occupation, in case
    # a language is missing a preferred label for some occupation (shouldn't happen
    # per notebook 01's finding, but don't assume)
    aligned = en_pref_sorted.join(lang_pref, on="uri", how="inner")
    if aligned.height < en_pref_sorted.height:
        print(f"  [{lang}] note: only {aligned.height}/{en_pref_sorted.height} occupations have a preferred label")

    lang_cand_emb = encode_texts(model, lang_pref["title_lang"].to_list(), batch_size=64, show_progress_bar=False)
    # re-derive query embeddings restricted to the aligned uri set, same order as lang_pref candidates
    q_idx = {u: i for i, u in enumerate(en_pref_sorted["uri"].to_list())}
    aligned_q_emb = en_query_emb[[q_idx[u] for u in aligned["uri"].to_list()]]

    cand_uri_list = lang_pref["uri"].to_list()
    cand_idx = {u: i for i, u in enumerate(cand_uri_list)}
    true_idx = np.array([cand_idx[u] for u in aligned["uri"].to_list()])

    top_idx_lang, _ = batched_topk(aligned_q_emb, lang_cand_emb, k=5)
    acc1 = float(np.mean(top_idx_lang[:, 0] == true_idx))
    acc5 = float(np.mean((top_idx_lang == true_idx[:, None]).any(axis=1)))
    results.append({"language": lang, "n_queries": aligned.height, "accuracy_at_1": round(acc1, 4), "accuracy_at_5": round(acc5, 4)})
    print(f"[{lang}] n={aligned.height:5d}  acc@1={acc1:.3f}  acc@5={acc5:.3f}")

cross_lingual_df = pl.DataFrame(results)
cross_lingual_df.write_csv(f"{RESULTS_DIR}/cross_lingual_retrieval_accuracy.csv")
cross_lingual_df

English preferred-label queries: 1699 (one per ESCO occupation)

[fr] n= 1699  acc@1=0.464  acc@5=0.639

[de] n= 1699  acc@1=0.359  acc@5=0.537

[es] n= 1699  acc@1=0.533  acc@5=0.730

[it] n= 1699  acc@1=0.459  acc@5=0.636

[pt] n= 1699  acc@1=0.498  acc@5=0.690

[nl] n= 1699  acc@1=0.492  acc@5=0.675

[pl] n= 1699  acc@1=0.479  acc@5=0.659

[sv] n= 1699  acc@1=0.428  acc@5=0.616

[cs] n= 1699  acc@1=0.487  acc@5=0.676

[ro] n= 1699  acc@1=0.463  acc@5=0.659

[hu] n= 1699  acc@1=0.442  acc@5=0.658

[el] n= 1699  acc@1=0.403  acc@5=0.613

[ar] n= 1699  acc@1=0.426  acc@5=0.657

shape: (13, 4)

language	n_queries	accuracy_at_1	accuracy_at_5
str	i64	f64	f64
"fr"	1699	0.4638	0.6392
"de"	1699	0.359	0.5368
"es"	1699	0.5327	0.7298
"it"	1699	0.4585	0.6357
"pt"	1699	0.4979	0.6904
…	…	…	…
"cs"	1699	0.4868	0.6763
"ro"	1699	0.4626	0.6592
"hu"	1699	0.442	0.658
"el"	1699	0.4032	0.6133
"ar"	1699	0.4261	0.6569

Every query and candidate here is an official ESCO preferred label — the cleanest possible case (formal terminology, not messy self-reported text). So these numbers are a ceiling for cross-lingual capability, not a prediction of real-world messy-text performance (that's what notebook 04's O*NET eval is for, and it's English-only, so treat this section and notebook 04 as two separate, non-comparable findings).

fig, ax = plt.subplots(figsize=(9, 5))
cl_sorted = cross_lingual_df.sort("accuracy_at_1")
ax.barh(cl_sorted["language"], cl_sorted["accuracy_at_1"], color=BLUE, height=0.6, label="accuracy@1")
ax.barh(cl_sorted["language"], cl_sorted["accuracy_at_5"] - cl_sorted["accuracy_at_1"],
        left=cl_sorted["accuracy_at_1"], color=ORANGE, height=0.6, alpha=0.55, label="+ accuracy@5")
ax.set_xlim(0, 1)
ax.set_xlabel("Retrieval accuracy (query = EN preferred label, candidates = target-language preferred labels)")
ax.set_title("Cross-lingual retrieval accuracy by language (frozen embedding model)", loc="left", fontweight="bold")
ax.legend(frameon=False, loc="lower right")
ax.spines[["top", "right"]].set_visible(False)
plt.tight_layout()
plt.savefig(f"{RESULTS_DIR}/cross_lingual_retrieval_accuracy.png", dpi=110)
plt.show()

Summary

• Embeddings for the same 16,919-title English ESCO candidate lexicon notebook 02 used are cached to data/processed/esco_en_candidate_embeddings.npy (+ row-aligned esco_en_candidates.csv) so notebook 04 can load them directly instead of re-encoding.
• Qualitatively, the embedding matcher finds semantically sensible ESCO candidates for titles the lexical baseline flatly rejected (zero token overlap) — see section 2. Whether the top-1 candidate is the textbook correct occupation, systematically, is measured honestly in notebook 04.
• The cross-lingual retrieval check (section 3) — which needs no O*NET data and has clean, non-circular ground truth via ESCO's own per-language preferred labels — is the one capability lexical/fuzzy matching cannot have at all by construction. See the actual per-language numbers above; read them off the printed table / chart rather than assuming a number here.