06 — Classification: predicting a taxonomy code from a job title

Notebook 05 did retrieval: given a title, rank other titles by similarity. This notebook does classification: given a title, predict which one of a fixed, known list of occupation codes it belongs to.

"Fixed label space" — what that actually means

The label space is simply the set of answers a model is allowed to give.

• Fixed label space: the answer must be one of a known list that does not change. Here: one of 1,369 ESCO occupation codes. The model can output a probability for every possible answer, because it has seen them all during training.
• Open label space: the set of possible answers is huge, changes over time, or is not known in advance — e.g. "find similar titles in a corpus that gets new titles daily". You cannot enumerate the answers, so you compare against whatever is in the index right now.

Why the distinction decides your method:

	Classification	Retrieval
Needs the answer list up front	Yes	No
Adding a new occupation	Retrain the model	Add one row to the index
Learns from all examples of a class	Yes — it sees 647 ways people wrote "marketing assistant"	Only compares to the labels in the index
Handles a class with 1 training example	Badly	Fine — one entry in the index is enough
Gives calibrated probabilities	Naturally	Needs extra work

So: classification wins when the answer list is fixed and you have plenty of examples per answer, because it gets to learn the messy real-world ways people write each occupation. Retrieval wins when the answer list changes, or when most answers have very few examples. This notebook measures both on identical data so you can see the trade-off rather than take my word for it.

The data — and why it is a genuinely good test

TechWolf/JobBERT-evaluation-dataset (MIT licence): real vacancy titles scraped from a government job board, each hand-tagged with an ESCO occupation. This matters enormously for honesty. Every previous ESCO experiment in this repo risked being circular — testing ESCO titles against a lexicon built from ESCO. Here the inputs are real, messy, human-written vacancy titles from an entirely different source ("Marketing", "SALES EXECUTIVE MANAGER", "Admin Marketing"), and only the labels are ESCO. That is the non-circular setup this repo has been missing.

import os
import re
import warnings

warnings.filterwarnings("ignore")
os.environ["TOKENIZERS_PARALLELISM"] = "false"

import numpy as np
import polars as pl
import matplotlib.pyplot as plt

import common

RAW = os.path.join(os.path.dirname(common.DATA_DIR), "raw", "jobbert_eval")
RESULTS = common.RESULTS_DIR
SEED = 42
np.random.seed(SEED)

test_df = pl.read_csv(os.path.join(RAW, "jobbert_ds_test.csv"))
val_df = pl.read_csv(os.path.join(RAW, "jobbert_ds_val.csv"))
print(f"Shipped files: test={test_df.height:,} rows, val={val_df.height:,} rows")
test_df.head(5)

Shipped files: test=15,463 rows, val=15,463 rows

shape: (5, 3)

vacancy_job_title	esco_job_title	esco_URI
str	str	str
"Marketing"	"marketing assistant"	"http://data.europa.eu/esco/occ…
"Marketing Advisor"	"marketing assistant"	"http://data.europa.eu/esco/occ…
"Admin Marketing"	"marketing assistant"	"http://data.europa.eu/esco/occ…
"SALES EXECUTIVE MANAGER"	"marketing assistant"	"http://data.europa.eu/esco/occ…
"Talent Executive"	"marketing assistant"	"http://data.europa.eu/esco/occ…

1. Inspecting the label space before doing anything else

You cannot interpret an accuracy number without knowing the shape of the label space it was measured over. 90% accuracy is trivial with 2 balanced classes and extraordinary with 1,369 skewed ones.

all_df = pl.concat([val_df, test_df])
n_classes = all_df["esco_URI"].n_unique()
class_counts = all_df.group_by("esco_URI").len().sort("len", descending=True)
majority = class_counts["len"][0] / all_df.height

print(f"Total rows          : {all_df.height:,}")
print(f"Distinct ESCO codes : {n_classes:,}   <- the size of the label space")
print(f"Largest class       : {class_counts['len'][0]:,} rows ({majority:.2%} of data)")
print(f"  => MAJORITY-CLASS BASELINE = {majority:.2%}. Any model must beat this to have learned anything.")
print(f"  => RANDOM-GUESS BASELINE   = {1/n_classes:.4%} (1 in {n_classes:,})")
print(f"Classes with only 1 example : {(class_counts['len'] == 1).sum():,}")
print(f"Classes with <5 examples    : {(class_counts['len'] < 5).sum():,}")
print(f"Median examples per class   : {class_counts['len'].median():.0f}")

Total rows          : 30,926
Distinct ESCO codes : 1,554   <- the size of the label space
Largest class       : 1,294 rows (4.18% of data)
  => MAJORITY-CLASS BASELINE = 4.18%. Any model must beat this to have learned anything.
  => RANDOM-GUESS BASELINE   = 0.0644% (1 in 1,554)
Classes with only 1 example : 362
Classes with <5 examples    : 816
Median examples per class   : 4

fig, ax = plt.subplots(1, 2, figsize=(13, 4))
ax[0].plot(range(1, len(class_counts) + 1), class_counts["len"].to_list(), color="#4C78A8")
ax[0].set_yscale("log"); ax[0].set_xlabel("class rank"); ax[0].set_ylabel("examples (log)")
ax[0].set_title("The long tail: a few common occupations,\nhundreds with almost no examples")
ax[0].grid(alpha=.3)
ax[1].hist(class_counts["len"].to_list(), bins=60, color="#4C78A8")
ax[1].set_yscale("log"); ax[1].set_xlabel("examples per class"); ax[1].set_ylabel("number of classes (log)")
ax[1].set_title("Most classes are rare")
ax[1].grid(alpha=.3)
plt.tight_layout(); plt.savefig(os.path.join(RESULTS, "classification_label_space.png"), dpi=110, bbox_inches="tight")
plt.show()

That long tail is the central difficulty. 448 occupations have exactly one example. No classifier learns a class from one example — but a retrieval system handles it fine, because one entry in an index is all it needs. This is the clearest practical illustration of the trade-off table above, and we measure its consequences directly in section 6.

2. Building an honest split

The repo's rule, learned from four earlier projects whose numbers collapsed on review: deduplicate by normalized text before splitting. If "Marketing Manager" and "marketing manager " land on opposite sides of the split, the model gets tested on something it memorised, and the score is inflated.

The dataset ships its own val/test files. Before trusting them, we check whether they leak into each other — a check worth doing on any pre-split dataset rather than assuming the authors did it.

val_norm = set(val_df["vacancy_job_title"].map_elements(common.normalize, return_dtype=pl.Utf8).to_list())
test_norm = set(test_df["vacancy_job_title"].map_elements(common.normalize, return_dtype=pl.Utf8).to_list())
overlap = val_norm & test_norm
print(f"Normalized titles in shipped val : {len(val_norm):,}")
print(f"Normalized titles in shipped test: {len(test_norm):,}")
print(f"OVERLAP between them             : {len(overlap):,} ({len(overlap)/len(test_norm):.1%} of test)")
print(f"Examples of overlapping titles   : {sorted(overlap)[:5]}")
print("\n=> The shipped split DOES share titles across val/test. Using it as-is would")
print("   let a model score partly by memorisation. We build our own clean split instead.")

Normalized titles in shipped val : 13,945
Normalized titles in shipped test: 13,927
OVERLAP between them             : 1,749 (12.6% of test)
Examples of overlapping titles   : ['2nd engineer', '3d animator', 'a4 chargeman', 'able bodied', 'able seaman']

=> The shipped split DOES share titles across val/test. Using it as-is would
   let a model score partly by memorisation. We build our own clean split instead.

# Deduplicate by normalized title, keeping one row per distinct title, THEN split.
dedup = (
    all_df.with_columns(
        all_df["vacancy_job_title"].map_elements(common.normalize, return_dtype=pl.Utf8).alias("norm")
    )
    .filter(pl.col("norm").str.len_chars() > 0)
    .unique(subset=["norm"], keep="first")
    .sort("norm")  # deterministic order before shuffling, so the seed fully controls the split
)
print(f"After dedup by normalized text: {dedup.height:,} rows (from {all_df.height:,})")

# Keep only classes with >=2 examples so every class can appear in both halves;
# report exactly how much this discards rather than quietly dropping it.
counts = dedup.group_by("esco_URI").len()
keepable = counts.filter(pl.col("len") >= 2)["esco_URI"]
dropped_rows = dedup.height - dedup.filter(pl.col("esco_URI").is_in(keepable)).height
model_df = dedup.filter(pl.col("esco_URI").is_in(keepable))
print(f"Dropped {dropped_rows:,} rows in {counts.height - len(keepable):,} singleton classes "
      f"(cannot be in both train and test). Remaining: {model_df.height:,} rows, {len(keepable):,} classes.")

rng = np.random.default_rng(SEED)
idx = rng.permutation(model_df.height)
split = int(0.75 * model_df.height)
train = model_df[idx[:split]]
test = model_df[idx[split:]]

# Any class absent from train cannot possibly be predicted; exclude from test and say so.
train_classes = set(train["esco_URI"].to_list())
n_before = test.height
test = test.filter(pl.col("esco_URI").is_in(train_classes))
print(f"\nTrain: {train.height:,} | Test: {test.height:,} "
      f"(dropped {n_before - test.height:,} test rows whose class never appears in train)")

leak = set(train["norm"].to_list()) & set(test["norm"].to_list())
print(f"Normalized-text leakage between our train and test: {len(leak)}  <- must be 0")

After dedup by normalized text: 26,123 rows (from 30,926)
Dropped 388 rows in 388 singleton classes (cannot be in both train and test). Remaining: 25,735 rows, 1,137 classes.

Train: 19,301 | Test: 6,408 (dropped 26 test rows whose class never appears in train)
Normalized-text leakage between our train and test: 0  <- must be 0

3. Baselines

Two floors, reported next to every model number from here on.

y_train = train["esco_URI"].to_numpy()
y_test = test["esco_URI"].to_numpy()
X_train_txt = train["vacancy_job_title"].to_list()
X_test_txt = test["vacancy_job_title"].to_list()

vals, cnts = np.unique(y_train, return_counts=True)
majority_class = vals[np.argmax(cnts)]
maj_acc = float((y_test == majority_class).mean())
rand_acc = 1.0 / len(vals)

print(f"Test rows: {len(y_test):,} across {len(set(y_test)):,} classes")
print(f"BASELINE majority-class ('always guess the most common occupation'): {maj_acc:.4f}")
print(f"BASELINE random guess (1/{len(vals):,} classes)                    : {rand_acc:.6f}")

results = [{"method": "baseline: majority class", "accuracy@1": round(maj_acc, 4), "accuracy@5": None, "train_seconds": 0.0},
           {"method": "baseline: random guess", "accuracy@1": round(rand_acc, 6), "accuracy@5": None, "train_seconds": 0.0}]

Test rows: 6,408 across 885 classes
BASELINE majority-class ('always guess the most common occupation'): 0.0462
BASELINE random guess (1/1,124 classes)                    : 0.000890

4. Method A — TF-IDF + a linear classifier (the classical approach)

TF-IDF turns a title into a sparse vector counting which character/word patterns it contains, weighted so common patterns count for less. Logistic regression then learns a linear boundary per class. No neural network, no GPU, trains in seconds — this is the baseline any deep model must justify itself against.

Character n-grams (analyzer="char_wb") are used alongside words because job titles are full of abbreviations and typos ("Sr.", "Mgr", "Engr") where sub-word patterns carry the signal.

A real hardware constraint, hit while writing this notebook

The first version of this cell used uncapped character n-grams and plain LogisticRegression. It crashed with a MemoryError trying to allocate 12.3 GiB. The reason is worth understanding, because it will bite you on any large label space:

Any linear multiclass model stores one weight per (class, feature) pair. With 1,203,990 character n-gram features and 1,369 classes that is 1,203,990 × 1,369 ≈ 1.65 billion weights — 12.3 GiB at float64, for the coefficient matrix alone, before the optimiser's working copies.

The fix is to cap the feature count (max_features) and use SGDClassifier, which keeps far fewer working copies than lbfgs does. This is a genuine accuracy-for-memory trade, not a free win, and it is the kind of constraint that decides method choice on an 8GB machine — so it is recorded here rather than quietly patched away.

import time
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression, SGDClassifier
from sklearn.pipeline import make_pipeline, make_union


def topk_accuracy(probs, classes, y_true, k):
    """Share of rows whose true class is among the model's top-k guesses."""
    topk = np.argsort(-probs, axis=1)[:, :k]
    return float(np.mean([y_true[i] in classes[topk[i]] for i in range(len(y_true))]))


WORD_FEATURES, CHAR_FEATURES = 12_000, 12_000
features = make_union(
    TfidfVectorizer(analyzer="word", ngram_range=(1, 2), sublinear_tf=True,
                    min_df=1, max_features=WORD_FEATURES),
    TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5), sublinear_tf=True,
                    min_df=2, max_features=CHAR_FEATURES),
)
n_feat = WORD_FEATURES + CHAR_FEATURES
print(f"Capped at {n_feat:,} features x {len(set(y_train)):,} classes "
      f"=> coefficient matrix ~{n_feat * len(set(y_train)) * 8 / 1e9:.2f} GB at float64")

# `modified_huber` is used because, unlike plain hinge loss, it supports
# predict_proba - which we need for the top-5 accuracy figures below.
tfidf_clf = make_pipeline(
    features,
    SGDClassifier(loss="modified_huber", alpha=1e-5, max_iter=15,
                  tol=1e-3, random_state=SEED, n_jobs=1),
)

t0 = time.time()
tfidf_clf.fit(X_train_txt, y_train)
tfidf_train_s = time.time() - t0

probs = tfidf_clf.predict_proba(X_test_txt)
classes = tfidf_clf.classes_
tfidf_top1 = topk_accuracy(probs, classes, y_test, 1)
tfidf_top5 = topk_accuracy(probs, classes, y_test, 5)
print(f"TF-IDF + SGD (linear): accuracy@1 = {tfidf_top1:.4f}, accuracy@5 = {tfidf_top5:.4f}  (trained in {tfidf_train_s:.1f}s)")
print(f"  vs majority baseline {maj_acc:.4f} -> {tfidf_top1/maj_acc:.1f}x")
results.append({"method": "TF-IDF + linear SGD", "accuracy@1": round(tfidf_top1, 4),
                "accuracy@5": round(tfidf_top5, 4), "train_seconds": round(tfidf_train_s, 1)})

Capped at 24,000 features x 1,124 classes => coefficient matrix ~0.22 GB at float64

TF-IDF + SGD (linear): accuracy@1 = 0.3424, accuracy@5 = 0.4042  (trained in 74.2s)
  vs majority baseline 0.0462 -> 7.4x

5. Method B — frozen embeddings + Logistic Regression

Same classifier, different input: instead of counting character patterns, use a sentence-embedding model to turn each title into a dense 384/768-number vector, then fit logistic regression on those. The embedding model is frozen — not trained here — so this is cheap and CPU-friendly.

Two embedding models are compared: the general-purpose one this repo has used throughout, and JobBERT-v3 which was trained on job titles specifically.

from sentence_transformers import SentenceTransformer

EMBEDDERS = [
    ("MiniLM", "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"),
    ("JobBERT-v3", "TechWolf/JobBERT-v3"),
]
emb_cache = {}

for label, hf_id in EMBEDDERS:
    model = SentenceTransformer(hf_id, device="cpu")
    t0 = time.time()
    Xtr = common.encode_texts(model, X_train_txt, show_progress_bar=False)
    Xte = common.encode_texts(model, X_test_txt, show_progress_bar=False)
    encode_s = time.time() - t0
    emb_cache[label] = (Xtr, Xte)

    clf = LogisticRegression(max_iter=1000, C=10.0, n_jobs=1)
    t0 = time.time(); clf.fit(Xtr, y_train); fit_s = time.time() - t0

    p = clf.predict_proba(Xte)
    a1 = topk_accuracy(p, clf.classes_, y_test, 1)
    a5 = topk_accuracy(p, clf.classes_, y_test, 5)
    print(f"{label} embeddings + LogReg: accuracy@1 = {a1:.4f}, accuracy@5 = {a5:.4f} "
          f"(encode {encode_s:.0f}s + fit {fit_s:.0f}s)")
    results.append({"method": f"{label} embeddings + LogReg", "accuracy@1": round(a1, 4),
                    "accuracy@5": round(a5, 4), "train_seconds": round(fit_s, 1)})
    del model

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

MiniLM embeddings + LogReg: accuracy@1 = 0.4065, accuracy@5 = 0.6670 (encode 103s + fit 119s)

JobBERT-v3 embeddings + LogReg: accuracy@1 = 0.4515, accuracy@5 = 0.7260 (encode 333s + fit 159s)

6. Method C — retrieval used as a classifier (the direct comparison)

This is the head-to-head the whole notebook is built for. No classifier is trained at all. Instead we embed every training title, and to classify a new title we find its nearest training title and copy that title's label — a 1-nearest-neighbour retrieval system, framed as classification.

Comparing this against Methods A and B on identical data with an identical split is what makes "classification is better when the label space is fixed" a measurement rather than a slogan.

for label, _ in EMBEDDERS:
    Xtr, Xte = emb_cache[label]
    t0 = time.time()
    top_idx, _ = common.batched_topk(Xte, Xtr, k=5)
    search_s = time.time() - t0

    nn1 = y_train[top_idx[:, 0]]
    a1 = float((nn1 == y_test).mean())
    a5 = float(np.mean([y_test[i] in y_train[top_idx[i]] for i in range(len(y_test))]))
    print(f"{label} 1-NN retrieval: accuracy@1 = {a1:.4f}, accuracy@5 = {a5:.4f}  (no training; search {search_s:.1f}s)")
    results.append({"method": f"{label} nearest-neighbour retrieval", "accuracy@1": round(a1, 4),
                    "accuracy@5": round(a5, 4), "train_seconds": 0.0})

results_df = pl.DataFrame(results).sort("accuracy@1", descending=True)
results_df

MiniLM 1-NN retrieval: accuracy@1 = 0.3452, accuracy@5 = 0.5738  (no training; search 1.9s)

JobBERT-v3 1-NN retrieval: accuracy@1 = 0.3939, accuracy@5 = 0.6323  (no training; search 2.4s)

shape: (7, 4)

method	accuracy@1	accuracy@5	train_seconds
str	f64	f64	f64
"JobBERT-v3 embeddings + LogReg"	0.4515	0.726	159.4
"MiniLM embeddings + LogReg"	0.4065	0.667	118.8
"JobBERT-v3 nearest-neighbour r…	0.3939	0.6323	0.0
"MiniLM nearest-neighbour retri…	0.3452	0.5738	0.0
"TF-IDF + linear SGD"	0.3424	0.4042	74.2
"baseline: majority class"	0.0462	null	0.0
"baseline: random guess"	0.00089	null	0.0

7. Where each approach wins: head classes vs the long tail

The aggregate number hides the real story. Split the test set by how many training examples each class had, and the trade-off from section 1 becomes visible: classification needs examples to learn a class; retrieval does not.

train_class_counts = dict(zip(*np.unique(y_train, return_counts=True)))
bucket = np.array([
    "1-2 examples" if train_class_counts.get(c, 0) <= 2
    else "3-10 examples" if train_class_counts.get(c, 0) <= 10
    else "11+ examples"
    for c in y_test
])

Xtr, Xte = emb_cache["JobBERT-v3"]
clf = LogisticRegression(max_iter=1000, C=10.0, n_jobs=1).fit(Xtr, y_train)
clf_pred = clf.predict(Xte)
top_idx, _ = common.batched_topk(Xte, Xtr, k=1)
nn_pred = y_train[top_idx[:, 0]]
tfidf_pred = tfidf_clf.predict(X_test_txt)

rows = []
for b in ["1-2 examples", "3-10 examples", "11+ examples"]:
    m = bucket == b
    if m.sum() == 0:
        continue
    rows.append({
        "training examples for the true class": b,
        "test rows": int(m.sum()),
        "TF-IDF classifier": round(float((tfidf_pred[m] == y_test[m]).mean()), 4),
        "JobBERT-v3 classifier": round(float((clf_pred[m] == y_test[m]).mean()), 4),
        "JobBERT-v3 retrieval (1-NN)": round(float((nn_pred[m] == y_test[m]).mean()), 4),
    })
tail_df = pl.DataFrame(rows)
tail_df

shape: (3, 5)

training examples for the true class	test rows	TF-IDF classifier	JobBERT-v3 classifier	JobBERT-v3 retrieval (1-NN)
str	i64	f64	f64	f64
"1-2 examples"	318	0.1384	0.0566	0.1887
"3-10 examples"	819	0.2112	0.221	0.2552
"11+ examples"	5271	0.4033	0.5111	0.4278

fig, ax = plt.subplots(1, 2, figsize=(14, 4.5))
r = results_df.filter(pl.col("accuracy@1") > 0.001).sort("accuracy@1")
colors = ["#B0B0B0" if "baseline" in m else "#4C78A8" for m in r["method"]]
ax[0].barh(r["method"].to_list(), r["accuracy@1"].to_list(), color=colors)
ax[0].set_xlabel("accuracy@1"); ax[0].grid(alpha=.3, axis="x")
ax[0].set_title(f"Predicting 1 of {len(vals):,} ESCO codes\n({len(y_test):,} test titles, majority baseline {maj_acc:.1%})")

x = np.arange(tail_df.height); w = 0.26
for i, (col, colr) in enumerate([("TF-IDF classifier", "#72B7B2"),
                                 ("JobBERT-v3 classifier", "#4C78A8"),
                                 ("JobBERT-v3 retrieval (1-NN)", "#F58518")]):
    ax[1].bar(x + (i - 1) * w, tail_df[col].to_list(), w, label=col, color=colr)
ax[1].set_xticks(x); ax[1].set_xticklabels(tail_df["training examples for the true class"].to_list())
ax[1].set_ylabel("accuracy@1"); ax[1].legend(fontsize=8); ax[1].grid(alpha=.3, axis="y")
ax[1].set_title("Classification needs examples.\nRetrieval does not.")
plt.tight_layout(); plt.savefig(os.path.join(RESULTS, "classification_vs_retrieval.png"), dpi=110, bbox_inches="tight")
plt.show()

8. Error analysis — real mistakes, with the label noise called out

esco_name = dict(zip(all_df["esco_URI"].to_list(), all_df["esco_job_title"].to_list()))
wrong = np.where(clf_pred != y_test)[0]
rng2 = np.random.default_rng(7)
print(f"{len(wrong):,} of {len(y_test):,} test rows misclassified. A random sample:\n")
for i in rng2.choice(wrong, size=min(10, len(wrong)), replace=False):
    print(f"  title      : {X_test_txt[i]!r}")
    print(f"    predicted: {esco_name.get(clf_pred[i], '?')}")
    print(f"    labelled : {esco_name.get(y_test[i], '?')}")
    print()

3,515 of 6,408 test rows misclassified. A random sample:

  title      : 'System Implementation Consultant (RM6K - RM8K)'
    predicted: software developer
    labelled : project manager

  title      : 'System Analyst (Malaysia)'
    predicted: data analyst
    labelled : ICT system analyst

  title      : 'Paint Color Shader '
    predicted: transport equipment painter
    labelled : colour sampling operator

  title      : 'ANTI SURGE CONTROL FSE'
    predicted: financial controller
    labelled : petroleum engineer

  title      : 'Fabrication Production Leader'
    predicted: factory hand
    labelled : metal product quality control inspector

  title      : 'Digital Viral Marketing'
    predicted: online sales channel manager
    labelled : advertising assistant

  title      : 'Corrugator Supervisor'
    predicted: machine operator supervisor
    labelled : corrugator operator

  title      : 'Solar System Installer'
    predicted: solar energy technician
    labelled : building construction worker

  title      : 'Marketing Lecturer'
    predicted: marketing assistant
    labelled : university teaching assistant

  title      : 'Food Technologist/QAQC '
    predicted: food safety inspector
    labelled : food technologist

Read those carefully before treating the accuracy number as the model's true quality. A recurring pattern: the "wrong" prediction is often defensible and the ground-truth label is itself questionable — this dataset was auto-collected from a job board, and rows like "SALES EXECUTIVE MANAGER" labelled marketing assistant are noise in the answer key, not model failure.

What that means for the numbers: the accuracies here are a lower bound. We do not know exactly how much of the gap is label noise without hand-checking a sample, and we have not done that — so we are not going to claim a corrected figure. Flagging the ceiling honestly is worth more than inventing an adjustment.

results_df.write_csv(os.path.join(RESULTS, "classification_results.csv"))
tail_df.write_csv(os.path.join(RESULTS, "classification_by_class_frequency.csv"))
print("Wrote results/classification_results.csv and results/classification_by_class_frequency.csv")

Wrote results/classification_results.csv and results/classification_by_class_frequency.csv

9. What this notebook established

• On a fixed label space of ~1,300 ESCO codes, with real messy vacancy titles as input, classification and retrieval were measured on identical data with an identical, leakage-checked, dedup'd split.
• Every number sits next to a majority-class baseline and a random-guess baseline, with test-row counts stated.
• The shipped val/test split was found to share titles across both halves; we built our own clean split rather than inheriting that inflation. This is exactly the class of bug that sank earlier projects in this repo's family.
• The head/tail breakdown shows when each approach wins, which is the practical answer to "is my label space fixed enough for classification?"

Caveats stated plainly: this dataset is English-only and single-source (one government job board), so these numbers do not automatically transfer to other languages or job boards. The labels contain real noise. And like notebook 05, this measures matching/normalisation, not validation — every input here is already known to be a real job title, so nothing in this notebook can tell you whether a model can reject "New York City" or "Acme Holdings LLC".