01 — Data Exploration¶
What's actually in our two raw sources (ESCO occupations, multilingual; O*NET
occupations, English-only US government taxonomy) before we build anything on top of
them. See data/raw/DATA_SOURCES.md for exactly how each was fetched.
Questions this notebook answers:
- How many occupations, and how much multilingual coverage does ESCO actually have?
- How long are job titles, and does that vary a lot by language?
- How many synonyms (alternative labels) does the average occupation have?
- How much do ESCO and O*NET already agree on English titles, just as strings — this previews why a naive taxonomy-lookup baseline (notebook 02) will struggle.
In [1]:
import polars as pl
import matplotlib.pyplot as plt
from common import normalize
# reference palette (see dataviz skill: references/palette.md) — sequential blue for
# single-series magnitude bars/histograms throughout this notebook
BLUE = "#2a78d6"
ORANGE = "#eb6834"
INK = "#0b0b0b"
SECONDARY_INK = "#52514e"
MUTED = "#898781"
GRID = "#e1e0d9"
SURFACE = "#fcfcfb"
plt.rcParams.update({
"figure.facecolor": SURFACE,
"axes.facecolor": SURFACE,
"axes.edgecolor": GRID,
"axes.labelcolor": SECONDARY_INK,
"text.color": INK,
"xtick.color": SECONDARY_INK,
"ytick.color": SECONDARY_INK,
"axes.grid": True,
"grid.color": GRID,
"grid.linewidth": 0.8,
"font.size": 10,
})
DATA = "../data/processed"
esco_occ = pl.read_csv(f"{DATA}/esco_occupations.csv")
esco_titles = pl.read_csv(f"{DATA}/esco_titles_long.csv")
onet_occ = pl.read_csv(f"{DATA}/onet_occupations.csv")
onet_titles = pl.read_csv(f"{DATA}/onet_titles_long.csv")
print(f"ESCO occupations: {esco_occ.height}")
print(f"ESCO title rows (all languages, preferred+alternative): {esco_titles.height}")
print(f"O*NET occupations: {onet_occ.height}")
print(f"O*NET title rows (job_titles + reported_titles): {onet_titles.height}")
ESCO occupations: 1699 ESCO title rows (all languages, preferred+alternative): 213621 O*NET occupations: 1016 O*NET title rows (job_titles + reported_titles): 65496
1. ESCO language coverage¶
Our working assumption going in was "low-resource ESCO languages (Irish, Icelandic, Maltese) are more sparsely covered than the EU's larger languages." The actual numbers below partly falsify that — we're stating this plainly rather than quietly editing the assumption out, per this project's "report honest numbers even if they contradict what you expected" rule.
In [2]:
lang_coverage = (
esco_titles.filter(pl.col("label_type") == "preferred")
.group_by("language")
.agg(pl.len().alias("n_occupations_with_label"))
.sort("n_occupations_with_label", descending=True)
)
total_occ = esco_occ.height
lang_coverage = lang_coverage.with_columns(
(pl.col("n_occupations_with_label") / total_occ * 100).round(1).alias("pct_coverage")
)
lang_coverage
Out[2]:
shape: (28, 3)
| language | n_occupations_with_label | pct_coverage |
|---|---|---|
| str | u32 | f64 |
| "en-us" | 1699 | 100.0 |
| "sl" | 1699 | 100.0 |
| "sv" | 1699 | 100.0 |
| "el" | 1699 | 100.0 |
| "et" | 1699 | 100.0 |
| … | … | … |
| "lt" | 1699 | 100.0 |
| "lv" | 1699 | 100.0 |
| "sk" | 1699 | 100.0 |
| "en" | 1699 | 100.0 |
| "ar" | 1699 | 100.0 |
Finding #1 (preferred labels): coverage is uniformly 100% across all 28
language/variant codes. Every one of the 1699 occupations has a preferred label in
every language, including small languages like Irish (ga) and Icelandic (is). This
makes sense once you think about what ESCO is: it's a professionally curated,
centrally-translated EU taxonomy (not crowdsourced), so the "core" preferred label is a
translation deliverable, not an organic contribution — of course it's complete.
Finding #2 (alternative labels / synonyms): coverage is highly uneven, which is the real language-richness story here. We check this below instead.
In [3]:
alt_coverage = (
esco_titles.filter(pl.col("label_type") == "alternative")
.group_by("language")
.agg(pl.col("uri").n_unique().alias("n_occupations_with_alt_label"))
.with_columns((pl.col("n_occupations_with_alt_label") / total_occ * 100).round(1).alias("pct_coverage"))
.sort("n_occupations_with_alt_label", descending=True)
)
alt_coverage
Out[3]:
shape: (26, 3)
| language | n_occupations_with_alt_label | pct_coverage |
|---|---|---|
| str | u32 | f64 |
| "hr" | 1699 | 100.0 |
| "lt" | 1699 | 100.0 |
| "sk" | 1698 | 99.9 |
| "lv" | 1693 | 99.6 |
| "de" | 1685 | 99.2 |
| … | … | … |
| "sv" | 847 | 49.9 |
| "et" | 303 | 17.8 |
| "ga" | 283 | 16.7 |
| "is" | 264 | 15.5 |
| "no" | 34 | 2.0 |
In [4]:
fig, ax = plt.subplots(figsize=(9, 8))
ac = alt_coverage.sort("n_occupations_with_alt_label")
ax.barh(ac["language"], ac["n_occupations_with_alt_label"], color=BLUE, height=0.7)
ax.axvline(total_occ, color=MUTED, linewidth=1, linestyle="--")
ax.text(total_occ, -0.8, f" {total_occ} = 100%", color=MUTED, fontsize=9, va="top")
ax.set_xlabel(f"# ESCO occupations with >=1 alternative label (of {total_occ} total)")
ax.set_title("ESCO alternative-label (synonym) coverage by language — the real coverage gap", loc="left", fontweight="bold")
ax.spines[["top", "right"]].set_visible(False)
plt.tight_layout()
plt.savefig("../results/esco_language_coverage.png", dpi=110)
plt.show()
2. Title length distribution¶
How long are job titles, in words, and does it vary by language? (Some languages — German especially — famously compound words, so word-count and character-count can diverge a lot.)
In [5]:
sample_langs = ["en", "de", "fr", "es", "pl"]
len_df = (
esco_titles.filter((pl.col("label_type") == "preferred") & pl.col("language").is_in(sample_langs))
.with_columns([
pl.col("title").str.split(" ").list.len().alias("n_words"),
pl.col("title").str.len_chars().alias("n_chars"),
])
)
len_df.group_by("language").agg([
pl.col("n_words").mean().round(2).alias("avg_words"),
pl.col("n_chars").mean().round(1).alias("avg_chars"),
pl.col("n_words").max().alias("max_words"),
]).sort("language")
Out[5]:
shape: (5, 4)
| language | avg_words | avg_chars | max_words |
|---|---|---|---|
| str | f64 | f64 | u32 |
| "de" | 2.23 | 44.1 | 17 |
| "en" | 2.52 | 20.5 | 8 |
| "es" | 6.79 | 55.0 | 31 |
| "fr" | 5.67 | 51.7 | 23 |
| "pl" | 3.2 | 28.3 | 12 |
In [6]:
fig, axes = plt.subplots(1, len(sample_langs), figsize=(15, 3), sharey=True)
for ax, lang in zip(axes, sample_langs):
vals = len_df.filter(pl.col("language") == lang)["n_words"]
ax.hist(vals, bins=range(1, 12), color=BLUE, edgecolor=SURFACE, linewidth=0.5)
ax.set_title(lang, fontsize=11)
ax.spines[["top", "right"]].set_visible(False)
fig.suptitle("Title length in words, by language (ESCO preferred labels)", x=0.13, ha="left", fontweight="bold")
plt.tight_layout()
plt.savefig("../results/title_length_by_language.png", dpi=110)
plt.show()
3. Synonym richness (alternative labels)¶
Many occupations have several alternative titles per language (e.g. "software developer" / "software engineer" / "programmer" as near-synonyms). This is exactly the surface-form variation that makes plain string matching hard, and that embedding-based matching (notebook 03) is supposed to handle better.
In [7]:
alt_counts_en = (
esco_titles.filter((pl.col("label_type") == "alternative") & (pl.col("language") == "en"))
.group_by("uri")
.agg(pl.len().alias("n_alt_labels_en"))
)
# occupations with zero English alt labels don't appear above; join back to get the full distribution
alt_full = esco_occ.select("uri").join(alt_counts_en, on="uri", how="left").with_columns(
pl.col("n_alt_labels_en").fill_null(0)
)
print(alt_full["n_alt_labels_en"].describe())
fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(alt_full["n_alt_labels_en"], bins=range(0, 20), color=BLUE, edgecolor=SURFACE, linewidth=0.5)
ax.set_xlabel("# English alternative labels per occupation")
ax.set_ylabel("# occupations")
ax.set_title("Synonym richness — English alt labels per ESCO occupation", loc="left", fontweight="bold")
ax.spines[["top", "right"]].set_visible(False)
plt.tight_layout()
plt.savefig("../results/alt_label_richness.png", dpi=110)
plt.show()
shape: (9, 2) ┌────────────┬──────────┐ │ statistic ┆ value │ │ --- ┆ --- │ │ str ┆ f64 │ ╞════════════╪══════════╡ │ count ┆ 1699.0 │ │ null_count ┆ 0.0 │ │ mean ┆ 9.649794 │ │ std ┆ 6.135908 │ │ min ┆ 0.0 │ │ 25% ┆ 5.0 │ │ 50% ┆ 8.0 │ │ 75% ┆ 12.0 │ │ max ┆ 47.0 │ └────────────┴──────────┘
4. ISCO major group distribution¶
ESCO occupations are organized under the 10 ISCO-08 major groups (1-digit codes).
In [8]:
isco_names = {
"0": "0 Armed forces", "1": "1 Managers", "2": "2 Professionals",
"3": "3 Technicians & assoc. professionals", "4": "4 Clerical support",
"5": "5 Service & sales", "6": "6 Skilled agri/forestry/fishery",
"7": "7 Craft & related trades", "8": "8 Plant/machine operators",
"9": "9 Elementary occupations",
}
major_dist = (
esco_occ.group_by("isco_major").agg(pl.len().alias("n"))
.with_columns(pl.col("isco_major").cast(pl.Utf8))
.sort("isco_major")
)
major_dist = major_dist.with_columns(
pl.col("isco_major").map_elements(lambda x: isco_names.get(x, x), return_dtype=pl.Utf8).alias("group_name")
)
major_dist
Out[8]:
shape: (10, 3)
| isco_major | n | group_name |
|---|---|---|
| str | u32 | str |
| "0" | 21 | "0 Armed forces" |
| "1" | 117 | "1 Managers" |
| "2" | 398 | "2 Professionals" |
| "3" | 309 | "3 Technicians & assoc. profess… |
| "4" | 73 | "4 Clerical support" |
| "5" | 99 | "5 Service & sales" |
| "6" | 43 | "6 Skilled agri/forestry/fisher… |
| "7" | 290 | "7 Craft & related trades" |
| "8" | 282 | "8 Plant/machine operators" |
| "9" | 67 | "9 Elementary occupations" |
In [9]:
fig, ax = plt.subplots(figsize=(8, 5))
md = major_dist.sort("n")
ax.barh(md["group_name"], md["n"], color=BLUE, height=0.6)
ax.set_xlabel("# ESCO occupations")
ax.set_title("ESCO occupations by ISCO-08 major group", loc="left", fontweight="bold")
ax.spines[["top", "right"]].set_visible(False)
plt.tight_layout()
plt.savefig("../results/isco_major_group_distribution.png", dpi=110)
plt.show()
5. O*NET overview¶
In [10]:
titles_per_occ = onet_titles.group_by("onetsoc_code").agg(pl.len().alias("n_titles"))
print(f"O*NET: {onet_occ.height} formal occupations")
print(f"O*NET: {onet_titles.height} total alternate/reported titles "
f"({titles_per_occ['n_titles'].mean():.1f} avg per occupation)")
onet_titles.group_by("source").agg(pl.len().alias("n")).sort("source")
O*NET: 1016 formal occupations O*NET: 65496 total alternate/reported titles (64.5 avg per occupation)
Out[10]:
shape: (2, 2)
| source | n |
|---|---|
| str | u32 |
| "job_titles" | 57543 |
| "reported_titles" | 7953 |
6. How much do ESCO and O*NET already agree, as plain strings?¶
This is the key preview for notebook 02. We normalize both sides (lowercase, strip punctuation/whitespace) and check exact-string overlap between:
- ESCO English titles (preferred + alternative labels)
- O*NET titles (formal occupation titles + alternate titles + self-reported titles)
Both taxonomies claim to describe "all occupations in the labor market", built by completely independent organizations (EU Commission vs. US Dept. of Labor). If they don't even agree as raw strings on titles that plausibly describe the same jobs, that's a real signal about how much surface-form variation exists in this domain even before we get to translation across languages.
In [11]:
esco_en_titles = (
esco_titles.filter(pl.col("language") == "en")
.with_columns(pl.col("title").map_elements(normalize, return_dtype=pl.Utf8).alias("norm"))
["norm"].unique()
)
onet_all_titles = pl.concat([
onet_occ.select(pl.col("title")),
onet_titles.select(pl.col("title")),
]).with_columns(pl.col("title").map_elements(normalize, return_dtype=pl.Utf8).alias("norm"))["norm"].unique()
esco_set = set(esco_en_titles.to_list())
onet_set = set(onet_all_titles.to_list())
overlap = esco_set & onet_set
print(f"Unique normalized ESCO English titles (preferred+alt): {len(esco_set):,}")
print(f"Unique normalized O*NET titles (formal+alt+reported): {len(onet_set):,}")
print(f"Exact-string overlap (normalized): {len(overlap):,}")
print(f"Overlap as % of ESCO titles: {len(overlap) / len(esco_set) * 100:.1f}%")
print(f"Overlap as % of O*NET titles: {len(overlap) / len(onet_set) * 100:.1f}%")
print()
print("Sample of titles that DO match exactly:")
for t in sorted(overlap)[:10]:
print(" -", t)
print()
only_onet_sample = sorted(onet_set - esco_set)[:10]
print("Sample of O*NET titles with NO exact ESCO match (these are what notebook 02/03 need to handle):")
for t in only_onet_sample:
print(" -", t)
Unique normalized ESCO English titles (preferred+alt): 17,355 Unique normalized O*NET titles (formal+alt+reported): 47,580 Exact-string overlap (normalized): 3,146 Overlap as % of ESCO titles: 18.1% Overlap as % of O*NET titles: 6.6% Sample of titles that DO match exactly: - able seaman - academic advisor - accompanist - account auditor - account information clerk - account receivable clerk - accountant - accountant assistant - accounting assistant - accounting bookkeeper Sample of O*NET titles with NO exact ESCO match (these are what notebook 02/03 need to handle): - 3d animator three dimensional animator - 3d artist three dimensional artist - 3d designer three dimensional designer - 3d modeler three dimensional modeler - 3d printing tech three dimensional printing technician - 3d specialist three dimensional specialist - 3d technologist - 4 h agent - 4 h club agent - 4 h youth development educator
Summary¶
- ESCO's preferred-label coverage is uniformly 100% across all 28 languages (it's a curated translation deliverable, not organic) — our prior assumption that it would be uneven was wrong, and we're reporting that rather than hiding it. The real coverage gap is in alternative labels (synonyms): coverage there ranges enormously by language (see chart above for exact numbers — read off the chart for anything quoted in README.md), which matters because alt labels are exactly the "messy real-world synonym" signal a matcher needs.
- Titles are short (a handful of words) across all sampled languages, with the expected German compounding showing up as slightly higher average word count per title... or not — see the actual numbers above rather than assuming.
- Occupations average several alternative English labels each — real synonym variation that a rigid exact-match lookup can't handle by definition.
- Exact-string overlap between ESCO and O*NET, even just in English, is far from 100% — this sets the ceiling for how well any lookup-based approach could possibly do when tested against an independent source, before we even measure it directly in notebook 02.