data: BLS OEWS and CES series ingest for PFS-relevant occupations and health-care employment #695

New Issue

kert · 2026-09-09T14:30:05Z

kert commented

2026-09-09 14:30:05 +00:00

Spec docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md (§P50).

Ingest BLS OEWS (occupation × area wages: 29-1xxx physicians/NPs/PAs, 29-2xxx health technologists, 31-9xxx medical assistants and other clinical support — the actor element's "clinical staff") and CES (health-care employment, ambulatory settings) into bls.oews / bls.ces with the BLS series ids kept as provenance and bib items per release. Care-management families pay for clinical-staff time (99490 = 20 min clinical staff; APCM actor = clinical staff directed by the practitioner), so the wage series for those occupations is the first comparison.

Done when the series are queryable by year and occupation, cited in bib, and a notebook overlays 31-9092 wages on the CCM/APCM exposure calendar.

Spec docs/superpowers/specs/2026-09-09-code-family-longitudinal-design.md (§P50). Ingest BLS OEWS (occupation × area wages: 29-1xxx physicians/NPs/PAs, 29-2xxx health technologists, 31-9xxx medical assistants and other clinical support — the `actor` element's "clinical staff") and CES (health-care employment, ambulatory settings) into `bls.oews` / `bls.ces` with the BLS series ids kept as provenance and bib items per release. Care-management families pay for clinical-staff time (99490 = 20 min clinical staff; APCM `actor` = clinical staff directed by the practitioner), so the wage series for those occupations is the first comparison. Done when the series are queryable by year and occupation, cited in bib, and a notebook overlays 31-9092 wages on the CCM/APCM exposure calendar.

kert added this to the P50: Legal epidemiology — outcome series around code-family events (utilization, BLS wages, industry markets) milestone 2026-09-09 14:30:05 +00:00

kert added the bib data labels 2026-09-09 14:30:05 +00:00

kert referenced this issue from a commit

2026-09-22 20:26:23 +00:00

feat(bls): OEWS occupation wages + CES health-care employment — bls.oews / bls.ces, stack bls oews|ces, notebook 7f (refs #695)

kert referenced this issue from a commit

2026-09-22 20:26:23 +00:00

merge: #695 — BLS OEWS wages + CES health-care employment: bls.oews / bls.ces, stack bls, notebook 7f (refs #695)

kert referenced this issue from a commit

2026-09-22 20:29:40 +00:00

fix(bls): upper-case the OEWS sheet headers — the May 2019 release ships them lower-case, which loaded 0 occupations (refs #695)

kert commented

2026-09-22 20:30:49 +00:00

Merged to main as eb90484 (feature b1e06b1; follow-up 2d5d12b for the 2019 file's lower-case headers) and loaded the same day.

OEWS (bls.oews) — the BLS API only serves the latest OEWS year, so history comes from the annual oesm{yy}nat.zip files on bls.gov (2013–2024). Those refuse plain scripted downloads (403) but serve a request carrying a full browser header set, which bls.oews.download sends; the zips are cached at data/bls/oews/. Each national_M{year}_dl sheet (xls in 2013, xlsx after; OCC_GROUP through 2018 under the 2010 SOC, O_GROUP + area/industry columns from 2019 under the 2018 SOC, lower-case headers in 2019 only) is normalised to one row per SOC occupation: employment, mean/median hourly and annual wage, 10/25/75/90th percentiles, with */#/** suppression as NULL and the BLS series-id prefix (OEUN000000000000 + occupation digits) on every row. Loaded: 15,277 + 1,329 = 16,606 rows, 12 releases; one bib Source per release (PUNS6S4N … XPDSWMK8; module:bls, table:bls.oews, source:bls-website, year:N) and a cms.ingest_log row per release with the zip's sha256.

CES (bls.ces) — a closed registry of employment (…01, thousands, SA) and average hourly earnings (…03) series for health care, ambulatory care, offices of physicians, home health, hospitals and nursing/residential care (CES65<industry><type>), pulled through the public API in 10-year × 25-series chunks (BLS_API_KEY widens to 20 × 50; none is set). Loaded: 2,868 monthly observations, 12 series, 2006–2025; one bib Source per series (url data.bls.gov/timeseries/<id>), series id on every row.

CLI: stack bls oews [--year]... [--write [--offline]] [--occ 31-9092]... and stack bls ces [--write] [--series ID]... [--annual]. Notebook code_families.py §7f overlays clinical-staff wages (31-9092 medical assistants, 29-2061 LPN/LVN, 29-1141 RN, 29-1171 NP) and physician-office / ambulatory employment on the family's becomes-payable/ends dates from §7c; provenance counts include both tables.

First comparison (31-9092, the CCM "clinical staff" occupation): mean annual wage $30,780 (2013) → $34,540 (2018) → $36,930 (2020) → $40,700 (2022) → $44,720 (2024), employment 572k → 793k; offices-of-physicians employment (CES6562110001, SA annual mean) 2,732k (2021) → 3,008k (2025).

Tests: 28 unit (tests/bls/), 7 CLI (tests/cli/test_bls_cli.py), notebook structure + live run green; pre-commit suites (bls, cli, notebooks) passed.

**Merged to main as eb90484** (feature b1e06b1; follow-up 2d5d12b for the 2019 file's lower-case headers) and loaded the same day. **OEWS (`bls.oews`)** — the BLS API only serves the *latest* OEWS year, so history comes from the annual `oesm{yy}nat.zip` files on bls.gov (2013–2024). Those refuse plain scripted downloads (403) but serve a request carrying a full browser header set, which `bls.oews.download` sends; the zips are cached at `data/bls/oews/`. Each `national_M{year}_dl` sheet (xls in 2013, xlsx after; `OCC_GROUP` through 2018 under the 2010 SOC, `O_GROUP` + area/industry columns from 2019 under the 2018 SOC, lower-case headers in 2019 only) is normalised to one row per SOC occupation: employment, mean/median hourly and annual wage, 10/25/75/90th percentiles, with `*`/`#`/`**` suppression as NULL and the BLS series-id prefix (`OEUN000000000000` + occupation digits) on every row. Loaded: **15,277 + 1,329 = 16,606 rows, 12 releases**; one bib Source per release (PUNS6S4N … XPDSWMK8; `module:bls`, `table:bls.oews`, `source:bls-website`, `year:N`) and a `cms.ingest_log` row per release with the zip's sha256. **CES (`bls.ces`)** — a closed registry of employment (`…01`, thousands, SA) and average hourly earnings (`…03`) series for health care, ambulatory care, offices of physicians, home health, hospitals and nursing/residential care (`CES65<industry><type>`), pulled through the public API in 10-year × 25-series chunks (`BLS_API_KEY` widens to 20 × 50; none is set). Loaded: **2,868 monthly observations, 12 series, 2006–2025**; one bib Source per series (url `data.bls.gov/timeseries/<id>`), series id on every row. **CLI**: `stack bls oews [--year]... [--write [--offline]] [--occ 31-9092]...` and `stack bls ces [--write] [--series ID]... [--annual]`. **Notebook** `code_families.py` §7f overlays clinical-staff wages (31-9092 medical assistants, 29-2061 LPN/LVN, 29-1141 RN, 29-1171 NP) and physician-office / ambulatory employment on the family's `becomes-payable`/`ends` dates from §7c; provenance counts include both tables. **First comparison** (31-9092, the CCM "clinical staff" occupation): mean annual wage $30,780 (2013) → $34,540 (2018) → $36,930 (2020) → $40,700 (2022) → $44,720 (2024), employment 572k → 793k; offices-of-physicians employment (CES6562110001, SA annual mean) 2,732k (2021) → 3,008k (2025). Tests: 28 unit (`tests/bls/`), 7 CLI (`tests/cli/test_bls_cli.py`), notebook structure + live run green; pre-commit suites (bls, cli, notebooks) passed.

kert closed this issue

2026-09-22 20:30:49 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#695