docs(plan): market tier — EDGAR entities, Tiingo prices, LDA, FEC, Congress members/committees/bills, district footprint, event windows, abnormal returns, enrichment
Some checks failed
CI / lint (push) Successful in 1m18s
CI / test (push) Failing after 4m4s
CI / notebooks-smoke (push) Successful in 2m58s
Deploy / changes (push) Successful in 18s
Infra CI / zotero (push) Successful in 26s
Infra CI / notebooks (push) Successful in 1m10s
Infra CI / docs (push) Successful in 1m39s
Infra CI / api (push) Successful in 1m24s
Infra CI / llm (push) Successful in 1m2s
Infra CI / mc (push) Successful in 1m29s
Deploy / notebooks (push) Has been skipped
Infra CI / bcda (push) Successful in 3m4s
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Successful in 1m23s
Deploy / api (push) Successful in 2m4s
Deploy / bcda (push) Successful in 2m7s
Deploy / mc (push) Has been skipped
Deploy / report (push) Has been cancelled
Deploy / llm (push) Has been cancelled

This commit is contained in:
kert
2026-11-12 21:31:06 -05:00
parent a33101a7d1
commit 6226341ff9
2 changed files with 966 additions and 2 deletions

View File

@@ -0,0 +1,964 @@
# Market and enrichment tier (slice S4b) Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Track how the money CMS law moves shows up in markets and in power: a cited map of public companies (and lobbying organizations) exposed to each code family and specialty, their daily prices against SPY/XLV, lobbying (LDA), PAC money (FEC), Congress members, committees, bills and district footprints, event windows (rule releases, RUC/Panel meetings, seat tenures), a market-model abnormal-return engine with portfolios, and enrichment readers that put BLS, FRED, SOI and PSPS beside the market series. This closes #797.
**Architecture:** A new `market` package (DuckDB schemas `market` and `econ`) with one module per source. Each module follows the PSPS/seats pattern: an honest-UA client with a per-source pacer, a hoarded raw cache under `data/market/`, pure parsers to frozen dataclasses, delete-then-insert writers in `market/tables.py`, a bib citation, a `cms.ingest_log` row and a `gap:` line when a source refuses or a key is missing. A single `stack market` CLI group drives fetching (no DB lock held during network work), then short `duckdb_batch` writes and a replica publish. Analysis is pure library code: `market/abnormal.py` (market model, CAR, analytic and bootstrap intervals, sharing the bootstrap helpers extracted from `pfs/event_study.py`), `market/portfolio.py` and `market/enrich.py` readers. There is no UI; the S4 Evaluate view reads these tables, and the #797 notebook panel is the only display.
**Tech Stack:** Python 3.13, uv, httpx, DuckDB (`data/aco.duckdb` via `conf.connect.duckdb_batch("aco")`, replica `data/replica/aco.ro.duckdb`), bib SQLite (`fr_anchors`, `fr_anchor_docs`, `items`), numpy (already locked; imported lazily), PyYAML, typer, pytest + xdist, the local Ollama pool for the one optional classification step.
**Spec:** `docs/superpowers/specs/2026-11-12-cms-legal-epi-platform-design.md`, sections "Market and enrichment tier" and "Congress and districts" (commit c1288c2), slice S4b. Stakes context: `docs/superpowers/specs/2026-11-11-code-family-dossier-design.md` § Stakes. User decisions of 2026-11-12 are in `.superpowers/sdd/2026-11-11-code-family-dossier-home-visits/progress.md` (lines 428–433). Issue #797 (market series) is resolved by Tasks 3 and 15.
## Global Constraints
- Commit straight to `main` in `/home/kert/stack`. No branches, worktrees, merge commits, PRs, `git stash`. Before every commit run `git status --short`; if anything you did not stage is staged, stop and ask (another session shares this checkout). `git add` explicit paths only.
- Commit messages: conventional prefix (`feat(market): …`); no `Co-Authored-By` or any other trailer.
- Never run a write/ingest command from a worktree. Every live command runs from the main checkout through the env wrapper: `bash -c 'cd /home/kert/stack && set -a && . ./.env && set +a && uv run --no-sync <cmd>'` (written **ENV** below). Before the first live run: `cat .venv/lib/python3*/site-packages/stack.pth` must print `/home/kert/stack/src` (else `uv sync --frozen`). Long runs: `mkdir -p .state/market` once, then `setsid nohup ENV … > .state/market/<task>.log 2>&1 &` with a Monitor on the log; rerunning the same command resumes.
- DuckDB: write `data/aco.duckdb` only inside `conf.connect.duckdb_batch("aco")`, opened after all network work (fetch first, then a short batch that only writes), then `conf.connect.publish_replica("aco")`. `market.*` and `econ.*` modules never open a database at import.
- New tables: DDL in `market.tables._DDL`, a frozen dataclass, a delete-then-insert writer and a reader in `src/market/tables.py`; each table gets a round-trip case in `tests/market/test_tables.py::ROUNDTRIP`.
- **User-Agent** on every market client: `stack-pfs-dossier/1.0 (ci@mail.fhirworx.io)` (`market.http.USER_AGENT`). SEC requires the contact; it is the stack's own mailbox, never a personal address. Never a browser UA. A 401/403, an exhausted 429 or a bot/challenge page stops that source and becomes a `gap:` line plus a `market.source_gap` row (with `--write`); never work around it.
- **Keys** (all accounts registered to ci@mail.fhirworx.io, user rule; values only in `.env`):
- `TIINGO_API_KEY` (present, verified 2026-11-12);
- `API_DATA_GOV_KEY` (pending, the controller captures it from the mailbox), serving openFEC and Congress.gov, with optional overrides `FEC_API_KEY` and `CONGRESS_API_KEY`;
- `LDA_API_KEY` (optional, lda.gov account; anonymous access works);
- `HUD_API_TOKEN` (pending, HUD USPS API).
- Keys travel only in headers (`Authorization: Token …`, `X-Api-Key: …`, `Authorization: Bearer …`), never in a URL, cache path, bib item, ingest_log row, log line or exception text.
- A task whose key is missing prints the gap, records it, writes nothing else, exits 0, and stays fully testable offline.
- **Hoard:** never overwrite a source file. Raw responses live under `data/market/<source>/…`; a re-download with different bytes lands beside the old file as `<stem>.<sha8><suffix>` (`pfs.seats.save_snapshot`). Readers use the newest copy (`pfs.psps.newest_cache`).
- **Prices are internal-use only** (Tiingo Starter licence: "you may not display or share the data with another person or organization"):
- they stay in `data/market/tiingo/` and the `market.price_daily`/`market.series` tables;
- never a bib attachment (Zotero syncs), never in git, never in a commit message, issue comment or report beyond row counts;
- test fixtures use synthetic prices only.
- Public data only. No material non-public information. No ACO/HarmonyCares/WNOT/Wakely material; the seed list never names them.
- Local models only, and only in Task 2's optional passage classifier: Ollama via `llm.pool.HostPool` / `llm.structured.json_extractor` with `cfg.instruct_model`, resumable through `pfs.runlog` with `--limit`/`--dry-run`. Never a cloud LLM API.
- No AMA text: a stored SEC passage is ≤ 400 characters of public filing text and is withheld when it shares an 8-word run (`pfs.origin.shares_run`) with any `pfs.cpt_code.descriptor` of a code it names.
- People: store names of members of Congress (public officials) and organizations as the public record names them. Do not store FEC Schedule A individual donors or LDA lobbyist personal names (not needed).
- Market results are associations, not causal claims. Every stored result row carries its caveats.
- `cli/market.py` imports only stdlib, typer and httpx at module level (the api image has the `cli` extra only). numpy, duckdb and yaml are imported inside functions.
- Tests need no network, live DuckDB, Ollama or keys (httpx `MockTransport`, `duckdb.connect(":memory:")` + `ensure_tables`, `Store(":memory:")`, monkeypatched `cli.market._batch/_read/_store/_publish`, as in `tests/cli/test_pfs_cli.py`). Live tests skip without the replica. Coverage gate per task: `uv run --no-sync pytest -q -p no:cacheprovider -n auto tests/market tests/cli/test_market_cli.py --cov=src/market --cov=src/cli --cov-report=term-missing` and the `src/market/*` and `src/cli/market.py` lines must read ≥ 99% (CI runs `--cov=src --cov-fail-under=99`).
- Lint: `uv run --no-sync ruff format <paths> && uv run --no-sync ruff check <paths>`.
## Review Focus
1. **Wrong event day.** A rule filed for public inspection after the 4 p.m. close is counted on the same trading day, or the FR publication date stands in for the release. The CY2023 final rule was filed 2022-11-02 16:15 ET and published 2022-11-18, 16 days later. Expected: day 0 is the first trading day whose close follows the release. Pinned in Task 10 (`test_after_close_filing_moves_day0`, `test_day0_after_close_friday_is_monday`, `test_cy2023_final_release_not_publication`, `test_publication_fallback_caveat`) and Task 11 (`test_offsets_count_trading_days`).
2. **Price series that are not the issuer's.** A later dividend rescales every past Tiingo `adjClose`, so naive appends jump at the pull boundary. A recycled ticker (`AGL` was AGL Resources before agilon's 2021-04-15 listing) attaches another company's history. Expected: returns are computed within a pull, and rows outside the security's validity window are dropped. Pinned in Task 3 (`test_incremental_pull_rebased_adjclose_keeps_returns`, `test_prices_clipped_to_security_window`).
3. **Survivorship.** Acquired or delisted firms disappear because EDGAR drops their tickers (`OSH` has `tickers: []`), or a 2015 event's portfolio is formed from today's members. Expected: delisted firms are kept, members are taken as of day 0, and missing price series are counted and shown as a gap. Pinned in Task 1 (`test_delisted_seed_entity_kept_without_current_ticker`) and Task 12 (`test_members_as_of_day0_include_later_delisted`, `test_no_price_series_counted_as_gap`).
4. **Money counted twice.**
- LDA: an amendment or termination is summed beside the report it replaces, or an LD-2 amount is repeated per issue code.
- FEC: memo rows (`memo_code` `X`) are summed with their parents.
- Expected: one amount per filing period, the latest version, never split or repeated by issue.
- Pinned in Task 4 (`test_amendment_supersedes_original`, `test_issue_codes_never_split_amount`) and Task 5 (`test_memo_rows_excluded_from_totals`).
5. **Secrets and licensed data leaking.** An api.data.gov or Tiingo key ends up in a cache path, a bib URL, `cms.ingest_log` or an `httpx` error message, or prices are attached to bib, where they would sync to Zotero. Pinned in Task 1 (`test_key_sent_in_header_never_in_url`, `test_refused_message_has_no_query_string`), Task 3 (`test_tiingo_cite_has_no_attachment`) and Task 5 (`test_fec_error_message_has_no_key`).
## Decisions
Each item below is a planner's call. Nobody was available to ask; each is made here and the tasks implement it.
**Layout**
- One `market` package, one `stack market` CLI group (`src/cli/market.py`), and DuckDB schemas `market` and `econ`, with all DDL in `src/market/tables.py` (planner's call).
- The Congress and district layer lives in the same package (planner's call).
- No new dependencies and no new extra. numpy, httpx and PyYAML are already locked (planner's call).
**Keys and API access**
- Keys go only in headers. api.data.gov keys use `X-Api-Key` (planner's call).
- `API_DATA_GOV_KEY` is shared, with `FEC_API_KEY` and `CONGRESS_API_KEY` as overrides, per the coordinator's instruction.
- `LDA_API_KEY` is optional because anonymous lda.gov works; `HUD_API_TOKEN` is gap-gated (planner's call).
- Pacers, set below each provider's limit (planner's call): SEC 0.15 s, Tiingo 75 s, LDA 4.2 s (0.55 s keyed), FEC 3.7 s, Congress.gov 0.8 s, GitHub 61 s, FR 0.5 s.
- The Tiingo budget is capped at 450 symbols a month and 950 requests a day (planner's call).
**Benchmarks and money data**
- The sector index is **XLV**, which starts 1998-12-22. IHF (from 2006-05-05) is a providers sensitivity index. SPY (from 1993-01-29) is the market index (planner's call).
- Returns come from the `adjClose` ratio within a pull, with the raw total return kept as a cross-check. #797's `market.series` is a monthly total-return index (planner's call).
- LDA and FEC start in 2008, post-HLOGA (planner's call).
- FEC is the openFEC API only, per the user's decision. The keyless FEC bulk files are noted, not used.
**Entity map**
- Organizations are matched by exact normalized name for LDA and FEC, and by prefix for subsidiaries in the CMS footprint, which needs an alias of at least 5 characters (planner's call).
- Seed entries must be cited or loading fails. HarmonyCares, WNOT and Wakely are refused (planner's call).
- An fts link needs a health SIC (the 19 `HEALTH_SICS`) or seed membership.
- Passages are at most 400 characters and at most 3 per term, with an AMA 8-word guard.
- The optional local classifier uses the labels revenue_exposure, risk_factor and incidental (planner's call).
- SIC alone only links to the `sector` scope; family and specialty links need a passage or a curated cite (planner's call).
**Event study**
- The event date is the FR public-inspection `filed_at`. At or after 16:00 ET, day 0 is the next trading day; otherwise it is the publication date, with a caveat (planner's call).
- RUC and Panel meetings come from FR paragraph mentions (month precision, cited) and, where readable, AMA schedule pages (day precision). Month-precision events never enter daily windows (planner's call).
- The estimation window is (−250, −31) with a minimum of 120 returns. Event windows are (0,0), (−1,1), (0,1), (−5,5) and (0,10). The default model is `market` (SPY), because a sector index absorbs sector-wide rule effects; `sector` and `two-factor` are sensitivities (planner's call).
- Portfolios are equal-weighted and rebalanced daily, with membership as of day 0, and the model is run on the portfolio series to handle cross-correlation. Pooled means use the shared `pfs.event_study` bootstrap (planner's call).
- The tenure comparison uses annual BHAR against SPY, comparing seat years with non-seat years, as association only. The market model stays in `market/abnormal.py`; `pfs/event_study.py` only gains the shared bootstrap helpers (planner's call).
**Congress and districts**
- Committee assignments come from the `unitedstates/congress-legislators` snapshots in its git history, one per year from 2013. Members come from Congress.gov when keyed and from that repository's YAML when not, which also gives the FEC crosswalk (planner's call).
- Bills cover the 110th–119th Congresses. They come from the Congress.gov bill list with a local `BILL_RE` title filter plus `policyArea == "Health"` (planner's call).
- Districts are CD118 and CD119 from the 2020 Census relationship files, weighted by land area. HUD business and residential ratios are preferred when keyed (planner's call).
- The footprint is a cross-section: data year 2024, the CMS All Owners snapshot of 2026-09-29, the Tuva 2025-09 NPPES cut, and CD119. ZIPs with no district are kept as `??` (planner's call).
**Enrichment**
- FRED comes from the keyless fredgraph CSV, using 8 series. SOI comes from the ZIP-level files only (district AGI); the county file is skipped (planner's call).
- Company revenue comes from XBRL companyfacts (`Revenues`). Segment revenue is a stated gap (planner's call).
- The enrichment readers are annual, plus monthly for CES, FRED and market series. The specialty→SOC and family→CES maps are as listed in Task 14 (planner's call).
**People**
- No FEC individual donors and no lobbyist names are stored (planner's call).
---
## File map
| File | Responsibility |
|---|---|
| `src/market/http.py` | UA, keys, pacer, `get` with backoff/refusal, hoard paths |
| `src/market/tables.py` | DDL (`market`, `econ`), dataclasses, writers, readers, source gaps |
| `src/market/seed.yaml`, `src/market/seed.py` | Curated companies and organizations, each cited; name normalization and alias matching |
| `src/market/edgar.py` | company_tickers, submissions, SIC sweep, filings index; full-text search and passages (Task 2) |
| `src/market/tiingo.py` | EOD prices, metadata, symbol budget, monthly total-return series (#797) |
| `src/market/lda.py`, `src/market/fec.py` | Lobbying filings; PAC committees, totals, contributions |
| `src/market/congress.py` | Members, committee assignments, bills |
| `src/market/districts.py`, `src/market/footprint.py` | ZIP/county → district weights; company and family-biller footprints; district view |
| `src/market/events.py` | Rule, RUC and Panel event windows; seat tenures |
| `src/market/abnormal.py`, `src/market/portfolio.py` | Market model and CAR; portfolios, pooled results, tenure contrast |
| `src/market/enrich.py` | Joint readers: market beside BLS, FRED, SOI, PSPS, utilization |
| `src/econ/fred.py`, `src/econ/irs_soi.py` (mod) | fredgraph CSV and SOI ZIP-file parsers |
| `src/pfs/event_study.py` (mod) | Shared `bootstrap_draws` / `percentile_interval` |
| `src/cli/market.py`, `src/cli/__init__.py` (mod) | `stack market …` |
| `notebooks/event_study.py` (mod) | #797 market panel |
**Order (planner's call):**
- Task 1 comes first.
- Acquisition, Tasks 2–9 and 13, can then run in any order, except that Task 9 needs 6 and 8.
- Task 11, the engine, depends only on Task 1 and may run in parallel with acquisition.
- Task 10 needs 1. Task 12 needs 3, 10 and 11, and uses Task 2's links when present. Task 14 needs 3, 9, 12 and 13. Task 15 comes last.
- GPU is used only in Task 2 with `--classify`.
---
### Task 1: Market foundation, curated seed, EDGAR entity registry — `stack market entities`
**Files:**
- Create: `src/market/__init__.py`, `src/market/http.py`, `src/market/tables.py`, `src/market/seed.yaml`, `src/market/seed.py`, `src/market/edgar.py`, `src/cli/market.py`, `tests/market/__init__.py`, `tests/market/fixtures/` (`company_tickers.json` with 3 entries, `submissions_CIK0001564406.json` trimmed to 3 filings, `sic_8082.atom` with 2 entries)
- Modify: `src/cli/__init__.py` (`app.add_typer(market_app, name="market", help="Market and enrichment tier: exposed companies, prices, lobbying, Congress.")`), `tests/cli/test_cli_import_without_optional_deps.py`
- Test: `tests/market/test_http.py`, `tests/market/test_seed.py`, `tests/market/test_edgar.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`
**Interfaces:**
- Consumes: `pfs.seats.save_snapshot`, `pfs.seats.html_text`, `pfs.psps.newest_cache`, `cms.ingest_log.log_ingest`, `bib.item.Source`/`Download`, `bib.tag.Tag`, `pfs.commenters.specialty_vocab`.
- Produces:
- `market.http`:
- `USER_AGENT = "stack-pfs-dossier/1.0 (ci@mail.fhirworx.io)"`.
- `KEY_ENV = {"tiingo": ("TIINGO_API_KEY",), "fec": ("FEC_API_KEY", "API_DATA_GOV_KEY"), "congress": ("CONGRESS_API_KEY", "API_DATA_GOV_KEY"), "lda": ("LDA_API_KEY",), "hud": ("HUD_API_TOKEN",)}`.
- `api_key(source: str, env: Mapping[str, str] | None = None) -> str | None`: the first non-empty variable.
- `key_headers(source: str, key: str | None) -> dict[str, str]`: tiingo/lda `Authorization: Token k`; fec/congress `X-Api-Key: k`; hud `Authorization: Bearer k`; `{}` for None.
- `client(source: str, key: str | None = None) -> httpx.Client`: UA, key headers, timeout 120, follow_redirects.
- `class Pacer(min_interval: float, *, clock=time.monotonic, sleep=time.sleep)` with `.wait() -> None`.
- `class Refused(Exception)` with `source: str`, `status: int`, `reason: str`; its `str()` never contains a query string.
- `get(cl, url, *, params=None, pacer: Pacer, source: str, sleep=time.sleep, retries=3, max_retry_after=600.0, ok_404=False) -> httpx.Response`:
- 5xx retried with a `2·2**n` s backoff;
- 429 with `Retry-After ≤ max_retry_after` waits once and retries, otherwise `Refused`;
- 401/403 → `Refused`;
- 404 is returned when `ok_404`, else `Refused`.
- `hoard_root() -> Path` (`conf.path("db.aco").parent / "market"`); `hoard(*parts: str) -> Path`.
- `save = pfs.seats.save_snapshot`; `newest = pfs.psps.newest_cache`.
- `slug(text: str) -> str`: lower-case alnum and `-`, at most 60 chars, plus `-<sha8>` when truncated.
- `market.seed`:
- `@dataclass(frozen=True) Cite(url: str = "", title: str = "", accessed: str = "", adsh: str = "", filename: str = "", quote: str = "", item_key: str = "")`. Exactly one of `url` / `adsh+filename` / `item_key` is set; `quote` ≤ 400 chars.
- `SeedSecurity(ticker: str, exchange: str, valid_from: date | None, valid_to: date | None)`.
- `SeedLink(scope: str, target: str, cite: Cite)` with scope ∈ `("family", "specialty", "program")`.
- `SeedCompany(key, name, cik, status, ended_on: date | None, successor: str, aliases: tuple[str, ...], securities: tuple[SeedSecurity, ...], links: tuple[SeedLink, ...], cite: Cite)` with status ∈ `("listed", "acquired", "delisted", "bankrupt")`.
- `SeedOrg(key, name, org_type, specialty, ruc_seat, aliases, fec: tuple[str, ...], cite: Cite)` with org_type ∈ `("specialty_society", "trade_group", "professional_assoc")`.
- `load_seed(path: Path | None = None) -> tuple[tuple[SeedCompany, ...], tuple[SeedOrg, ...]]` raises `ValueError(f"{key}: no citation")` for any uncited entry or link.
- `EXCLUDED = ("HARMONYCARES", "WAKELY", "WNOT")`: a seed name or alias containing one raises.
- `norm_org(name: str) -> str`: upper-case, `&`→`AND`, punctuation dropped, spaces collapsed, leading `THE ` and trailing `INC|INCORPORATED|CORP|CORPORATION|CO|LLC|L L C|LP|LTD|PLC` dropped.
- `alias_match(name: str, aliases: Sequence[str], *, prefix: bool) -> bool`: `norm_org` equality; with `prefix`, also `norm(name).startswith(alias + " ")` for aliases ≥ 5 chars.
- Tables:
- `EntityRow(key, name, kind, cik, sic, sic_desc, status, ended_on: date | None, successor, specialty, org_type, source_url, item_key)` → `market.entity`;
- `SecurityRow(entity_key, ticker, exchange, valid_from: date | None, valid_to: date | None, source)` → `market.security`;
- `LinkRow(entity_key, scope, target, basis, strength, cite_kind, cite_ref, passage_id, item_key)` → `market.entity_link`, with:
- scope ∈ `("family", "specialty", "program", "sector")`;
- basis ∈ `("curated", "sic", "fts")`;
- strength ∈ `("curated", "sector", "revenue_exposure", "risk_factor", "mention")`;
- `FilingRow(cik, adsh, form, filed: date, accepted: str, report_date: str, items: str, primary_doc: str)` → `market.filing`;
- `GapRow(source, note, recorded_on: date)` → `market.source_gap`.
- Functions:
- `ensure_tables(con)`;
- `write_entities(con, rows)`, which deletes by key;
- `write_securities(con, entity_key, rows)`;
- `write_links(con, basis, rows)`, which replaces that basis;
- `write_filings(con, cik, rows)`;
- `write_gap(con, source, note)` and `clear_gap(con, source)`;
- readers `read_entities(con, kind=None)`, `read_securities(con, entity_key=None)`, `read_links(con, *, scope=None, target=None, basis=None)`, `read_filings(con, cik, forms=None)`, `read_gaps(con)`.
- `market.edgar`:
- URLs: `TICKERS = "https://www.sec.gov/files/company_tickers.json"`, `SUBMISSIONS = "https://data.sec.gov/submissions/CIK{cik10}.json"`, `SIC_ATOM = "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&SIC={sic}&type=10-K&owner=include&count=100&start={start}&output=atom"`.
- Pacer 0.15 s (SEC limit 10 req/s).
- `HEALTH_SICS = ("2834", "2835", "2836", "3841", "3842", "3845", "5047", "5122", "6324", "8000", "8011", "8050", "8051", "8060", "8062", "8071", "8082", "8090", "8093")`.
- Functions:
- `cik10(cik: str | int) -> str`;
- `parse_tickers(payload) -> dict[str, list[str]]`, mapping cik10 to tickers;
- `parse_sic_atom(xml: bytes) -> list[str]` (cik10s);
- `entity_from_submissions(sub: dict, seed: SeedCompany | None) -> EntityRow` (the seed fixes status, ended_on and successor);
- `securities_for(sub, seed, tickers) -> list[SecurityRow]` (seed windows win; current tickers from submissions get `valid_from=None`);
- `filings_from_submissions(sub: dict) -> list[FilingRow]` (recent plus each `filings.files` page);
- `sic_links(entity: EntityRow) -> list[LinkRow]` (scope `sector`, target the SIC, basis `sic`, strength `sector`, only for `HEALTH_SICS`);
- `curated_links(seed) -> list[LinkRow]` (basis/strength `curated`);
- `cite_registry(store) -> str` (bib Source, tags `module:market`, `table:market.entity`, `source:sec-edgar`).
- CLI:
- `stack market entities [--sic-sweep] [--refresh] [--offline] [--write]`. It loads the seed and the tickers, then submissions for the seed CIKs (plus every `HEALTH_SICS` CIK with `--sic-sweep`). Caches live at `data/market/sec/{company_tickers.json, submissions/CIK…json, sic/<sic>.<start>.atom}`. It writes entities, securities, filings, and the curated and sic links. Organizations from the seed become `EntityRow(kind="organization")`.
- `stack market gaps` prints `market.source_gap`.
- Module indirections `_batch`, `_read`, `_store`, `_publish`, `_asker` (as in `cli/pfs.py`) and `_gap(source, note, *, write)`, which echoes `gap: {source}: {note}` and, with write, calls `write_gap` in a batch.
- Seed content (planner's call; the implementer resolves each CIK with `https://efts.sec.gov/LATEST/search-index?keysTyped=<name>` and cites every link):
- Companies, each with ≥ 1 cited link:
- home-based and value-based primary care: OSH (CIK 1564406, 2020-08-06..2023-05-02, acquired), SGFY (1828182, 2021-02-11..2023-04-03, acquired), AMED (896262, 1994-08-18..2025-08-14, acquired), LHCG (2005-06-09..2023-02-22), ONEM (2020-01-31..2023-02-22), AGL (1831097, from 2021-04-15), PRVA (1759655, from 2021-04-28), CANO, ALHC (1832466), CLOV, EVH (1628908), ADUS (1468328), EHAB (2022-07-01..2026-05-14), CHE (19584);
- physician and outpatient services: MD (893949), RDNT (790526), USPH (885978), LFST;
- dialysis: DVA (927066);
- payers: UNH (731766), HUM, CVS, CI, ELV;
- labs: LH, DGX;
- virtual care: TDOC, AMWL;
- hospitals: HCA (860730, from 2011-03-10), THC, UHS, CYH;
- post-acute: ENSG (1125376).
- Organizations:
- one specialty society per 2024 `pfs.governance_seat` RUC label, with `ruc_seat` set to that label (e.g. `Family Medicine` → American Academy of Family Physicians, FEC `C00411553`);
- trade groups AMA, AOA, AHA, AHIP, MGMA, AMGA, NAACOS, America's Physician Groups, the home-care association, AAHCM and ATA;
- cite: a public page naming the organization's role.
- [ ] **Step 1: Write the failing tests**
- `test_api_key_falls_back_to_api_data_gov`:
- env `{"API_DATA_GOV_KEY": "k1"}` → `api_key("fec") == api_key("congress") == "k1"`;
- with `FEC_API_KEY="k2"` → `"k2"`;
- `api_key("tiingo", {}) is None`.
- `test_key_sent_in_header_never_in_url`: MockTransport records the request; `headers["X-Api-Key"] == "k1"` and `"k1" not in str(request.url)`.
- `test_user_agent_has_contact`: `USER_AGENT == "stack-pfs-dossier/1.0 (ci@mail.fhirworx.io)"` and `client("fec","k").headers["user-agent"] == USER_AGENT`.
- `test_pacer_spaces_requests`: fake clock at 0.0 then 1.0, `Pacer(4.2)` → the second `wait()` sleeps `pytest.approx(3.2)`.
- `test_get_retries_5xx_then_ok`: 503, 200 → sleeps `[2.0]`.
- `test_get_429_retry_after_waits_once`: 429 `Retry-After: 30` then 200 → `sleep(30.0)`.
- `test_get_429_twice_raises_refused`.
- `test_refused_message_has_no_query_string`: 403 on `…/x?api_key=secret` → `Refused.status == 403` and `"secret" not in str(exc)` and `"?" not in str(exc)`.
- `test_get_404_ok_when_allowed`.
- `test_slug`: `slug("Medicare Physician Fee Schedule") == "medicare-physician-fee-schedule"`.
- `test_seed_loads_and_every_entry_cited`.
- `test_seed_entry_without_cite_rejected`: a tmp YAML without a cite → `ValueError` naming the key.
- `test_seed_has_no_excluded_names`.
- `test_security_windows_parse`: OSH `valid_from == date(2020,8,6)` and `valid_to == date(2023,5,2)`.
- `test_norm_org`:
- `"Amedisys Holding, L.L.C."` → `"AMEDISYS HOLDING"`;
- `"The American College of Physicians, Inc."` → `"AMERICAN COLLEGE OF PHYSICIANS"`.
- `test_alias_match_exact_and_prefix`:
- `"AMERICAN ACADEMY OF FAMILY PHYSICIANS FOUNDATION"` does not match alias `AMERICAN ACADEMY OF FAMILY PHYSICIANS` with `prefix=False`;
- `"AMEDISYS HOLDING LLC"` matches `AMEDISYS` with `prefix=True`, but `"AMEDISYSTEMS"` does not;
- `"HCA SOMETHING"` does not match the 3-char alias `HCA` with `prefix=True`.
- `test_parse_tickers`.
- `test_parse_sic_atom`: fixture → `["0001725255", "0001468328"]`.
- `test_entity_from_osh_submissions`: sic `"8000"`, status `"acquired"`, ended_on `date(2023,5,2)` from the seed.
- `test_delisted_seed_entity_kept_without_current_ticker`: AMED is absent from the tickers fixture → one `EntityRow` and `SecurityRow("AMEDISYS","AMED","NASDAQ",date(1994,8,18),date(2025,8,14),"seed")`.
- `test_filings_from_submissions_include_8k_items`: `0001193125-23-132570` has items `"1.01,1.02,2.01,3.01,3.03,5.01,5.02,5.03,8.01,9.01"` and accepted `"2023-05-02T20:42:01.000Z"`.
- `test_sic_link_only_for_health_sics`: 8082 → one sector link; 7372 → none.
- `test_ensure_tables_idempotent`.
- `test_roundtrip[entity|security|link|filing|gap]`.
- `test_entities_offline_writes_and_logs` (CLI): fixture caches → entity, security and link rows, one `cms.ingest_log` row per source file, `_publish` called once.
- `test_entities_refusal_prints_gap` (CLI): 403 → output has `gap: sec-edgar:` and nothing is written.
- `test_gaps_command_lists_rows`.
- In `test_cli_import_without_optional_deps.py`:
- add `cli.market` to `_PROBE`;
- new `test_cli_market_imports_without_numpy` (the probe also blocks `numpy`, `duckdb` and `yaml` and imports `cli.market`).
- [ ] **Step 2: Run** `uv run --no-sync pytest -q -p no:cacheprovider -n auto tests/market tests/cli/test_market_cli.py tests/cli/test_cli_import_without_optional_deps.py` → FAIL (`ModuleNotFoundError: market`).
- [ ] **Step 3: Implement** the files above. In `cli/market.py`, each command fetches everything first, then opens one `_batch()` for the writes and `log_ingest(module="market.edgar", table_name=…, rows=n, source_file=<cache path>, pincite_key=<bib key>)`, then calls `_publish()`.
- [ ] **Step 4: Run** the Step 2 command → PASS; run the coverage gate and ruff.
- [ ] **Step 5: Live** (network, no GPU):
- Run `ENV stack market entities --write`, then `ENV stack market entities --sic-sweep --write` (≈ 19 SIC atoms plus their submissions at 0.15 s).
- Report the entity counts by kind and status. Confirm that OSH, SGFY and AMED rows exist with tickers from the seed, and give the count of `sic` links per SIC.
- [ ] **Step 6: Commit**:
```bash
git add src/market/__init__.py src/market/http.py src/market/tables.py src/market/seed.yaml src/market/seed.py src/market/edgar.py src/cli/market.py src/cli/__init__.py tests/market tests/cli/test_market_cli.py tests/cli/test_cli_import_without_optional_deps.py
git commit -m "feat(market): EDGAR entity registry, cited seed of exposed companies and organizations, market CLI group"
```
---
### Task 2: EDGAR full-text search, exposure passages, optional local classification — `stack market fts`
**Files:**
- Modify: `src/market/edgar.py`, `src/market/tables.py`, `src/cli/market.py`
- Create: `tests/market/fixtures/efts_mpfs_2023.json` (the RadNet hit), `tests/market/fixtures/filing_snippet.htm`
- Test: `tests/market/test_fts.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`
**Interfaces:**
- Consumes: Task 1, `pfs.families.FAMILIES`/`refresh_from`, `pfs.origin.shares_run`, `pfs.families.CODE_RE`, `pfs.codetables.read_cpt_codes`, `pfs.runlog.RunLog`/`run_units`/`input_hash`/`ledger_path`, `cli.pfs._asker`.
- Produces:
- Search setup:
- `EFTS = "https://efts.sec.gov/LATEST/search-index"`; a page is 100 hits; `from` is the offset.
- `PROGRAM_TERMS = ("Medicare Physician Fee Schedule", "physician fee schedule", "RBRVS", "Relative Value Scale Update Committee", "CPT Editorial Panel")`.
- `family_terms(fam: Family) -> tuple[str, ...]`: the family name plus every multi-word synonym.
- `fts_params(term: str, form: str, year: int, offset: int) -> dict` gives `{"q": f'"{term}"', "forms": form, "dateRange": "custom", "startdt": f"{year}-01-01", "enddt": f"{year}-12-31", "from": offset}`. One query per (term, form, year) keeps each under the 10,000-hit cap.
- `HitRow(term, scope, target, cik, adsh, filename, form, file_date: date, period_ending: str, sics: str, display_name: str)` → `market.filing_hit`. `parse_hits(payload, *, term, scope, target) -> list[HitRow]`: `_id` is `"{adsh}:{filename}"`; cik is the first of `ciks`; `sics` are comma-joined.
- `doc_url(cik10: str, adsh: str, filename: str) -> str` returns `https://www.sec.gov/Archives/edgar/data/{int(cik)}/{adsh without dashes}/{filename}`. Docs are cached at `data/market/sec/docs/<cik10>/<adsh>/<filename>`.
- `PassageRow(passage_id, cik, adsh, filename, term, char_start, passage, label, label_version, item_key)` → `market.filing_passage`, where `passage_id = f"{adsh}:{filename}:{char_start}"`.
- `passages(text: str, term: str, *, max_chars=400, max_n=3) -> list[tuple[int, str]]`: a case-insensitive match, widened to sentence bounds within `max_chars`.
- `ama_safe(passage: str, descriptors: Sequence[str]) -> bool`.
- Classification:
- `EXPOSURE_LABELS = ("revenue_exposure", "risk_factor", "incidental")` and `EXPOSURE_PROMPT_VERSION = "exposure-v1"`.
- Schema `{"label": enum, "reason": str ≤ 200}`.
- `classify_passage(passage: str, term: str, ask) -> str | None`. The ledger is `ledger_path("market-exposure", "all")`.
- `fts_links(hits, passages, entities: Mapping[str, EntityRow], seed_ciks: set[str]) -> list[LinkRow]`, with basis `fts`:
- a link needs the hit's SIC in `HEALTH_SICS` or the CIK in the seed;
- strength is the passage label, or `mention` when the passage is unlabelled;
- `incidental` makes no link;
- a hit without a kept passage links with `passage_id=""` and `cite_ref=adsh:filename`.
- `cite_filing(store, hit, path) -> str`: a bib Download titled `"{display_name} Form {form} filed {file_date} (accession {adsh})"`, website_title `"U.S. Securities and Exchange Commission"`, tags `module:market`, `table:market.filing_passage`, `source:sec-edgar`, `year:<y>`, with the document attached (public record).
- CLI `stack market fts [--family KEY …] [--program] [--forms 10-K] [--from 2001] [--to <this year>] [--limit N] [--dry-run] [--classify] [--write]`:
- FTS pages are cached at `data/market/sec/fts/<slug(term)>/<form>/<year>.<offset>.json`;
- a saved page or document is never refetched (resumable);
- `--classify` runs the local model through the ledger;
- with `--write`: hits, passages, and the fts links are recomputed from all passages, with bib items for the cited filings.
- Hits for CIKs not yet in `market.entity` get a submissions fetch and an `EntityRow` in the same run, so delisted filers enter the registry with no ticker.
- [ ] **Step 1: Failing tests:**
- `test_fts_params_one_year`: term `"Medicare Physician Fee Schedule"`, 2023 → `q == '"Medicare Physician Fee Schedule"'`, `startdt == "2023-01-01"`, `enddt == "2023-12-31"`, `from == 0`.
- `test_parse_hits`: the RadNet fixture → `HitRow(cik="0000790526", adsh="0000790526-23-000003", filename="rdnt-20221231.htm", form="10-K", file_date=date(2023,3,1), sics="8071", …)`.
- `test_fts_pages_until_total`: total 150 → offsets `[0, 100]`.
- `test_doc_url` → `"https://www.sec.gov/Archives/edgar/data/790526/000079052623000003/rdnt-20221231.htm"`.
- `test_family_terms`: HOME-VISITS includes `"house calls"` and `"home or residence"`, and excludes `"domiciliary"`.
- `test_passages_window_and_cap`:
- term at char 1000 → `char_start` ≤ 1000, length ≤ 400, the term is included;
- 5 occurrences → 3 passages.
- `test_passage_sharing_cpt_descriptor_withheld`: the hit is kept, the passage is dropped, the link has `passage_id == ""`.
- `test_fts_links_require_health_sic_or_seed`: SIC 8071 links; SIC 6798 outside the seed gives no link but the hit is stored.
- `test_link_strength_from_label`: None → `mention`; `incidental` → no link; `revenue_exposure` → `revenue_exposure`.
- `test_classify_passage_closed_labels`: the fake ask returns `{"label":"revenue_exposure"}` → that label; `{"label":"bogus"}` → None.
- `test_roundtrip[filing_hit|filing_passage]`.
- `test_fts_cli_dry_run_lists_queries`: prints one line per (term, form, year) and makes no request.
- `test_fts_cli_classify_resumes_from_ledger`: a second run makes 0 new model calls.
- [ ] **Step 2: Run** `tests/market/test_fts.py tests/cli/test_market_cli.py` → FAIL.
- [ ] **Step 3: Implement.**
- [ ] **Step 4: Run** → PASS; coverage gate; ruff.
- [ ] **Step 5: Live:**
- Run `ENV stack market fts --program --from 2001 --dry-run | tail -3`, then `--write` as a long run (`.state/market/fts-program.log`).
- Run `ENV stack market fts --family HOME-VISITS --family CCM --family TCM --family APCM --write`.
- Optional GPU run: check `ENV stack llm hosts`, then `ENV stack market fts --program --classify --limit 200 --write`.
- Report hits per term, the passages withheld by the AMA guard, and new fts links per family.
- [ ] **Step 6: Commit** `git add src/market/edgar.py src/market/tables.py src/cli/market.py tests/market/test_fts.py tests/market/test_tables.py tests/market/fixtures/efts_mpfs_2023.json tests/market/fixtures/filing_snippet.htm tests/cli/test_market_cli.py && git commit -m "feat(market): EDGAR full-text exposure passages per program term and family, cited and AMA-guarded"`
---
### Task 3: Daily prices from Tiingo and the monthly series (#797) — `stack market prices`
**Files:**
- Create: `src/market/tiingo.py`, `tests/market/fixtures/supported_tickers_head.csv` (5 lines)
- Modify: `src/market/tables.py`, `src/cli/market.py`
- Test: `tests/market/test_tiingo.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`
**Interfaces:**
- Consumes: Task 1 (`client("tiingo", key)`, `Pacer`, `get`, `hoard`, `save`, `read_securities`, `read_links`).
- Produces:
- Endpoints:
- `EOD = "https://api.tiingo.com/tiingo/daily/{ticker}/prices"` with params `startDate`, `endDate`, `format=json`, `resampleFreq=daily`;
- `META = "https://api.tiingo.com/tiingo/daily/{ticker}"`;
- `SUPPORTED = "https://apimedia.tiingo.com/docs/tiingo/daily/supported_tickers.zip"` (keyless, no budget).
- `BENCHMARKS = {"SPY": ("market", date(1993,1,29)), "XLV": ("sector", date(1998,12,22)), "IHF": ("providers", date(2006,5,5))}`. Start dates are from Tiingo metadata read 2026-11-12. XLV is the sector index (planner's call); IHF is a providers sensitivity index.
- `@dataclass Budget(month: str, symbols: set[str], day_counts: dict[str, int])`:
- stored in `.state/market/tiingo-<YYYY-MM>.json`;
- caps `MONTHLY_SYMBOLS = 450`, `DAILY_REQUESTS = 950`, pacer 75.0 s (48 per hour). The Starter limits are 500 symbols a month, 1,000 requests a day and 50 an hour.
- `Budget.allow(ticker: str, today: date) -> bool`; `Budget.spend(ticker, today) -> None`.
- `PriceRow(ticker, day: date, open, high, low, close, volume, adj_close, div_cash, split_factor, ret: float | None, ret_raw: float | None, pulled_on: date, source_file: str)` → `market.price_daily`.
- `to_prices(ticker, records, *, pulled_on, source_file, overlap: date | None) -> list[PriceRow]`:
- `ret = adjClose_t / adjClose_{t-1} − 1` within this pull only;
- `ret_raw = (close_t + divCash_t) · splitFactor_t / close_{t-1} − 1`;
- the `overlap` day (the last stored day, re-requested as the pull's first row) is dropped, so stored rows keep their returns.
- `clip(rows, valid_from, valid_to) -> list[PriceRow]`.
- `window_gap(meta: dict, sec: SecurityRow, tol_days=7) -> str | None`: Tiingo's `startDate`/`endDate` against the security window.
- `MonthlyRow(series_id, symbol, source, year, month, value, item_key)` → `market.series`, the #797 shape. `tri_monthly(ticker, rows, *, item_key) -> list[MonthlyRow]`:
- `series_id = f"tiingo:{ticker}:tri"`;
- the value is a total-return index chained from `ret`, base 100.0 on the first stored day, taken at each month's last trading day.
- `cite(store, ticker) -> str`: a bib Source with url `EOD` formatted, **no attachment**, tags `module:market`, `table:market.price_daily`, `source:tiingo`.
- Readers in tables:
- `read_returns(con, tickers, start: date, end: date) -> dict[str, dict[date, float]]`;
- `trading_days(con, ticker="SPY") -> list[date]`;
- `read_series(con, symbols) -> list[MonthlyRow]`.
- Writer `write_prices(con, ticker, rows)` deletes that ticker's rows in `[min(day), max(day)]`, then inserts.
- CLI `stack market prices [--ticker T …] [--linked] [--benchmarks] [--limit N] [--refresh] [--dry-run] [--write]`:
- `--linked` covers every security of an entity with any link.
- A first pull asks for full history from `max(valid_from, 1993-01-29)`. Later pulls start at the last stored day.
- Each pull is cached at `data/market/tiingo/eod/<T>/<start>_<end>.json`, metadata at `tiingo/meta/<T>.json`.
- No `TIINGO_API_KEY` → `gap: tiingo: TIINGO_API_KEY not in .env`.
- [ ] **Step 1: Failing tests** (synthetic prices only):
- `test_to_prices_returns_within_pull`: adj 100, 110, 99 → ret `[None, 0.1, -0.1]` (approx).
- `test_ret_raw_with_split_and_dividend`: close 100 → 50 with split 2.0 gives 0.0; close 100 → 99 with div 1.0 gives 0.0.
- `test_incremental_pull_rebased_adjclose_keeps_returns`: pull 1 is days d1, d2 at adj 100, 110. Pull 2 is d2, d3 at adj 55, 60.5 (rebased by ×0.5) with overlap d2. Result: the stored d2 row is pull 1's; d3 ret = 0.1.
- `test_prices_clipped_to_security_window`: AGL rows dated 2015-06-01 and 2021-05-03 with `valid_from=2021-04-15` → only 2021 is kept.
- `test_window_gap`: metadata `startDate` 2021-04-15 vs `valid_from` 2021-04-15 → None; `endDate` 2023-05-02 vs `valid_to` 2023-06-30 → a note naming both dates.
- `test_budget_monthly_symbols_and_daily_cap`: with 450 symbols used, a new symbol is refused and a known one allowed; at 950 requests today, everything is refused.
- `test_monthly_tri_month_end`: rets on 2024-01-02 None, 2024-01-03 0.1, 2024-02-01 −0.1 → rows `(2024,1)=110.0`, `(2024,2)=99.0`.
- `test_benchmarks_pinned`.
- `test_tiingo_cite_has_no_attachment`: the fake store records `upsert` and `attach_file` is never called; the url contains no `token`.
- `test_roundtrip[price_daily|series]`.
- `test_read_returns_and_trading_days`.
- `test_prices_without_key_records_gap` (CLI): exit 0, output starts with `gap: tiingo:`.
- `test_prices_dry_run_reports_budget` (CLI).
- [ ] **Step 2: Run** → FAIL.
- [ ] **Step 3: Implement.**
- [ ] **Step 4: Run** → PASS; coverage; ruff.
- [ ] **Step 5: Live:**
- Run `ENV stack market prices --benchmarks --write` (3 symbols). Report the row counts and first/last days per symbol: SPY must start 1993-01-29 and XLV 1998-12-22.
- Then run `ENV stack market prices --linked --dry-run`, followed by the long run `--linked --write` (`.state/market/prices.log`, about 48 symbols an hour).
- Report symbols fetched, the window gaps, and the days where `|ret − ret_raw| > 0.001`. Report counts only.
- [ ] **Step 6: Commit** `git add src/market/tiingo.py src/market/tables.py src/cli/market.py tests/market/test_tiingo.py tests/market/test_tables.py tests/market/fixtures/supported_tickers_head.csv tests/cli/test_market_cli.py && git commit -m "feat(market): Tiingo daily prices with total returns, benchmarks SPY/XLV/IHF, monthly series (#797)"`
---
### Task 4: Lobbying disclosures (LDA) — `stack market lda`
**Files:**
- Create: `src/market/lda.py`, `tests/market/fixtures/lda_aafp_2024_q1.json` (one filing, lobbyists removed)
- Modify: `src/market/tables.py`, `src/cli/market.py`
- Test: `tests/market/test_lda.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`
**Interfaces:**
- Consumes: Task 1 (`client("lda", key)`, `api_key("lda")`, `alias_match(prefix=False)`, organization and company entities with `aliases`).
- Produces:
- Endpoint `LDA_API = "https://lda.gov/api/v1/"`. `lda.senate.gov` was sunset on 2026-07-31 and returns a 503 maintenance page.
- `page_size = 25` (the API maximum). Pacer 4.2 s anonymous (the limit is 15/min) and 0.55 s with `LDA_API_KEY` (120/min).
- `FIRST_YEAR = 2008` (planner's call: LD-203 starts in 2008).
- `LDA_TERMS = ("physician fee schedule", "conversion factor", "relative value", "evaluation and management", "home-based")`, sent as `filing_specific_lobbying_issues="<term>"`.
- `HEALTH_ISSUES = ("MMM", "HCR", "MED", "INS", "PHA", "BUD", "TAX")`.
- Rows:
- `LdaFilingRow(filing_uuid, filing_type, filing_year, filing_period, dt_posted, registrant_id: int, registrant_name, client_id: int, client_name, income: float | None, expenses: float | None, amount: float | None, superseded: bool, document_url, entity_key, matched_by, source_file)` → `market.lda_filing`;
- `LdaIssueRow(filing_uuid, general_issue_code, description)` → `market.lda_issue`.
- `to_filing(rec, *, entity_key, matched_by, source_file) -> tuple[LdaFilingRow, list[LdaIssueRow]]`:
- `amount = income` when income is not None, else `expenses`;
- registrations (`RR`, `RA`) have `amount=None`;
- decimal strings become floats.
- `supersede(rows) -> set[str]`: within each (registrant_id, client_id, filing_year, filing_period), every report-type filing except the latest `dt_posted` is superseded.
- `query_pages(cl, params, *, pacer) -> Iterator[dict]` follows `next`.
- `spend_by_year(con, entity_key) -> dict[int, float]` sums non-superseded amounts once per filing. `filings_by_issue(con, entity_key, code) -> list[str]`.
- `write_lda(con, rows, issues)` deletes by `filing_uuid`; then `mark_superseded(con)` recomputes the flag in SQL.
- CLI `stack market lda [--org KEY …] [--terms] [--from 2008] [--to <this year>] [--limit N] [--write]`:
- tracked entities are queried by `client_name` and `registrant_name` per year, and the API's fuzzy name matches are post-filtered with `alias_match(prefix=False)`;
- `--terms` adds the discovery queries, with `entity_key=""` and `matched_by="term:<t>"`;
- the cache is `data/market/lda/<year>/<slug(query)>.<page>.json`;
- one bib Source per filing year (url `https://lda.gov/api/v1/filings/?filing_year=<y>`), with no attachments.
- [ ] **Step 1: Failing tests:**
- `test_lda_base_is_lda_gov`.
- `test_lda_pacer_anonymous_and_keyed`: no key → 4.2; key → 0.55 and the header `Authorization: Token k`.
- `test_to_filing_amounts`: fixture uuid `df5f17c4-62a7-4d7f-8ebe-aeb15147f396` → filing_type `"Q1"`, amount `825016.0`, income None, and the issues include `"BUD"`.
- `test_amendment_supersedes_original`: a Q1 posted 2024-04-15 and a `1A` posted 2024-05-01 for the same pair and period → Q1 superseded, 1A not.
- `test_registration_has_no_amount`.
- `test_issue_codes_never_split_amount`: a filing with 3 issues and amount 300.0 → `spend_by_year == {2024: 300.0}`, and `filings_by_issue` returns it for each code.
- `test_name_post_filter_drops_foundation`.
- `test_query_pages_follows_next`.
- `test_roundtrip[lda_filing|lda_issue]`.
- `test_lda_cli_terms_offline` (CLI).
- [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff.
- [ ] **Step 5: Live:** `ENV stack market lda --from 2024 --to 2024 --limit 50` (anonymous), then the long run `ENV stack market lda --terms --write` (`.state/market/lda.log`; about 14 requests a minute). Report filings per organization type, the superseded count, and the top 10 clients by 2024 spend.
- [ ] **Step 6: Commit** `git add src/market/lda.py src/market/tables.py src/cli/market.py tests/market/test_lda.py tests/market/test_tables.py tests/market/fixtures/lda_aafp_2024_q1.json tests/cli/test_market_cli.py && git commit -m "feat(market): LDA lobbying filings and issue codes for tracked organizations and PFS terms (lda.gov)"`
---
### Task 5: FEC committees, totals and contributions — `stack market fec`
**Files:**
- Create: `src/market/fec.py`, `tests/market/fixtures/fec_committees_aafp.json`, `fec_totals_C00411553_2024.json`, `fec_schedule_b_page1.json` (synthetic, 3 rows incl. one memo)
- Modify: `src/market/tables.py`, `src/cli/market.py`
- Test: `tests/market/test_fec.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`
**Interfaces:**
- Consumes: Task 1 (`api_key("fec")`, `client("fec", key)`, `alias_match`, seed `fec` ids).
- Produces:
- `FEC_API = "https://api.open.fec.gov/v1/"`. The key goes in the `X-Api-Key` header. Pacer 3.7 s (api.data.gov allows 1,000 an hour). `CYCLES = tuple(range(2008, 2028, 2))`.
- Rows:
- `FecCommitteeRow(committee_id, name, committee_type, designation, entity_key, first_file_date, last_file_date, source_file)` → `market.fec_committee`;
- `FecTotalsRow(committee_id, cycle, receipts, disbursements, individual_contributions, fed_candidate_committee_contributions, independent_expenditures, coverage_start: date | None, coverage_end: date | None, source_file)` → `market.fec_totals`; the field names were verified 2026-11-12;
- `FecContributionRow(sub_id, committee_id, cycle, day: date | None, amount, recipient_committee_id, recipient_name, candidate_id, candidate_name, candidate_office, candidate_state, candidate_district, memo_code, source_file)` → `market.fec_contribution`.
- `resolve(cl, org: SeedOrg | SeedCompany, *, pacer) -> tuple[list[FecCommitteeRow], list[dict]]`:
- uses `/committees/?q=<name>&committee_type=Q&committee_type=N&per_page=20`;
- auto-accepts a committee when a normalized alias occurs in its normalized name, and returns the rest as candidates;
- seed `fec` ids override.
- `totals(cl, cid, cycle, *, pacer) -> FecTotalsRow | None` uses `/committee/{cid}/totals/?cycle=<c>`.
- `contributions(cl, cid, cycle, *, pacer) -> list[FecContributionRow]` uses `/schedules/schedule_b/?committee_id=…&two_year_transaction_period=…&per_page=100&sort=-disbursement_date`. Keyset paging takes `last_index` and `last_disbursement_date` from `pagination.last_indexes`.
- `contributions_by_cycle(con, entity_key) -> dict[int, float]` excludes `memo_code == "X"`.
- CLI `stack market fec [--org KEY …] [--cycles 2020,2022] [--limit N] [--write]`:
- with no key: `gap: fec: API_DATA_GOV_KEY (or FEC_API_KEY) not in .env — key pending (ci@mail.fhirworx.io)`, plus a gap row with `--write`, exit 0;
- the cache is `data/market/fec/{committees/<slug>.json, totals/<cid>/<cycle>.json, schedule_b/<cid>/<cycle>/<n>.json}`;
- candidates the code did not accept are printed for curation into `seed.yaml`.
- [ ] **Step 1: Failing tests:**
- `test_fec_without_key_records_gap`.
- `test_committee_auto_accept_contains_alias`: `C00411553` "AMERICAN ACADEMY OF FAMILY PHYSICIANS POLITICAL ACTION COMMITTEE (FAMMEDPAC)" is accepted for AAFP; an unrelated name becomes a candidate.
- `test_seed_fec_ids_override_search`.
- `test_totals_row`: cycle 2024, receipts 513798.01, disbursements 696685.91, fed_candidate_committee_contributions 683300.0, coverage `date(2023,1,1)`..`date(2024,12,31)`.
- `test_schedule_b_keyset_paging`: the second request carries `last_index=123` and `last_disbursement_date=2024-03-01`.
- `test_memo_rows_excluded_from_totals`: rows 100 + 50 (memo X) + 25 → 125.0.
- `test_fec_error_message_has_no_key`.
- `test_roundtrip[fec_committee|fec_totals|fec_contribution]`.
- `test_fec_cli_offline_from_cache` (CLI).
- [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff.
- [ ] **Step 5: Live:**
- If the key is present: `ENV stack market fec --org AAFP --cycles 2024 --write`, then a long run without `--org`. Report the committees resolved and unresolved, and the 2024 receipts per org type.
- If not: `ENV stack market fec --write` must print the gap line and leave one `market.source_gap` row.
- [ ] **Step 6: Commit** `git add src/market/fec.py src/market/tables.py src/cli/market.py tests/market/test_fec.py tests/market/test_tables.py tests/market/fixtures/fec_*.json tests/cli/test_market_cli.py && git commit -m "feat(market): FEC PAC committees, cycle totals and committee contributions (key-gated, gap until present)"`
---
### Task 6: Members of Congress and committee assignments — `stack market members`, `stack market committees`
**Files:**
- Create: `src/market/congress.py`, `tests/market/fixtures/congress_members_118.json` (2 members), `legislators_snippet.yaml`, `membership_snippet.yaml`
- Modify: `src/market/tables.py`, `src/cli/market.py`
- Test: `tests/market/test_congress.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`
**Interfaces:**
- Consumes: Task 1.
- Produces:
- Endpoints:
- `CONGRESS_API = "https://api.congress.gov/v3/"`, with `X-Api-Key`, pacer 0.8 s (the limit is 5,000 an hour), `limit=250` and `offset`.
- `LEGISLATORS = "https://raw.githubusercontent.com/unitedstates/congress-legislators/{ref}/{name}"` (public domain, keyless) for `legislators-current.yaml`, `legislators-historical.yaml` and `committee-membership-current.yaml`.
- `GH_COMMITS = "https://api.github.com/repos/unitedstates/congress-legislators/commits"`, pacer 61 s (60 an hour unauthenticated).
- `congress_of(year: int) -> int` = `(year − 1789) // 2 + 1`. `CONGRESSES = range(110, 120)`.
- `HEALTH_COMMITTEES = ("HSWM", "HSWM02", "HSIF", "HSIF14", "SSFI", "SSFI10", "SSHR", "SSHR12")`, verified 2026-11-12 in `committees-current.yaml`: W&M Health, E&C Health, Finance Health Care, HELP Primary Health and Retirement Security.
- Rows:
- `MemberRow(congress, bioguide_id, name, party, state, district: int | None, chamber, fec_ids: str, source)` → `market.congress_member`; chamber is `house` or `senate`; source is `congress.gov` or `congress-legislators`;
- `AssignmentRow(congress, snapshot: date, committee_id, bioguide_id, rank: int | None, title, party, source_sha)` → `market.committee_assignment`.
- Parsers:
- `members_from_api(payload, congress) -> list[MemberRow]` takes the term that covers the congress;
- `members_from_yaml(docs, congresses) -> list[MemberRow]`, one row per (congress, bioguide) whose term overlaps it;
- `fec_ids_by_bioguide(docs) -> dict[str, tuple[str, ...]]`;
- `membership_rows(doc: dict, *, snapshot: date, sha: str) -> list[AssignmentRow]` keeps `HEALTH_COMMITTEES` only.
- Committee assignments come from congress-legislators snapshots (planner's call): the Congress.gov API has no committee membership. One snapshot per year, the last commit before March 1, for 2013 through this year; earlier years are a gap.
- CLI:
- `stack market members [--from 110] [--to 119] [--write]` uses the YAML always (keyless, FEC crosswalk) and the Congress.gov API when keyed; without a key it prints the gap and keeps the YAML rows. The reader prefers `congress.gov` rows.
- `stack market committees [--from 2013] [--to <this year>] [--write]`.
- Caches: `data/market/congress/members/<c>/<offset>.json` and `data/market/legislators/<name>.<sha8>.yaml`.
- [ ] **Step 1: Failing tests:**
- `test_congress_of_year`: 2013 → 113, 2026 → 119, 2027 → 120.
- `test_members_from_api_page`: chamber `house`, state `MI`, district 12.
- `test_members_from_legislators_yaml`.
- `test_fec_ids_by_bioguide`.
- `test_membership_snapshot_rows`: SSAF and HSWM02 in the fixture → only HSWM02 rows, with rank and title kept.
- `test_github_commit_lookup_params`: `until == "2015-03-01T00:00:00Z"`, `path == "committee-membership-current.yaml"`, `per_page == 1`.
- `test_members_without_key_uses_yaml_and_records_gap` (CLI).
- `test_roundtrip[congress_member|committee_assignment]`.
- [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff.
- [ ] **Step 5: Live:** `ENV stack market members --write`, then `ENV stack market committees --write` (a long run of 14 snapshots at 61 s). Report members per congress (≈ 540) and the HSWM02 and HSIF14 member counts for 2025.
- [ ] **Step 6: Commit** `git add src/market/congress.py src/market/tables.py src/cli/market.py tests/market/test_congress.py tests/market/test_tables.py tests/market/fixtures/congress_members_118.json tests/market/fixtures/legislators_snippet.yaml tests/market/fixtures/membership_snippet.yaml tests/cli/test_market_cli.py && git commit -m "feat(market): members of Congress by district and health-committee assignments over time"`
---
### Task 7: Bills and actions on Medicare physician payment — `stack market bills`
**Files:**
- Modify: `src/market/congress.py`, `src/market/tables.py`, `src/cli/market.py`
- Create: `tests/market/fixtures/congress_bills_page.json`, `congress_bill_detail.json`, `congress_bill_actions.json`, `congress_bill_cosponsors.json` (synthetic)
- Test: `tests/market/test_congress.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`
**Interfaces:**
- Consumes: Task 6 (`CONGRESS_API`, client, pacer).
- Produces:
- `BILL_RE = re.compile(r"(?i)\b(physician fee schedule|physician payment|conversion factor|sustainable growth rate|relative value|evaluation and management|telehealth|home-based|house call|care management|primary care)\b")`. A matched title must also have detail `policyArea.name == "Health"` to be kept.
- Rows:
- `BillRow(congress, bill_type, number: int, title, introduced: date | None, policy_area, latest_action_date: date | None, latest_action, matched_term, url, item_key)` → `market.bill`;
- `BillActionRow(congress, bill_type, number, action_date: date, text, action_code, committee_codes)` → `market.bill_action`;
- `BillMemberRow(congress, bill_type, number, bioguide_id, role, since: date | None)` → `market.bill_member`, with role `sponsor` or `cosponsor`.
- Helpers:
- `ordinal(n: int) -> str`;
- `bill_url(congress, bill_type, number)` gives `https://www.congress.gov/bill/{ordinal}-congress/{house-bill|senate-bill|house-resolution|senate-resolution|house-joint-resolution|senate-joint-resolution|house-concurrent-resolution|senate-concurrent-resolution}/{n}`;
- `match_title(title) -> str | None`.
- CLI `stack market bills [--from 110] [--to 119] [--limit N] [--write]`:
- it lists `/bill/{congress}?limit=250&offset=N`, filters locally, then fetches `/bill/{c}/{type}/{n}`, `/actions` and `/cosponsors` for each match;
- one bib Source per kept bill (url = `bill_url`);
- the key is required (gap otherwise).
- [ ] **Step 1: Failing tests:**
- `test_bill_title_filter`: "Medicare Physician Fee Schedule Update Act" → `"physician fee schedule"`; "Telehealth Modernization Act" → `"telehealth"`; "Farm Bill" → None.
- `test_bill_kept_only_health_policy_area`.
- `test_ordinal`: 111 → "111th", 112 → "112th", 121 → "121st", 122 → "122nd", 123 → "123rd".
- `test_bill_url`: (118, "HR", 1234) → `"https://www.congress.gov/bill/118th-congress/house-bill/1234"`.
- `test_actions_and_cosponsors_rows`.
- `test_bills_without_key_records_gap` (CLI).
- `test_roundtrip[bill|bill_action|bill_member]`.
- [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff.
- [ ] **Step 5: Live:** with the key, `ENV stack market bills --from 118 --to 119 --write`, then all congresses. Report matched and kept bills per congress and the share sponsored by HSWM02/HSIF14/SSFI10 members. Without the key, report the gap row.
- [ ] **Step 6: Commit** `git add src/market/congress.py src/market/tables.py src/cli/market.py tests/market/test_congress.py tests/market/test_tables.py tests/market/fixtures/congress_bill*.json tests/cli/test_market_cli.py && git commit -m "feat(market): Medicare physician-payment bills with actions, sponsors and cosponsors (Congress.gov)"`
---
### Task 8: District crosswalks — `stack market districts`
**Files:**
- Create: `src/market/districts.py`, `tests/market/fixtures/tab20_cd11920_zcta520_head.txt` (header plus 5 synthetic rows), `hud_zip_cd.json` (synthetic)
- Modify: `src/market/tables.py`, `src/cli/market.py`
- Test: `tests/market/test_districts.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`
**Interfaces:**
- Consumes: Task 1.
- Produces:
- Census source: `CENSUS_REL = "https://www2.census.gov/geo/docs/maps-data/data/rel2020/cd-sld/tab20_cd{c}20_{geo}_natl.txt"` for `c ∈ (118, 119)` and `geo ∈ ("zcta520", "county20")`. Files are pipe-delimited with a BOM; checked 2026-11-12.
- HUD source: `HUD_API = "https://www.huduser.gov/hudapi/public/usps"` with params `type=5` (zip-cd), `query=All`, `year` and `quarter`, and header `Authorization: Bearer <HUD_API_TOKEN>`. The huduser.gov download pages sit behind an AWS WAF challenge (HTTP 202 `x-amzn-waf-action: challenge`, 2026-11-12), so they are never scraped.
- Rows:
- `ZipDistrictRow(congress, zip, state_fips, district, weight, method, vintage, source_file)` → `market.zip_district`, with method ∈ `("census-zcta-land", "hud-res", "hud-bus")`;
- `CountyDistrictRow(congress, county_fips, district, weight, source_file)` → `market.county_district`.
- Parsers:
- `parse_census_zcta(text, congress) -> list[ZipDistrictRow]`:
- `weight = AREALAND_PART / AREALAND_ZCTA5_20`;
- rows with a blank ZCTA are skipped;
- `GEOID_CD{c}_20[:2]` is the state and `[2:]` the district (`"00"` at-large); district `"ZZ"` is skipped.
- `parse_census_county(text, congress)`, with weights by `AREALAND_PART` over the county's land area.
- `parse_hud(payload, congress) -> list[ZipDistrictRow]` gives two rows per result, `hud-res` = `res_ratio` and `hud-bus` = `bus_ratio`.
- `zip_weights(con, congress, *, prefer=("hud-bus", "census-zcta-land")) -> dict[str, list[tuple[str, str, float]]]` maps a ZIP to `(state_fips, district, weight)` from the first method that has the ZIP.
- CLI `stack market districts [--congress 118 --congress 119] [--hud-year Y --hud-quarter Q --hud-congress C] [--write]`:
- Census files are cached at `data/market/geo/census/`, each with one bib Download and the file attached (public);
- HUD is cached at `data/market/geo/hud/zip_cd/<y>q<q>.json`; without a token the CLI prints `gap: hud: HUD_API_TOKEN not in .env — key pending (ci@mail.fhirworx.io)`.
- [ ] **Step 1: Failing tests:**
- `test_census_zcta_weights`: ZCTA 10001 split 3:1 → weights 0.75/0.25; a blank-ZCTA row is skipped; GEOID `"0100"` → state `"01"`, district `"00"`; `"ZZ"` is skipped.
- `test_census_real_header_row`: the real header line plus the row `…|0101|…|36009|…|417245447|…|4938464|0` → weight `pytest.approx(0.0118358727)`.
- `test_hud_rows_two_methods`: res 0.6, bus 0.9 → two rows.
- `test_zip_weights_prefer_hud_bus_then_census`.
- `test_hud_without_token_records_gap` (CLI).
- `test_roundtrip[zip_district|county_district]`.
- [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff.
- [ ] **Step 5: Live:** run `ENV stack market districts --congress 118 --congress 119 --write`; add the `--hud-*` flags once the token lands, recording which CD vintage HUD returns. Report ZIPs per congress and the share of ZIPs whose weights sum to 1 ± 0.001.
- [ ] **Step 6: Commit** `git add src/market/districts.py src/market/tables.py src/cli/market.py tests/market/test_districts.py tests/market/test_tables.py tests/market/fixtures/tab20_cd11920_zcta520_head.txt tests/market/fixtures/hud_zip_cd.json tests/cli/test_market_cli.py && git commit -m "feat(market): ZCTA/county to congressional district crosswalks (Census rel files; HUD USPS when keyed)"`
---
### Task 9: District footprint of exposed companies and family billers — `stack market footprint`, `stack market district-view`
**Files:**
- Create: `src/market/footprint.py`, `tests/market/fixtures/hha_all_owners_head.csv` (synthetic, 4 rows), `prov_svc_head.csv` (synthetic, 5 rows, real header)
- Modify: `src/market/tables.py`, `src/cli/market.py`
- Test: `tests/market/test_footprint.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`
**Interfaces:**
- Consumes:
- Tasks 1, 6 and 8;
- `cms.{hospital,home_health_agency,hospice,skilled_nursing_facility,federally_qualified_health_center,rural_health_clinic}_enrollments` (`enrollment_id`, `ccn`, `organization_name`, `doing_business_as_name`, `zip_code`);
- the hoarded All Owners CSVs `data/public/lead_enrichment/cms_<kind>_all_owners/<snapshot>/*_All_Owners_*.csv` (`ENROLLMENT ID`, `ORGANIZATION NAME - OWNER`, `ROLE TEXT - OWNER`, `PERCENTAGE OWNERSHIP`);
- `terminology.provider` (Tuva 2025-09 NPPES cut; `entity_type_code = '2'`, `provider_organization_name`, `parent_organization_name`, `practice_zip_code`);
- the hoarded `data/public/lead_enrichment/cms_medicare_physician_other_practitioners_by_provider_and_service/*/PHY_R26_P05_V10_D24_Prov_Svc.csv` (data year 2024);
- `pfs.families` codes.
- Produces:
- Rows:
- `SiteRow(entity_key, site_kind, site_id, name, zip, state, note, source, source_file)` → `market.entity_site`, with site_kind ∈ `("facility", "org_npi")`;
- `FootprintRow(level, key, congress, state_fips, district, year, n_providers, n_sites, services, payment, method, source)` → `market.footprint`, with level ∈ `("entity", "family")`.
- Site and biller extraction:
- `owner_sites(owners_csv: Path, enrollments: Mapping[str, dict], entities) -> list[SiteRow]` matches the owner organization, or the facility's own name or DBA, with `alias_match(prefix=True)`. One site per (entity, enrollment), with the owner role kept in `note`.
- `npi_sites(rows, entities) -> list[SiteRow]`.
- `family_billers(con, csv: Path, codes) -> list[tuple[str, str, float, float]]` returns `(npi, zip5, services, payment)` with `payment = Tot_Srvcs × Avg_Mdcr_Pymt_Amt`. It is read by DuckDB `read_csv(all_varchar=true)` filtered on `HCPCS_Cd`.
- `rollup(items, weights, *, level, key, congress, year, method) -> list[FootprintRow]`:
- allocation is fractional by weight, so `n_providers` may be fractional;
- a ZIP with no weight goes to `district="??"`, never dropped.
- `@dataclass(frozen=True) DistrictView(state_fips, district, bioguide_id, member, party, health_committees: tuple[str, ...], n_providers, n_sites, services, payment, agi_k: float | None)`. `district_view(con, *, congress, level, key, year) -> list[DistrictView]` joins `market.footprint`, `market.congress_member` (house) and the `market.committee_assignment` snapshot for that congress. `agi_k` is always None here; Task 14 fills it from `econ.soi_zip`.
- CLI:
- `stack market footprint (--entities | --family KEY) [--congress 119] [--write]`; the cross-section is year 2024, CD119 (planner's call; earlier years and districts are a gap note);
- `stack market district-view (--entity KEY | --family KEY) [--congress 119] [--top 20]`.
- [ ] **Step 1: Failing tests:**
- `test_owner_sites_from_owners_csv`: 3 Amedisys owner rows over 2 enrollments → 2 sites.
- `test_short_alias_exact_only`: the alias `HCA` does not prefix-match `HCA SOMETHING`.
- `test_npi_sites_from_nppes`.
- `test_family_billers_from_prov_svc`: codes `{"99347","99348"}` → 3 of 5 rows; `payment == 12 * 85.5`.
- `test_rollup_fractional_and_unmatched_kept`: ZIP split 0.75/0.25 with services 100 → 75.0/25.0; an unknown ZIP → `"??"` row.
- `test_district_view_joins_member_and_committees`: the member is on HSWM02 → `health_committees == ("HSWM02",)`.
- `test_roundtrip[entity_site|footprint]`.
- `test_footprint_cli_family_offline` (CLI).
- [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff.
- [ ] **Step 5: Live:** run `ENV stack market footprint --entities --write`, then `ENV stack market footprint --family HOME-VISITS --write`, then `ENV stack market district-view --family HOME-VISITS --top 10`. Report the sites per entity (top 10), the share of HOME-VISITS payment that fell to `??`, and the top-10 districts with their members' health committees.
- [ ] **Step 6: Commit** `git add src/market/footprint.py src/market/tables.py src/cli/market.py tests/market/test_footprint.py tests/market/test_tables.py tests/market/fixtures/hha_all_owners_head.csv tests/market/fixtures/prov_svc_head.csv tests/cli/test_market_cli.py && git commit -m "feat(market): district footprint of exposed companies and family billers, joined to members and health committees"`
---
### Task 10: Event windows — `stack market events`
**Files:**
- Create: `src/market/events.py`, `tests/market/fixtures/fr_pi_2026-22743.json`, `fr_pi_2022-23873.json`, `ama_schedule_snippet.html` (synthetic)
- Modify: `src/market/tables.py`, `src/cli/market.py`
- Test: `tests/market/test_events.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`
**Interfaces:**
- Consumes:
- `pfs.rounds.rule_items(store, [])`, which gives `item_key`, `date_published` and `kind`;
- `fr_anchor_docs` (item_key → document_number);
- `fr_anchors`, `pfs.codetables.read_all_events`, `pfs.codetables.read_seats`;
- `pfs.seats.cdx_snapshots`/`fetch_snapshot`/`html_text`/`is_refusal`;
- Task 1.
- Produces:
- `FR_PI = "https://www.federalregister.gov/api/v1/public-inspection-documents/{doc}.json"` (`filed_at`, ISO with offset), pacer 0.5 s, cached at `data/market/fr/pi/<doc>.json`. A 404 means no public-inspection record (pre-2013 rules).
- `EventRow(event_id, kind, event_date: date, after_close: bool, precision, scope, target, item_key, p_id: int, document_number, filed_at, note)` → `market.event`:
- kind ∈ `("rule-proposed", "rule-final", "ruc-meeting", "panel-meeting")`;
- precision ∈ `("day", "month")`;
- scope ∈ `("program", "family")`.
- `release(filed_at: str | None, published: date) -> tuple[date, bool, str]`:
- `(local ET date of filed_at, time ≥ 16:00 ET, "")`;
- without `filed_at`: `(published, False, "event date is FR publication; public-inspection time unknown")`.
- `rule_events(rules: Sequence[dict], docs: Mapping[str, str], pi: Mapping[str, dict]) -> list[EventRow]`:
- scope `program`;
- `event_id = f"{kind}:{document_number}"`.
- `family_rule_events(events, fr_events_by_code, codes, family) -> list[EventRow]`: copies with scope `family` for rules whose `item_key` anchors a `pfs.code_event` FR row (`source == "fr"`) of a family code.
- `MEETING_RE` matches `"<Month> <YYYY> (RUC|CPT Editorial Panel|CPT Panel) meeting"` and `"(RUC|CPT Editorial Panel|Panel) (at|during|in) (its|their) <Month> <YYYY> meeting"`.
- `meeting_mentions(paragraphs: Iterable[tuple[str, int, str]]) -> list[EventRow]`:
- month precision, `event_date` = the 1st of the month;
- `event_id = f"{kind}:{YYYY-MM}"`;
- the first cite is kept (lowest item_key, then p_id).
- `ama_schedule(text: str, body: str) -> list[EventRow]`: day precision from `"<Month> D–D, YYYY"` ranges, with the start day as `event_date`.
- `@dataclass(frozen=True) Tenure(body, specialty, organization, seat_type, first_year, last_year, left_censored: bool, right_censored: bool)`. `tenures(seats, *, body="RUC") -> list[Tenure]` gives runs of consecutive roster years per specialty; specialty `""` is skipped; censoring is at the min and max roster year.
- `day0(event: EventRow, calendar: Sequence[date]) -> date | None`: the first trading day ≥ `event_date`, or the next one when `after_close`; None for month precision.
- CLI `stack market events [--refresh] [--ama-url URL --body RUC|CPT_PANEL] [--write]`. AMA pages go through the seats Wayback path; a refusal or bot page is a gap.
- [ ] **Step 1: Failing tests:**
- `test_after_close_filing_moves_day0`: doc `2026-22743`, filed `"2026-11-04T17:00:00.000-05:00"` → `event_date == date(2026,11,4)`, `after_close` True; `day0` on a calendar of weekdays → `date(2026,11,5)`.
- `test_day0_after_close_friday_is_monday`: `after_close` on Friday 2026-11-06 → `day0` is Monday 2026-11-09 on a weekday calendar.
- `test_before_close_same_day`: `"2026-07-14T15:59:00.000-04:00"` → day0 2026-07-14.
- `test_cy2023_final_release_not_publication`: doc `2022-23873`, filed `"2022-11-02T16:15:00.000-04:00"`, published 2022-11-18 → event_date 2022-11-02, after_close True.
- `test_publication_fallback_caveat`: 404 → `event_date` = published and `note == "event date is FR publication; public-inspection time unknown"`.
- `test_rule_events_from_rule_items`.
- `test_family_rule_events`: a HOME-VISITS FR code_event in `PETM225D` → a family event for `rule-final:2022-23873`.
- `test_meeting_mentions_from_fr`: `("5ITGVDJV",193,"…April 2022 RUC meeting…")`, `("N6MTT7VK",584,"…RUC at its January 2021 meeting…")`, `("5ITGVDJV",291,"…February 2025 CPT Editorial Panel meeting…")` → ids `ruc-meeting:2022-04`, `ruc-meeting:2021-01`, `panel-meeting:2025-02`, all month precision.
- `test_meeting_mentions_dedupe_keep_first_cite`.
- `test_ama_schedule_day_dates`: `"April 24–27, 2024"` → `date(2024,4,24)`, day precision.
- `test_tenures_runs_and_censoring`: FAMILY PRACTICE 2019–2026 → 1 tenure, censored on both sides; NEPHROLOGY 2019 and 2021–2022 → 2 tenures.
- `test_day0_month_precision_is_none`.
- `test_roundtrip[event]`.
- `test_events_cli_offline` (CLI).
- [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff.
- [ ] **Step 5: Live:**
- Run `ENV stack market events --write`. Report rule events by kind with and without public-inspection times, meeting events per body, and the tenure count per specialty.
- Print the RUC `(organization, specialty)` pairs and flag the known mislabels `Radiology → RADIATION ONCOLOGY` and `Cardiothoracic Surgery → CARDIOLOGY` (2024 rows) for a seats fix. Do not patch them here.
- Try the AMA RUC and CPT Panel meeting-schedule pages with `--ama-url`, recording the URLs or the gap.
- [ ] **Step 6: Commit** `git add src/market/events.py src/market/tables.py src/cli/market.py tests/market/test_events.py tests/market/test_tables.py tests/market/fixtures/fr_pi_*.json tests/market/fixtures/ama_schedule_snippet.html tests/cli/test_market_cli.py && git commit -m "feat(market): event windows — rule releases at public inspection, RUC/Panel meetings, seat tenures"`
---
### Task 11: Abnormal-return engine (market model, CAR, intervals)
**Files:**
- Create: `src/market/abnormal.py`
- Modify: `src/pfs/event_study.py` (extract `bootstrap_draws`, `percentile_interval`; `did_effect` uses them)
- Test: `tests/market/test_abnormal.py`, `tests/pfs/test_event_study.py`
**Interfaces:**
- Consumes: Task 1 only. Inputs are plain mappings, so all tests are synthetic.
- Produces:
- In `pfs.event_study`:
- `bootstrap_draws(values: Sequence[float], *, n_boot: int = _BOOT, seed: int = _SEED) -> list[float]` returns resampled means in draw order: `random.Random(seed)`, then for each draw `sum(rng.choice(values) for _ in range(n)) / n`.
- `percentile_interval(draws: Sequence[float]) -> tuple[float, float]` = `(sorted[int(0.025·B)], sorted[int(0.975·B) − 1])`.
- `did_effect` builds `[t_change − m for m in bootstrap_draws(changes)]` and returns the identical interval.
- In `market.abnormal`:
- `MODELS = {"market": ("SPY",), "sector": ("XLV",), "two-factor": ("SPY", "XLV")}`, `EST = (-250, -31)`, `MIN_EST = 120`, `WINDOWS = ((0, 0), (-1, 1), (0, 1), (-5, 5), (0, 10))`.
- `@dataclass(frozen=True) Fit(alpha: float, betas: tuple[float, ...], sigma2: float, n: int, xtx_inv: tuple[tuple[float, ...], ...])`; `fit(y: Sequence[float], xs: Sequence[Sequence[float]]) -> Fit | None` is OLS with an intercept via `numpy.linalg.lstsq`, `sigma2 = SSR/(n − k)`, None when `n ≤ k`.
- `car_from_arrays(f: Fit, y_win, xs_win) -> tuple[float, float]` (car, se), where `AR_t = y_t − α − Σβ x_t` and:
```
Var(CAR) = σ² · (L + ιᵀ X* (XᵀX)⁻¹ X*ᵀ ι) # Campbell–Lo–MacKinlay, X* = [1, x_win]
```
- `@dataclass(frozen=True) Car(event_id, unit, ticker, model, window: tuple[int, int], car: float | None, se: float | None, ci_low, ci_high, n_est: int, n_members: int, caveats: tuple[str, ...])`. The CI is `car ± 1.96·se`.
- `event_car(returns: Mapping[date, float], factors: Mapping[str, Mapping[date, float]], calendar: Sequence[date], d0: date, *, model="market", est=EST, window=(-1, 1), min_est=MIN_EST, event_id="", unit="", ticker="", other_events: Sequence[date] = ()) -> Car`. Offsets count trading days in `calendar`. Exact caveat copy:
- `f"short estimation window (n={n})"`, which sets car to None;
- `"missing returns in event window"`, which sets car to None;
- `f"estimation window overlaps event on {d}"`;
- `"sector index absorbs sector-wide effects"` for the `sector`/`two-factor` models.
- `@dataclass(frozen=True) Pooled(unit, model, window, mean_car, ci_low, ci_high, n_events, caveats)`; `pooled(cars: Sequence[Car]) -> Pooled` gives the mean of non-None CARs and a bootstrap CI via `bootstrap_draws` + `percentile_interval` (None when n < 2); caveat `"fewer than 5 events"` when n < 5.
- Every Car and Pooled includes the caveat `"association, not a causal estimate"`.
- [ ] **Step 1: Failing tests:**
- `test_fit_recovers_alpha_beta`: `y = 0.001 + 1.5x` exactly over 130 points → α 0.001, β 1.5, σ² 0 (abs 1e-12).
- `test_car_variance_clm`: fit on x `[-1,0,1]`, y `[-0.9,-0.2,1.1]` → α 0.0, β 1.0, σ² 0.06; window x* `[1,2]`, y* `[1.5,2.0]` → car 0.5, se `pytest.approx(0.6855654600)`.
- `test_car_window_sum`: firm = market + 0.02 on day 0 → CAR(0,0) = CAR(−1,1) = 0.02.
- `test_short_estimation_window_caveat`: n = 100 → car None, caveat `"short estimation window (n=100)"`.
- `test_missing_window_return`.
- `test_offsets_count_trading_days`: with `d0` Monday 2026-11-09 on a weekday calendar, window (−1,1) covers Friday 2026-11-06, Monday and Tuesday 2026-11-10, and the estimation offsets skip weekends.
- `test_two_factor_and_sector_models`: betas of length 2 and 1; the sector caveat is present.
- `test_estimation_overlap_caveat`.
- `test_pooled_bootstrap_deterministic`: two calls give equal CIs; n = 3 → caveat `"fewer than 5 events"`.
- In `tests/pfs/test_event_study.py`:
- `test_percentile_interval_indices`: draws `range(500)` → `(12, 486)`;
- `test_bootstrap_draws_reproducible`;
- `test_did_effect_interval_unchanged_after_refactor`: before editing, record `(effect, ci_low, ci_high)` of `did_effect` on the module's existing fixture inputs as literals, and assert them after.
- [ ] **Step 2: Run** `tests/market/test_abnormal.py tests/pfs/test_event_study.py` → FAIL. **Step 3: Implement.** **Step 4: Run** `tests/market/test_abnormal.py tests/pfs/test_event_study.py tests/notebooks/test_event_study_nb.py` → PASS; coverage; ruff.
- [ ] **Step 5: Commit** `git add src/market/abnormal.py src/pfs/event_study.py tests/market/test_abnormal.py tests/pfs/test_event_study.py && git commit -m "feat(market): market-model abnormal returns with CLM intervals; shared bootstrap helpers in pfs.event_study"`
---
### Task 12: Portfolios, stored results, seat-tenure contrast — `stack market car`, `stack market tenure`
**Files:**
- Create: `src/market/portfolio.py`
- Modify: `src/market/tables.py`, `src/cli/market.py`
- Test: `tests/market/test_portfolio.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`
**Interfaces:**
- Consumes: Task 1 (links, securities, filings: 8-K items), Task 2 (fts links), Task 3 (`read_returns`, `trading_days`), Task 10 (`EventRow`, `day0`, `tenures`), Task 11.
- Produces:
- `@dataclass(frozen=True) Member(entity_key, ticker)`.
- `members(con, *, scope, target, on: date, strict=False) -> tuple[list[Member], list[str]]` returns the members and the linked entity keys with no usable price series (the gap):
- a member is linked by scope/target, has a security valid on `on`, and is not ended before `on`;
- `strict` keeps only `curated` and `revenue_exposure` strengths.
- `portfolio_returns(returns: Mapping[str, Mapping[date, float]], members) -> dict[date, float]`: equal weight across the members that have a return that day.
- `earnings_days(filings: Sequence[FilingRow]) -> set[date]`: 8-K filings whose items include `2.02`, dated by `accepted` (UTC → ET; at or after 16:00 counts as the next day).
- `CarRow(event_id, scope, target, unit, ticker, model, win_lo, win_hi, car, se, ci_low, ci_high, n_est, n_members, n_gap, caveats, computed_on: date)` → `market.abnormal_return`, with caveats as a JSON list. `write_cars(con, scope, target, model, rows)` and `read_cars(con, scope, target, model=None) -> list[CarRow]`.
- `run_events(con, events, *, scope, target, model, windows, strict) -> list[CarRow]`:
- for each day-precision event, one row per member ticker (`unit="firm"`), plus one portfolio row (`unit=f"portfolio:{scope}:{target}"`) whose market model runs on the portfolio series;
- plus one pooled row per window (`event_id="pooled"`);
- extra caveat `"earnings release (8-K Item 2.02) in window"` when a member's earnings day falls in the window.
- `year_bhar(port: Mapping[date, float], mkt: Mapping[date, float], year: int, *, min_days=200) -> float | None` = `Π(1+r_p) − Π(1+r_m)` over that year's common trading days.
- `@dataclass(frozen=True) TenureContrast(specialty, mean_on, mean_off, diff, ci_low, ci_high, n_on, n_off, caveats)`; `contrast(rows: Sequence[TenureRow]) -> TenureContrast`:
- the CI pairs `bootstrap_draws(on, seed=_SEED)` with `bootstrap_draws(off, seed=_SEED + 1)`;
- None when either group has fewer than 2 years;
- caveats always include `"association, not a causal estimate"` and `"roster years 2019–2026 only"`.
- `TenureRow(specialty, year, on_ruc: bool, bhar: float | None, n_members)` → `market.tenure_return`.
- CLI:
- `stack market car (--family KEY | --specialty S | --program) [--kind rule-final --kind rule-proposed …] [--model market] [--window -1,1 …] [--strict] [--write]` prints the portfolio and pooled rows; the defaults are `model=market` and all `WINDOWS`;
- `stack market tenure (--specialty S | --all) [--write]`.
- [ ] **Step 1: Failing tests:**
- `test_members_as_of_day0_include_later_delisted`: OSH linked to HOME-VISITS is a member on 2022-11-03 but not on 2024-11-05.
- `test_no_price_series_counted_as_gap`.
- `test_equal_weight_daily_rebalanced`: returns 0.1 and 0.0 → 0.05; when one is missing → the other's return.
- `test_strict_members_only_curated_or_revenue_exposure`.
- `test_earnings_in_window_caveat`: 8-K 2.02 accepted `2022-11-03T13:00:00.000Z` → 2022-11-03, inside (−1,1) around 2022-11-03.
- `test_month_precision_event_skipped`.
- `test_year_bhar`: `[0.1, −0.1]` vs `[0, 0]` with `min_days=2` → `pytest.approx(-0.01)`.
- `test_tenure_contrast`: on `[0.1, 0.3]`, off `[0.0, 0.2]` → diff 0.1, n_on 2, n_off 2; off `[0.0]` alone → CI None.
- `test_roundtrip[abnormal_return|tenure_return]`.
- `test_car_cli_writes_rows` (CLI): synthetic SPY and 2 tickers in `:memory:` → firm, portfolio and pooled rows, publish once.
- `test_tenure_cli_prints_contrast` (CLI).
- [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff.
- [ ] **Step 5: Live:**
- Run `ENV stack market car --family HOME-VISITS --kind rule-final --kind rule-proposed --write`, then `ENV stack market car --program --write`, then `ENV stack market tenure --all --write`.
- Report per family: events, members per event (and the gap count), and caveat frequencies. Give the CY2023 final (`rule-final:2022-23873`, day 0 2022-11-03) portfolio CAR(−1,1) with its CI, internally only.
- [ ] **Step 6: Commit** `git add src/market/portfolio.py src/market/tables.py src/cli/market.py tests/market/test_portfolio.py tests/market/test_tables.py tests/cli/test_market_cli.py && git commit -m "feat(market): family/specialty/program portfolios, stored CARs with caveats, RUC-tenure return contrast"`
---
### Task 13: FRED, IRS SOI and company revenue — `stack market fred`, `stack market soi`, `stack market revenue`
**Files:**
- Modify: `src/econ/fred.py`, `src/econ/irs_soi.py`, `src/market/edgar.py`, `src/market/tables.py`, `src/cli/market.py`
- Create: `tests/market/fixtures/companyfacts_snippet.json` (synthetic)
- Test: `tests/econ/test_fred_irs.py`, `tests/market/test_enrich_sources.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`
**Interfaces:**
- Consumes: Task 1; the existing `econ.fred.SeriesRow`.
- Produces:
- `econ.fred`:
- `FREDGRAPH = "https://fred.stlouisfed.org/graph/fredgraph.csv?id={series}"` (keyless; probed 2026-11-12);
- `parse_fredgraph_csv(text: str, series_id: str) -> tuple[SeriesRow, ...]` accepts the header `observation_date,<ID>` or `DATE,<ID>`, treats `.` or empty as None, and raises `ValueError` on any other header (an HTML page returned with status 200 for an unknown id).
- `market.tables.FRED_SERIES = ("CPIAUCSL", "CPIMEDSL", "CUSR0000SAM2", "PCU621111621111", "PCU621610621610", "VIXCLS", "USREC", "DTB3")`. These are the CPI (all items, medical care, medical care services), the PPI for physician offices and for home health, VIX, recessions and the 3-month T-bill. All were verified 2026-11-12. Table `econ.fred(series_id, year, period, value, item_key)`.
- `econ.irs_soi`:
- `SOI_ZIP = "https://www.irs.gov/pub/irs-soi/{yy}zpallagi.csv"`;
- `ZipIncome(year, zip, state_fips, n_returns: int | None, agi_k: int | None)`;
- `parse_soi_zip(csv_text: str, year: int) -> tuple[ZipIncome, ...]` sums `N1` and `A00100` over `agi_stub`, drops zipcodes `00000` and `99999`, and zero-pads to 5 digits. Table `econ.soi_zip`.
- `market.edgar`:
- `COMPANYFACTS = "https://data.sec.gov/api/xbrl/companyfacts/CIK{cik10}.json"`;
- `RevenueRow(cik, fy: int, fp, form, concept, value: float, filed: date, adsh)` → `market.revenue`;
- `revenue_rows(facts) -> list[RevenueRow]` uses `us-gaap:Revenues`, falling back to `RevenueFromContractWithCustomerExcludingAssessedTax`, `fp == "FY"` and 10-K forms only, keeping the latest `filed` per fy.
- CLI:
- `stack market fred [--series ID …] [--write]`: cache `data/market/econ/fred/<ID>.csv`, one bib Download per series with the file attached;
- `stack market soi [--year Y …] [--write]`: defaults to 2013 through the newest year whose URL returns CSV; cache `data/market/econ/soi/<yy>zpallagi.csv`;
- `stack market revenue [--linked] [--write]`.
- [ ] **Step 1: Failing tests:**
- `test_parse_fredgraph_csv`: `"observation_date,CPIMEDSL\n1947-01-01,13.200\n2026-10-01,.\n"` → `(SeriesRow("CPIMEDSL",1947,"1947-01-01",13.2), SeriesRow("CPIMEDSL",2026,"2026-10-01",None))`.
- `test_fredgraph_old_header`.
- `test_fredgraph_html_rejected`.
- `test_soi_zip_sums_agi_stubs`: zip 1001 with stubs 1–6 → `"01001"` and exact sums; `00000`/`99999` dropped.
- `test_companyfacts_revenue`: duplicate FY 2023 rows → the latest filed is kept; 10-Q rows dropped.
- `test_roundtrip[fred|soi_zip|revenue]`.
- `test_fred_cli_offline` and `test_soi_cli_offline` (CLI).
- [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; run the coverage gate with `--cov=src/econ` added; ruff.
- [ ] **Step 5: Live:** run `ENV stack market fred --write`, `ENV stack market soi --write` and `ENV stack market revenue --linked --write`. Report rows per series, the SOI years loaded, and the revenue coverage of linked companies.
- [ ] **Step 6: Commit** `git add src/econ/fred.py src/econ/irs_soi.py src/market/edgar.py src/market/tables.py src/cli/market.py tests/econ/test_fred_irs.py tests/market/test_enrich_sources.py tests/market/test_tables.py tests/market/fixtures/companyfacts_snippet.json tests/cli/test_market_cli.py && git commit -m "feat(market): FRED (fredgraph), IRS SOI ZIP income and EDGAR company revenue for enrichment"`
---
### Task 14: Enrichment joins (readers for the joint displays) — `stack market joint`
**Files:**
- Create: `src/market/enrich.py`
- Modify: `src/market/footprint.py` (`district_view` fills `agi_k` from `econ.soi_zip`), `src/cli/market.py`
- Test: `tests/market/test_enrich.py`, `tests/market/test_footprint.py`, `tests/cli/test_market_cli.py`
**Interfaces:**
- Consumes: Task 3 (returns, `market.series`), Task 12 (`portfolio_returns`, `year_bhar`), Task 13 (`econ.fred`, `econ.soi_zip`), `bls.oews`, `bls.ces`, `pfs.utilization`, `pfs.event_study.family_series`, `pfs.psps.read_by_specialty`.
- Produces:
- `SPECIALTY_SOC: dict[str, tuple[str, ...]]`: FAMILY PRACTICE 29-1215, INTERNAL MEDICINE 29-1216, PEDIATRIC MEDICINE 29-1221, PSYCHIATRY 29-1223, ANESTHESIOLOGY 29-1211, CARDIOLOGY 29-1212, DERMATOLOGY 29-1213, EMERGENCY MEDICINE 29-1214, NEUROLOGY 29-1217, OBSTETRICS GYNECOLOGY 29-1218, PATHOLOGY 29-1222, DIAGNOSTIC RADIOLOGY 29-1224, OPHTHALMOLOGY 29-1241, ORTHOPEDIC SURGERY 29-1242, GENERAL SURGERY 29-1249, PHYSICAL THERAPIST 29-1123, NURSE PRACTITIONER 29-1171, PHYSICIAN ASSISTANT 29-1071. All are present in `bls.oews`; the specialist codes start in 2019 or 2021.
- `FAMILY_INDUSTRY: dict[str, tuple[str, ...]] = {"HOME-VISITS": ("621100", "621600")}`, default `("621100",)` (CES).
- `@dataclass(frozen=True) JointYear(year: int, values: dict[str, float | None], sources: dict[str, str])`. Column keys are exactly:
- `market:<scope>:<target>:bhar`;
- `util:<family>:n_services`;
- `psps:<specialty>:allowed`;
- `oews:<soc>:a_median`;
- `ces:<industry>:employment` (annual mean);
- `fred:<id>` (annual mean);
- `sources` maps each key to its table plus the item_key.
- `joint_annual(con, *, family: str | None = None, specialty: str | None = None, years: Sequence[int]) -> list[JointYear]`. A missing input is None, never 0.
- `JointMonth(year, month, values, sources)`; `joint_monthly(con, *, symbols, ces, fred, start: date, end: date) -> list[JointMonth]`, where `market:<symbol>:tri` comes from `market.series`.
- `annual_tri(con, symbol) -> dict[int, float]`, the mean monthly TRI per year (the #797 notebook input).
- CLI `stack market joint (--family KEY | --specialty S) [--from 2015] [--to 2024] [--monthly]` prints the table (internal).
- [ ] **Step 1: Failing tests:**
- `test_joint_annual_columns_and_sources`: in-memory tables give exact values for each column key.
- `test_joint_missing_inputs_are_none_not_zero`.
- `test_joint_monthly_alignment`.
- `test_annual_tri`.
- `test_district_view_agi_from_soi` (in `test_footprint.py`).
- `test_specialty_soc_codes_in_oews`: live, skipped without the replica.
- `test_joint_cli_prints` (CLI).
- [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff.
- [ ] **Step 5: Live:** run `ENV stack market joint --family HOME-VISITS --from 2015 --to 2024` and `ENV stack market joint --specialty "FAMILY PRACTICE"`. Report which columns are populated by year.
- [ ] **Step 6: Commit** `git add src/market/enrich.py src/market/footprint.py src/cli/market.py tests/market/test_enrich.py tests/market/test_footprint.py tests/cli/test_market_cli.py && git commit -m "feat(market): joint readers — market series beside BLS, FRED, SOI, PSPS and utilization"`
---
### Task 15: #797 notebook market panel, acceptance, roll-out
**Files:**
- Modify: `notebooks/event_study.py` (a new section "Market series (#797)", placed after the wages/employment section)
- Test: `tests/notebooks/test_event_study_nb.py`
**Interfaces:**
- Consumes: `market.enrich.annual_tri` (Task 14), `market.tables.read_series` (Task 3), `market.tables.read_cars` (Task 12), and `pfs.event_study.pre_post`.
- Produces:
- For the selected exposure, `pre_post` runs on `annual_tri` for XLV, IHF and SPY and on the family portfolio's annual BHAR. There is no control set, and the panel says so in the same words as the wages panel.
- Below it sits a table of stored `market.abnormal_return` portfolio rows for that family's rule events, with their caveats.
- Each panel gets a provenance line built from the bib keys.
- When the market tables are absent, `not_built("stack market prices --write")`.
- No causal wording: the existing `test_no_causal_language` must still pass.
- [ ] **Step 1: Failing tests:**
- `test_market_panel_headless_not_built`: with no replica, the output contains `stack market prices --write`.
- `test_market_panel_live_ccm_2015_apcm_2025`: `_live`; the fixtures `("99490", 2015, "becomes-payable")` and `("G0556", 2025, "becomes-payable")` render a market table with an XLV row and provenance.
- [ ] **Step 2: Run** `tests/notebooks/test_event_study_nb.py` → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS.
- [ ] **Step 5: Full gate:** `uv run --no-sync pytest tests/ -x --cov=src --cov-report=term-missing --cov-fail-under=99 -q -n auto -p no:cacheprovider` → PASS; ruff on all touched paths.
- [ ] **Step 6: Commit** `git add notebooks/event_study.py tests/notebooks/test_event_study_nb.py && git commit -m "feat(notebooks): market series panel in the event-study notebook (#797)"`
- [ ] **Step 7: Acceptance (live):**
1. Confirm `ENV stack market gaps` lists only the pending-key sources (fec, congress bills, hud, and lda only if a key was expected).
2. Confirm, in the replica, that every `market.entity_link` row has a non-empty `cite_ref` or `item_key`, and that no `market.*` column contains a key value. Check with `grep -c` of the key values over the replica's `cms.ingest_log`, `data/bib.sqlite` URLs and `data/market` file names; each must be 0.
3. Confirm that no file under `data/market/tiingo` is attached in bib.
4. Confirm that `market.price_daily` covers SPY from 1993-01-29.
5. Confirm that HOME-VISITS rule events have stored portfolio CARs.
6. Push, and let the nightly `notebooks-integration` run render the panel.
7. Then comment on #797 (row counts and commit SHAs only, no prices) and close it. Use curl with `-H "User-Agent: curl/8.5.0"` and the `GITEA_TOKEN` from `.env`.
---
## Self-review (planner)
- **Spec coverage:**
- exposed entities, cited, with delisted firms kept: Tasks 1–2;
- prices with a market and a sector index, hoarded: Task 3;
- rule, RUC and Panel windows and seat tenures: Task 10;
- market-model event studies with intervals and caveats: Task 11;
- portfolios by family and specialty, and descriptive tenure comparisons: Task 12;
- enrichment: BLS and PSPS through readers, FRED and SOI in Task 13, NPPES in Task 9's footprint, SEC filing text in Task 2 and company revenue in Task 13, all joined in Task 14;
- LDA and FEC: Tasks 4–5;
- Congress members, committees and bills: Tasks 6–7;
- district crosswalks and footprints: Tasks 8–9;
- one api.data.gov key with overrides: the Task 1 `KEY_ENV`;
- #797: Tasks 3 and 15.
- **Gaps stated rather than built:**
- NAICS (EDGAR records SIC only);
- segment revenue (companyfacts omits dimensional facts);
- committee assignments before 2013;
- district geography before CD118;
- footprint years other than 2024.
- **Type names:** checked across tasks: `EntityRow`, `SecurityRow`, `LinkRow`, `FilingRow`, `PriceRow`, `MonthlyRow`, `EventRow`, `Car`, `CarRow`, `TenureRow`, `JointYear`, `ZipDistrictRow`, `FootprintRow`, `DistrictView`.
- **Function names:** `day0` is defined in Task 10 and used in Tasks 11–12. `bootstrap_draws` and `percentile_interval` are defined in Task 11 and used in Task 12.

View File

@@ -268,7 +268,7 @@ This extends the 2026-11-11 design.
- Descriptive comparisons of returns across seat tenures.
- These are explicitly associations, not causal claims, unless the design supports it.
- **Enrichment.** BLS OEWS and CES (wages and employment by occupation and industry, already in the stack), FRED (`src/econ/fred.py`), IRS SOI (`src/econ/irs_soi.py`), CMS PSPS and utilization, NPPES, and SEC filings text (exposure statements, segment revenue). These sit beside the market series in the joint displays.
- **Proposed additions (my call, for the power side):** Senate LDA lobbying disclosures and FEC filings by specialty societies and exposed firms, public and keyed by organization. They join the stakeholder units' power stakes.
- **Power-side sources (user-confirmed 2026-11-12):** Senate LDA lobbying disclosures (now served at lda.gov; lda.senate.gov closed 2026-07-31) and FEC filings by specialty societies and exposed firms, public and keyed by organization. They join the stakeholder units' power stakes.
**Congress and districts (user, 2026-11-12)**
@@ -288,7 +288,7 @@ This layer covers who sits in which district and on which committee, and where e
- Public data only. No material non-public information.
- Every data-source account and key is registered to ci@mail.fhirworx.io (user rule), with keys in `.env`.
- Honest user agent. SEC requires a contact; use the stack's own `ci@fhirworx.io`, never a personal address.
- Honest user agent. SEC requires a contact; use the stack's own `ci@mail.fhirworx.io`, never a personal address.
- Respect each provider's terms. Prices stay inside the stack and are never republished.
**Build order:** slice S4b, right after S4 (Evaluate). It shares its event-study engine and the exposure calendar. Entity mapping and EDGAR work can start before the price provider is settled.