diff --git a/docs/superpowers/plans/2026-11-12-market-tier.md b/docs/superpowers/plans/2026-11-12-market-tier.md index 2d82525..6611a57 100644 --- a/docs/superpowers/plans/2026-11-12-market-tier.md +++ b/docs/superpowers/plans/2026-11-12-market-tier.md @@ -2,105 +2,217 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -**Goal:** Track how the money CMS law moves shows up in markets and in power: a cited map of public companies (and lobbying organizations) exposed to each code family and specialty, their daily prices against SPY/XLV, lobbying (LDA), PAC money (FEC), Congress members, committees, bills and district footprints, event windows (rule releases, RUC/Panel meetings, seat tenures), a market-model abnormal-return engine with portfolios, and enrichment readers that put BLS, FRED, SOI and PSPS beside the market series. This closes #797. +**Goal:** Track how the money CMS law moves shows up in markets and in power. The tier builds: +- a cited map of the public companies and lobbying organizations exposed to each code family, specialty and program; +- their daily prices against SPY and XLV; +- lobbying (LDA), PAC money (FEC), Congress members, committees, bills and district footprints; +- event windows: rule releases, RUC/Panel meetings and seat tenures; +- a market-model abnormal-return engine with portfolios; +- enrichment readers that put BLS, FRED, SOI and PSPS beside the market series. -**Architecture:** A new `market` package (DuckDB schemas `market` and `econ`) with one module per source. Each module follows the PSPS/seats pattern: an honest-UA client with a per-source pacer, a hoarded raw cache under `data/market/`, pure parsers to frozen dataclasses, delete-then-insert writers in `market/tables.py`, a bib citation, a `cms.ingest_log` row and a `gap:` line when a source refuses or a key is missing. A single `stack market` CLI group drives fetching (no DB lock held during network work), then short `duckdb_batch` writes and a replica publish. Analysis is pure library code: `market/abnormal.py` (market model, CAR, analytic and bootstrap intervals, sharing the bootstrap helpers extracted from `pfs/event_study.py`), `market/portfolio.py` and `market/enrich.py` readers. There is no UI; the S4 Evaluate view reads these tables, and the #797 notebook panel is the only display. +It closes #797. -**Tech Stack:** Python 3.13, uv, httpx, DuckDB (`data/aco.duckdb` via `conf.connect.duckdb_batch("aco")`, replica `data/replica/aco.ro.duckdb`), bib SQLite (`fr_anchors`, `fr_anchor_docs`, `items`), numpy (already locked; imported lazily), PyYAML, typer, pytest + xdist, the local Ollama pool for the one optional classification step. +**Architecture:** +- A new `market` package (DuckDB schemas `market` and `econ`) with one module per source. +- Each module follows the PSPS/seats pattern: + - an honest-UA client with a per-source pacer; + - a hoarded raw cache under `data/market/`; + - pure parsers; + - row dataclasses and delete-then-insert or upsert writers in `market/tables.py`; + - a bib citation and a `cms.ingest_log` row; + - a `gap:` line, plus a `market.source_gap` row, when a source refuses or a key is missing. +- One `stack market` CLI group fetches first, holding no DB lock, then writes in a short `duckdb_batch` and publishes the replica. +- Analysis is pure library code: `market/abnormal.py` (market model, CAR, analytic and bootstrap intervals, using bootstrap helpers extracted from `pfs/event_study.py`), `market/portfolio.py` and the readers in `market/enrich.py`. +- There is no UI. The S4 Evaluate view reads these tables, and the #797 notebook panel is the only display. -**Spec:** `docs/superpowers/specs/2026-11-12-cms-legal-epi-platform-design.md`, sections "Market and enrichment tier" and "Congress and districts" (commit c1288c2), slice S4b. Stakes context: `docs/superpowers/specs/2026-11-11-code-family-dossier-design.md` § Stakes. User decisions of 2026-11-12 are in `.superpowers/sdd/2026-11-11-code-family-dossier-home-visits/progress.md` (lines 428–433). Issue #797 (market series) is resolved by Tasks 3 and 15. +**Tech Stack:** Python 3.13, uv, httpx, DuckDB (`data/aco.duckdb` via `conf.connect.duckdb_batch("aco")`, replica `data/replica/aco.ro.duckdb`), bib SQLite (`fr_anchors`, `fr_anchor_docs`, `items`), numpy (already locked; imported only inside `market.abnormal` functions), PyYAML, typer, pytest with xdist, and the local Ollama pool for the one optional classification step. + +**Spec:** `docs/superpowers/specs/2026-11-12-cms-legal-epi-platform-design.md`, sections "Market and enrichment tier" and "Congress and districts" (commit c1288c2), slice S4b. +- Stakes context: `docs/superpowers/specs/2026-11-11-code-family-dossier-design.md` § Stakes. +- User decisions of 2026-11-12: `.superpowers/sdd/2026-11-11-code-family-dossier-home-visits/progress.md`, lines 428–433. +- Pre-flight scan and controller rulings M1–M11, D1–D5 and H1–H2: `.superpowers/sdd/2026-11-12-market-tier/{preflight.md,progress.md}`. +- Issue #797 (market series) is resolved by Tasks 3 and 15. ## Global Constraints -- Commit straight to `main` in `/home/kert/stack`. No branches, worktrees, merge commits, PRs, `git stash`. Before every commit run `git status --short`; if anything you did not stage is staged, stop and ask (another session shares this checkout). `git add` explicit paths only. -- Commit messages: conventional prefix (`feat(market): …`); no `Co-Authored-By` or any other trailer. -- Never run a write/ingest command from a worktree. Every live command runs from the main checkout through the env wrapper: `bash -c 'cd /home/kert/stack && set -a && . ./.env && set +a && uv run --no-sync '` (written **ENV** below). Before the first live run: `cat .venv/lib/python3*/site-packages/stack.pth` must print `/home/kert/stack/src` (else `uv sync --frozen`). Long runs: `mkdir -p .state/market` once, then `setsid nohup ENV … > .state/market/.log 2>&1 &` with a Monitor on the log; rerunning the same command resumes. -- DuckDB: write `data/aco.duckdb` only inside `conf.connect.duckdb_batch("aco")`, opened after all network work (fetch first, then a short batch that only writes), then `conf.connect.publish_replica("aco")`. `market.*` and `econ.*` modules never open a database at import. -- New tables: DDL in `market.tables._DDL`, a frozen dataclass, a delete-then-insert writer and a reader in `src/market/tables.py`; each table gets a round-trip case in `tests/market/test_tables.py::ROUNDTRIP`. -- **User-Agent** on every market client: `stack-pfs-dossier/1.0 (ci@mail.fhirworx.io)` (`market.http.USER_AGENT`). SEC requires the contact; it is the stack's own mailbox, never a personal address. Never a browser UA. A 401/403, an exhausted 429 or a bot/challenge page stops that source and becomes a `gap:` line plus a `market.source_gap` row (with `--write`); never work around it. -- **Keys** (all accounts registered to ci@mail.fhirworx.io, user rule; values only in `.env`): - - `TIINGO_API_KEY` (present, verified 2026-11-12); - - `API_DATA_GOV_KEY` (pending, the controller captures it from the mailbox), serving openFEC and Congress.gov, with optional overrides `FEC_API_KEY` and `CONGRESS_API_KEY`; - - `LDA_API_KEY` (optional, lda.gov account; anonymous access works); - - `HUD_API_TOKEN` (pending, HUD USPS API). - - Keys travel only in headers (`Authorization: Token …`, `X-Api-Key: …`, `Authorization: Bearer …`), never in a URL, cache path, bib item, ingest_log row, log line or exception text. - - A task whose key is missing prints the gap, records it, writes nothing else, exits 0, and stays fully testable offline. -- **Hoard:** never overwrite a source file. Raw responses live under `data/market//…`; a re-download with different bytes lands beside the old file as `.` (`pfs.seats.save_snapshot`). Readers use the newest copy (`pfs.psps.newest_cache`). -- **Prices are internal-use only** (Tiingo Starter licence: "you may not display or share the data with another person or organization"): - - they stay in `data/market/tiingo/` and the `market.price_daily`/`market.series` tables; - - never a bib attachment (Zotero syncs), never in git, never in a commit message, issue comment or report beyond row counts; - - test fixtures use synthetic prices only. -- Public data only. No material non-public information. No ACO/HarmonyCares/WNOT/Wakely material; the seed list never names them. -- Local models only, and only in Task 2's optional passage classifier: Ollama via `llm.pool.HostPool` / `llm.structured.json_extractor` with `cfg.instruct_model`, resumable through `pfs.runlog` with `--limit`/`--dry-run`. Never a cloud LLM API. -- No AMA text: a stored SEC passage is ≤ 400 characters of public filing text and is withheld when it shares an 8-word run (`pfs.origin.shares_run`) with any `pfs.cpt_code.descriptor` of a code it names. -- People: store names of members of Congress (public officials) and organizations as the public record names them. Do not store FEC Schedule A individual donors or LDA lobbyist personal names (not needed). +**Repository and commits** +- Commit straight to `main` in `/home/kert/stack`. No branches, worktrees, merge commits, PRs or `git stash`. +- Before every commit, run `git status --short`. If anything you did not stage is staged, stop and ask: another session (Plan A) shares this checkout. `git add` explicit file paths only, never a directory. +- Commit messages take a conventional prefix (`feat(market): …`) and no `Co-Authored-By` or any other trailer. +- **Shared tree (H1, H2):** + - Market tasks run one at a time, in task order. + - No market task edits `src/pfs/*` or `src/cli/pfs.py`, with one exception: Task 11 may edit `src/pfs/event_study.py` and `tests/pfs/test_event_study.py`. Market code only imports `pfs.*` (read-only). + - `src/cli/__init__.py` gets one complete edit, in Task 1. + - Keep the tree importable at every save, and commit right after the scoped tests pass. + - If the pre-commit hook fails only on Plan A's uncommitted work, rerun after Plan A commits. Never use `--no-verify`. + +**Running live commands** +- Never run a write or ingest command from a worktree. +- Every live command runs from the main checkout through the env wrapper `bash -c 'cd /home/kert/stack && set -a && . ./.env && set +a && uv run --no-sync '`, written **ENV** below. +- Before the first live run, `cat .venv/lib/python3*/site-packages/stack.pth` must print `/home/kert/stack/src`; otherwise run `uv sync --frozen`. +- Long runs: `mkdir -p .state/market` once, then `setsid nohup ENV … > .state/market/.log 2>&1 &` with a Monitor on the log. Rerunning the same command resumes. + +**Database and tables** +- Write `data/aco.duckdb` only inside `conf.connect.duckdb_batch("aco")`, opened after all network work (fetch first, then a short batch that only writes), then call `conf.connect.publish_replica("aco")`. +- `market.*` and `econ.*` modules never open a database at import. +- Every `market.*` and `econ.*` table has: + - its DDL in `market.tables._DDL`; + - a frozen row dataclass, a writer and a reader in `src/market/tables.py`; + - a round-trip case in `tests/market/test_tables.py::ROUNDTRIP`. + The parsers in `econ/*` return their own parse dataclasses, and the CLI converts them to the table rows, which carry `item_key`. + +**User-Agent and refusals** +- Every market client sends `stack-pfs-dossier/1.0 (ci@mail.fhirworx.io)` (`market.http.USER_AGENT`). SEC requires the contact. It is the stack's own mailbox, never a personal address, and never a browser UA. +- A 401/403, an exhausted 429 or a bot/challenge page stops that source and becomes a gap. Never work around it. + +**Keys** (every account registered to ci@mail.fhirworx.io, a user rule; values only in `.env`) +- `TIINGO_API_KEY`: present, verified 2026-11-12. +- `API_DATA_GOV_KEY`: present, verified. It serves openFEC and Congress.gov; `FEC_API_KEY` and `CONGRESS_API_KEY` are optional overrides (absent). +- `LDA_API_KEY`: optional; anonymous lda.gov works. +- `HUD_API_TOKEN`: **pending**. It is the only pending key. +- Keys travel only in headers (`Authorization: Token …`, `X-Api-Key: …`, `Authorization: Bearer …`). They never appear in a URL, cache path, bib item, ingest_log row, log line or exception text. +- **A command missing its key (M11)** prints its gap line, records the gap row (with `--write`) and writes nothing else. It exits 0 and stays fully testable offline. + +**Gaps (W23)** +- A gap's `source` is `:`: `sec-edgar:entities`, `sec-edgar:fts`, `sec-edgar:revenue`, `tiingo:prices`, `lda:lda`, `fec:fec`, `congress:members`, `legislators:committees`, `congress:bills`, `census:districts`, `hud:hud`, `fr:events`, `ama:events`, `fred:fred`, `irs-soi:soi`. +- Every successful `--write` run calls `clear_gap(con, source)` for its source in the same batch. + +**Hoard** +- Never overwrite a source file. +- Raw responses live under `data/market//…`. A re-download with different bytes lands beside the old file as `.` (`pfs.seats.save_snapshot`). +- Readers use the newest copy (`pfs.psps.newest_cache`, which orders by mtime). An identical re-download after a different one does not change which copy is newest; that issue is parked in Plan A (W21). + +**Prices (D5)** +- Raw prices are internal-use only (Tiingo Starter licence: "you may not display or share the data with another person or organization"). + - They stay in `data/market/tiingo/` and in `market.price_daily` and `market.series`. + - They never become a bib attachment (bib syncs to Zotero) and never go into git, a commit message or an issue comment. + - Test fixtures use synthetic prices only. +- CARs, BHARs and portfolio statistics are derived analytics. They may be shown anywhere inside the stack (CLI, notebook, ledger, task reports) but never published outside it. + +**Data rules** +- Public data only and no material non-public information. No ACO/HarmonyCares/WNOT/Wakely material; the seed list never names them. +- Local models only, and only in Task 2's optional passage classifier: + - Ollama via `cli.market._asker` (built like `cli.pfs._asker`: `llm.config.load()`, `llm.pool.HostPool.from_config`, `llm.structured.json_extractor`) with `cfg.instruct_model`; + - resumable through `pfs.runlog`, with `--limit` and `--dry-run`; + - never a cloud LLM API. +- **No AMA text (D3):** + - A stored SEC passage is at most 400 characters of public filing text. + - **Every** passage is checked against the 8-word runs of **all** `pfs.cpt_code.descriptor` texts, using the same tokenization as `pfs.origin.shares_run`. + - A passage that shares a run is withheld. +- **Names (D2):** members of Congress, organizations, and FEC Schedule A/B names are stored as the public record files them (user ruling). LDA lobbyist personal names are not needed and are not stored. - Market results are associations, not causal claims. Every stored result row carries its caveats. -- `cli/market.py` imports only stdlib, typer and httpx at module level (the api image has the `cli` extra only). numpy, duckdb and yaml are imported inside functions. -- Tests need no network, live DuckDB, Ollama or keys (httpx `MockTransport`, `duckdb.connect(":memory:")` + `ensure_tables`, `Store(":memory:")`, monkeypatched `cli.market._batch/_read/_store/_publish`, as in `tests/cli/test_pfs_cli.py`). Live tests skip without the replica. Coverage gate per task: `uv run --no-sync pytest -q -p no:cacheprovider -n auto tests/market tests/cli/test_market_cli.py --cov=src/market --cov=src/cli --cov-report=term-missing` and the `src/market/*` and `src/cli/market.py` lines must read ≥ 99% (CI runs `--cov=src --cov-fail-under=99`). +- **Imports (M1):** no `market.*` module imports numpy or pandas at module level. numpy is imported only inside `market.abnormal` functions, and pandas is never used. Reason: `stack` imports every command module at start-up (`cli/__init__`), and only the statistics need numpy. `tests/market/test_imports.py` checks this. + +**Tests, coverage and lint** +- Tests need no network, live DuckDB, Ollama or keys. Use httpx `MockTransport`, `duckdb.connect(":memory:")` with `ensure_tables`, `Store(":memory:")`, and monkeypatched `cli.market._batch/_read/_store/_publish`, as in `tests/cli/test_pfs_cli.py`. Live tests skip without the replica. +- **Coverage (D4):** the CLI tests for each `stack market` command cover its offline `--write` path, its no-write or dry-run path where it has one, and its refusal or gap path where it makes network calls. Each task lists them by name. +- Gate per task: `uv run --no-sync pytest -q -p no:cacheprovider -n auto tests/market tests/cli/test_market_cli.py --cov=src/market --cov=src/cli --cov-report=term-missing`. The `src/market/*` and `src/cli/market.py` lines must read ≥ 99%; CI runs `--cov=src --cov-fail-under=99`. - Lint: `uv run --no-sync ruff format && uv run --no-sync ruff check `. ## Review Focus -1. **Wrong event day.** A rule filed for public inspection after the 4 p.m. close is counted on the same trading day, or the FR publication date stands in for the release. The CY2023 final rule was filed 2022-11-02 16:15 ET and published 2022-11-18, 16 days later. Expected: day 0 is the first trading day whose close follows the release. Pinned in Task 10 (`test_after_close_filing_moves_day0`, `test_day0_after_close_friday_is_monday`, `test_cy2023_final_release_not_publication`, `test_publication_fallback_caveat`) and Task 11 (`test_offsets_count_trading_days`). -2. **Price series that are not the issuer's.** A later dividend rescales every past Tiingo `adjClose`, so naive appends jump at the pull boundary. A recycled ticker (`AGL` was AGL Resources before agilon's 2021-04-15 listing) attaches another company's history. Expected: returns are computed within a pull, and rows outside the security's validity window are dropped. Pinned in Task 3 (`test_incremental_pull_rebased_adjclose_keeps_returns`, `test_prices_clipped_to_security_window`). -3. **Survivorship.** Acquired or delisted firms disappear because EDGAR drops their tickers (`OSH` has `tickers: []`), or a 2015 event's portfolio is formed from today's members. Expected: delisted firms are kept, members are taken as of day 0, and missing price series are counted and shown as a gap. Pinned in Task 1 (`test_delisted_seed_entity_kept_without_current_ticker`) and Task 12 (`test_members_as_of_day0_include_later_delisted`, `test_no_price_series_counted_as_gap`). +1. **Wrong event day.** A rule filed for public inspection after the 4 p.m. close is counted on the same trading day, or the FR publication date stands in for the release. + - The CY2023 final rule was filed 2022-11-02 16:15 ET and published 2022-11-18, 16 days later. + - Expected: day 0 is the first trading day whose close follows the release. + - Pinned in Task 10 (`test_after_close_filing_moves_day0`, `test_day0_after_close_friday_is_monday`, `test_cy2023_final_release_not_publication`, `test_publication_fallback_caveat`) and Task 11 (`test_offsets_count_trading_days`). +2. **Price series that are not the issuer's.** + - A later dividend rescales every past Tiingo `adjClose`, so naive appends jump at the pull boundary. + - A recycled ticker attaches another company's history: `AGL` was AGL Resources before agilon's 2021-04-15 listing. + - Expected: returns are computed within a pull, and rows outside the security's validity window are dropped. + - Pinned in Task 3 (`test_incremental_pull_rebased_adjclose_keeps_returns`, `test_prices_clipped_to_security_window`). +3. **Survivorship.** + - Acquired or delisted firms disappear because EDGAR drops their tickers (`OSH` has `tickers: []`). + - A 2015 event's portfolio is formed from today's members. + - Expected: delisted firms are kept, members are taken as of day 0, and missing price series are counted as a gap. + - Pinned in Task 1 (`test_delisted_seed_entity_kept_without_current_ticker`) and Task 12 (`test_members_as_of_day0_include_later_delisted`, `test_no_price_series_counted_as_gap`). 4. **Money counted twice.** - LDA: an amendment or termination is summed beside the report it replaces, or an LD-2 amount is repeated per issue code. - FEC: memo rows (`memo_code` `X`) are summed with their parents. - Expected: one amount per filing period, the latest version, never split or repeated by issue. - Pinned in Task 4 (`test_amendment_supersedes_original`, `test_issue_codes_never_split_amount`) and Task 5 (`test_memo_rows_excluded_from_totals`). -5. **Secrets and licensed data leaking.** An api.data.gov or Tiingo key ends up in a cache path, a bib URL, `cms.ingest_log` or an `httpx` error message, or prices are attached to bib, where they would sync to Zotero. Pinned in Task 1 (`test_key_sent_in_header_never_in_url`, `test_refused_message_has_no_query_string`), Task 3 (`test_tiingo_cite_has_no_attachment`) and Task 5 (`test_fec_error_message_has_no_key`). +5. **Secrets and licensed data leaking.** + - A key ends up in a cache path, bib URL, `cms.ingest_log` or an `httpx` error. + - Prices get attached to bib. + - A passage quoting a CPT descriptor gets stored without naming the code. + - Pinned in Task 1 (`test_key_sent_in_header_never_in_url`, `test_refused_message_has_no_query_string`), Task 3 (`test_tiingo_cite_has_no_attachment`), Task 5 (`test_fec_error_message_has_no_key`) and Task 2 (`test_passage_quoting_descriptor_without_code_withheld`). ## Decisions -Each item below is a planner's call. Nobody was available to ask; each is made here and the tasks implement it. +Each item is a planner's call unless it carries a ruling id. Nobody was available to ask, so each is made here and implemented by the tasks. **Layout** -- One `market` package, one `stack market` CLI group (`src/cli/market.py`), and DuckDB schemas `market` and `econ`, with all DDL in `src/market/tables.py` (planner's call). +- One `market` package and one `stack market` CLI group (`src/cli/market.py`), with DuckDB schemas `market` and `econ` (planner's call). - The Congress and district layer lives in the same package (planner's call). -- No new dependencies and no new extra. numpy, httpx and PyYAML are already locked (planner's call). +- No new dependencies and no new extra (planner's call). + +**Entity keys and links** +- **Entity keys:** a company's key is its 10-digit CIK, so seed and EDGAR-discovered companies share one key space. An organization's key is its seed key, such as `AAFP` (planner's call). +- `market.entity` stores each entity's `aliases`, normalized and `|`-joined (M5). +- **Links (M3):** `write_links` upserts by `(basis, entity_key, scope, target)`; for companies this is the ruling's (basis, cik, target). One basis never wipes another. Curated links removed from the seed are retired by `retire_links` (planner's call). +- **Target vocabularies (M6):** + - scope `family` takes a `pfs.families` key; + - scope `specialty` takes a CMS specialty key (`pfs.commenters.specialty_key` of a `specialty_vocab` entry, the values of `pfs.seats.RUC_SEAT_SPECIALTY`), or one of the seat-label keys `PRIMARY CARE` and `ONCOLOGY` (M8); + - scope `program` takes one of `PROGRAMS = ("PFS", "QPP-MIPS", "MSSP", "APM", "TELEHEALTH", "HOME-BASED-CARE", "CARE-MANAGEMENT")`; + - scope `sector` takes a SIC code. +- **Name matching, the one rule (M5):** + - LDA names, and the FEC fallback, match by exact normalized name (`alias_match(prefix=False)`). + - CMS owner and NPPES names (Task 9) may also prefix-match, with an alias of at least 5 characters, to catch subsidiaries. + - FEC committees come first from curated `fec_committee_ids` in the seed. The fallback accepts a committee only when its normalized name equals an alias, and cites the committee's fec.gov page. **Keys and API access** -- Keys go only in headers. api.data.gov keys use `X-Api-Key` (planner's call). -- `API_DATA_GOV_KEY` is shared, with `FEC_API_KEY` and `CONGRESS_API_KEY` as overrides, per the coordinator's instruction. -- `LDA_API_KEY` is optional because anonymous lda.gov works; `HUD_API_TOKEN` is gap-gated (planner's call). -- Pacers, set below each provider's limit (planner's call): SEC 0.15 s, Tiingo 75 s, LDA 4.2 s (0.55 s keyed), FEC 3.7 s, Congress.gov 0.8 s, GitHub 61 s, FR 0.5 s. -- The Tiingo budget is capped at 450 symbols a month and 950 requests a day (planner's call). +- Keys go only in headers; api.data.gov keys use `X-Api-Key` (planner's call). +- Pacers sit below each provider's limit (planner's call): SEC 0.15 s, Tiingo 75 s, LDA 4.2 s (0.55 s keyed), FEC 3.7 s, Congress.gov 0.8 s, GitHub 61 s, FR 0.5 s. -**Benchmarks and money data** -- The sector index is **XLV**, which starts 1998-12-22. IHF (from 2006-05-05) is a providers sensitivity index. SPY (from 1993-01-29) is the market index (planner's call). -- Returns come from the `adjClose` ratio within a pull, with the raw total return kept as a cross-check. #797's `market.series` is a monthly total-return index (planner's call). -- LDA and FEC start in 2008, post-HLOGA (planner's call). -- FEC is the openFEC API only, per the user's decision. The keyless FEC bulk files are noted, not used. +**Prices and benchmarks** +- **Price universe (M4):** + - The universe is the benchmarks, then the curated seed securities, then the securities of filing-linked entities (basis `fts`), ranked and capped at 450 symbols a month. + - Symbols already fetched this month are free. + - Entities with only `sic` links are never priced; SIC-sector breadth comes from the ETFs. +- **Benchmarks:** XLV is the sector index (from 1998-12-22) and SPY the market index (from 1993-01-29). IHF (from 2006-05-05) is a providers sensitivity index (planner's call). +- Returns are the `adjClose` ratio within each pull, with the raw total return kept as a cross-check (planner's call). +- #797's `market.series` is a monthly total-return index (planner's call). -**Entity map** -- Organizations are matched by exact normalized name for LDA and FEC, and by prefix for subsidiaries in the CMS footprint, which needs an alias of at least 5 characters (planner's call). -- Seed entries must be cited or loading fails. HarmonyCares, WNOT and Wakely are refused (planner's call). +**Filings and passages** - An fts link needs a health SIC (the 19 `HEALTH_SICS`) or seed membership. -- Passages are at most 400 characters and at most 3 per term, with an AMA 8-word guard. -- The optional local classifier uses the labels revenue_exposure, risk_factor and incidental (planner's call). -- SIC alone only links to the `sector` scope; family and specialty links need a passage or a curated cite (planner's call). +- A passage is at most 400 characters, with at most 3 passages per term. +- The optional classifier labels passages `revenue_exposure`, `risk_factor` or `incidental` (planner's call). +- SIC alone only ever links to scope `sector`. +- **Revenue (D1):** XBRL companyfacts annual revenue is keyed on the reported period's end date (and `frame` when present), never on the filing's `fy`. -**Event study** -- The event date is the FR public-inspection `filed_at`. At or after 16:00 ET, day 0 is the next trading day; otherwise it is the publication date, with a caveat (planner's call). -- RUC and Panel meetings come from FR paragraph mentions (month precision, cited) and, where readable, AMA schedule pages (day precision). Month-precision events never enter daily windows (planner's call). -- The estimation window is (−250, −31) with a minimum of 120 returns. Event windows are (0,0), (−1,1), (0,1), (−5,5) and (0,10). The default model is `market` (SPY), because a sector index absorbs sector-wide rule effects; `sector` and `two-factor` are sensitivities (planner's call). -- Portfolios are equal-weighted and rebalanced daily, with membership as of day 0, and the model is run on the portfolio series to handle cross-correlation. Pooled means use the shared `pfs.event_study` bootstrap (planner's call). -- The tenure comparison uses annual BHAR against SPY, comparing seat years with non-seat years, as association only. The market model stays in `market/abnormal.py`; `pfs/event_study.py` only gains the shared bootstrap helpers (planner's call). +**Event windows** +- **Rule events:** + - The event date is the FR public-inspection `filed_at`; at or after 16:00 ET it moves to the next trading day. Without `filed_at`, the publication date is used, with a caveat. + - Program events target `PFS`. Every program key uses these events, because QPP, MSSP and APM change through the annual PFS rule. + - Family events are the rules anchoring that family's `pfs.code_exposure` rows (the spec's exposure calendar) or its `pfs.code_event` FR rows. + - Specialty events are the rules whose CMS impact table gives the specialty a combined change of at least 1% in either direction (M6). + - `rule_items` entries of kind `rule` (undecidable) are skipped and counted. +- **Meetings:** RUC and Panel meetings come from FR paragraph mentions (month precision, cited) and, where readable, AMA schedule pages (day precision). Month-precision events never enter daily windows (planner's call). +- **Seats (M8):** + - Tenures key on `seat_key`: the CMS specialty, else the label key for primary-care and oncology seats, else none (category rows, AMA, AOA). + - Per-year presence comes from `pfs.seats.seats_by_specialty` (RUC permanent plus rotating > 0), so AAFP (2019–23) and "Family Medicine" (2024–26) make one FAMILY PRACTICE tenure. + - A year is "on the RUC" when that calendar year's roster holds the seat. Plan A has not decided between the same-year and year−1 roster (planner's call). +- **Model:** + - The estimation window is (−250, −31) trading days, with at least 120 returns. + - Event windows are (0,0), (−1,1), (0,1), (−5,5) and (0,10). + - The default model is `market` (SPY), because a sector index absorbs sector-wide rule effects. `sector` and `two-factor` are sensitivities. + - Portfolios are equal-weighted, rebalanced daily, with members as of day 0, and the model runs on the portfolio series. + - Pooled means use the shared `pfs.event_study` bootstrap. + - The tenure contrast is the annual BHAR against SPY, seat years against non-seat years. It is an association only (planner's call). +- **LDA supersede rule, stated once (D4):** within each (registrant, client, filing_year, filing_period), every report-type filing except the latest `dt_posted` is superseded, and registrations carry no amount. `market.lda.supersede` is the only implementation. **Congress and districts** -- Committee assignments come from the `unitedstates/congress-legislators` snapshots in its git history, one per year from 2013. Members come from Congress.gov when keyed and from that repository's YAML when not, which also gives the FEC crosswalk (planner's call). -- Bills cover the 110th–119th Congresses. They come from the Congress.gov bill list with a local `BILL_RE` title filter plus `policyArea == "Health"` (planner's call). -- Districts are CD118 and CD119 from the 2020 Census relationship files, weighted by land area. HUD business and residential ratios are preferred when keyed (planner's call). -- The footprint is a cross-section: data year 2024, the CMS All Owners snapshot of 2026-09-29, the Tuva 2025-09 NPPES cut, and CD119. ZIPs with no district are kept as `??` (planner's call). +- **District id (M7):** + - Every district is `"-"` per congress: `"MI-12"`, at-large `"AK-00"`, a non-voting delegate `"DC-98"`. + - Senators have `""`, and an unmatched ZIP is `"??"`. + - FIPS, postal code and state name convert through `reference_data.ansi_fips_state`. +- **Committees (M7):** one congress-legislators snapshot per congress, the last commit on or before April 1 of the congress's first year, for the 113th Congress on. Congress.gov has no membership data (planner's call). +- **Bills:** the 110th–119th Congresses, from the Congress.gov bill list with a local `BILL_RE` title filter and `policyArea == "Health"` (planner's call). +- **Districts:** CD118 and CD119 from the 2020 Census relationship files, weighted by land area. HUD business and residential ratios come from a separate key-gated command and are preferred when present (M11; planner's call). +- **Footprint:** a cross-section of data year 2024, the CMS All Owners snapshot of 2026-09-29, the Tuva 2025-09 NPPES cut, and CD119 (planner's call). **Enrichment** -- FRED comes from the keyless fredgraph CSV, using 8 series. SOI comes from the ZIP-level files only (district AGI); the county file is skipped (planner's call). -- Company revenue comes from XBRL companyfacts (`Revenues`). Segment revenue is a stated gap (planner's call). -- The enrichment readers are annual, plus monthly for CES, FRED and market series. The specialty→SOC and family→CES maps are as listed in Task 14 (planner's call). - -**People** -- No FEC individual donors and no lobbyist names are stored (planner's call). +- FRED comes from the keyless fredgraph CSV, using 8 series. SOI comes from the ZIP-level files only (planner's call). +- PSPS columns in the joint readers need a family's codes, matched on `specialty_key` (M9). --- @@ -108,165 +220,182 @@ Each item below is a planner's call. Nobody was available to ask; each is made h | File | Responsibility | |---|---| -| `src/market/http.py` | UA, keys, pacer, `get` with backoff/refusal, hoard paths | -| `src/market/tables.py` | DDL (`market`, `econ`), dataclasses, writers, readers, source gaps | -| `src/market/seed.yaml`, `src/market/seed.py` | Curated companies and organizations, each cited; name normalization and alias matching | -| `src/market/edgar.py` | company_tickers, submissions, SIC sweep, filings index; full-text search and passages (Task 2) | -| `src/market/tiingo.py` | EOD prices, metadata, symbol budget, monthly total-return series (#797) | -| `src/market/lda.py`, `src/market/fec.py` | Lobbying filings; PAC committees, totals, contributions | +| `src/market/http.py` | UA, keys, pacer, `get` with backoff and refusal, hoard paths | +| `src/market/tables.py` | DDL (`market`, `econ`), row dataclasses, writers and readers, source gaps, district ids | +| `src/market/seed.yaml`, `src/market/seed.py` | Curated companies and organizations, each cited; names, aliases, target vocabularies, `seat_key` | +| `src/market/edgar.py` | Tickers, submissions with their extra pages, SIC sweep, filings; full-text search and passages; companyfacts revenue | +| `src/market/tiingo.py` | Price universe, EOD prices, metadata, symbol budget, monthly total-return series | +| `src/market/lda.py`, `src/market/fec.py` | Lobbying filings; PAC committees, totals and Schedule B | | `src/market/congress.py` | Members, committee assignments, bills | | `src/market/districts.py`, `src/market/footprint.py` | ZIP/county → district weights; company and family-biller footprints; district view | -| `src/market/events.py` | Rule, RUC and Panel event windows; seat tenures | +| `src/market/events.py` | Rule, specialty, RUC and Panel events; seat tenures | | `src/market/abnormal.py`, `src/market/portfolio.py` | Market model and CAR; portfolios, pooled results, tenure contrast | -| `src/market/enrich.py` | Joint readers: market beside BLS, FRED, SOI, PSPS, utilization | -| `src/econ/fred.py`, `src/econ/irs_soi.py` (mod) | fredgraph CSV and SOI ZIP-file parsers | -| `src/pfs/event_study.py` (mod) | Shared `bootstrap_draws` / `percentile_interval` | -| `src/cli/market.py`, `src/cli/__init__.py` (mod) | `stack market …` | +| `src/market/enrich.py` | Joint readers | +| `src/econ/fred.py`, `src/econ/irs_soi.py` (mod) | fredgraph CSV and SOI ZIP parsers | +| `src/pfs/event_study.py` (mod, Task 11 only) | Shared `bootstrap_draws` / `percentile_interval` | +| `src/cli/market.py`, `src/cli/__init__.py` (one edit, Task 1) | `stack market …` | | `notebooks/event_study.py` (mod) | #797 market panel | -**Order (planner's call):** -- Task 1 comes first. -- Acquisition, Tasks 2–9 and 13, can then run in any order, except that Task 9 needs 6 and 8. -- Task 11, the engine, depends only on Task 1 and may run in parallel with acquisition. -- Task 10 needs 1. Task 12 needs 3, 10 and 11, and uses Task 2's links when present. Task 14 needs 3, 9, 12 and 13. Task 15 comes last. -- GPU is used only in Task 2 with `--classify`. +**Order (H1, W22):** the tasks run one at a time, in numeric order. Tasks 2–10 and 13 all append to `market/tables.py`, `cli/market.py`, `test_tables.py` and `test_market_cli.py`. Task 11 depends only on Task 1 and could run earlier, but it still runs alone. GPU is used only in Task 2 with `--classify`. --- -### Task 1: Market foundation, curated seed, EDGAR entity registry — `stack market entities` +### Task 1: Market foundation, curated seed, EDGAR entity registry — `stack market entities`, `stack market gaps` **Files:** -- Create: `src/market/__init__.py`, `src/market/http.py`, `src/market/tables.py`, `src/market/seed.yaml`, `src/market/seed.py`, `src/market/edgar.py`, `src/cli/market.py`, `tests/market/__init__.py`, `tests/market/fixtures/` (`company_tickers.json` with 3 entries, `submissions_CIK0001564406.json` trimmed to 3 filings, `sic_8082.atom` with 2 entries) -- Modify: `src/cli/__init__.py` (`app.add_typer(market_app, name="market", help="Market and enrichment tier: exposed companies, prices, lobbying, Congress.")`), `tests/cli/test_cli_import_without_optional_deps.py` -- Test: `tests/market/test_http.py`, `tests/market/test_seed.py`, `tests/market/test_edgar.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py` +- Create: + - `src/market/__init__.py`, `src/market/http.py`, `src/market/tables.py`, `src/market/seed.yaml`, `src/market/seed.py`, `src/market/edgar.py`, `src/cli/market.py`; + - `tests/market/__init__.py`, `tests/market/test_imports.py`; + - `tests/market/fixtures/company_tickers.json` (3 entries); + - `tests/market/fixtures/submissions_CIK0001564406.json` (trimmed to 3 recent filings, with `filings.files` naming one page); + - `tests/market/fixtures/CIK0001564406-submissions-001.json` (2 filings); + - `tests/market/fixtures/sic_8082.atom` (2 entries). +- Modify: `src/cli/__init__.py`, one complete edit: `from cli.market import app as market_app` and `app.add_typer(market_app, name="market", help="Market and enrichment tier: exposed companies, prices, lobbying, Congress.")`. +- Test: `tests/market/test_http.py`, `tests/market/test_seed.py`, `tests/market/test_edgar.py`, `tests/market/test_tables.py`, `tests/market/test_imports.py`, `tests/cli/test_market_cli.py`. **Interfaces:** -- Consumes: `pfs.seats.save_snapshot`, `pfs.seats.html_text`, `pfs.psps.newest_cache`, `cms.ingest_log.log_ingest`, `bib.item.Source`/`Download`, `bib.tag.Tag`, `pfs.commenters.specialty_vocab`. +- Consumes (read-only): `pfs.seats.save_snapshot`, `RUC_SEAT_SPECIALTY`, `_key` (seat-label normalization); `pfs.psps.newest_cache`; `pfs.codetables.SeatRow`/`read_seats`; `pfs.commenters.specialty_vocab`/`specialty_key`; `pfs.families.FAMILIES`/`refresh_from`; `cms.ingest_log.log_ingest`; `bib.item.Source`; `bib.tag.Tag`. - Produces: - - `market.http`: + - **`market.http`** - `USER_AGENT = "stack-pfs-dossier/1.0 (ci@mail.fhirworx.io)"`. - `KEY_ENV = {"tiingo": ("TIINGO_API_KEY",), "fec": ("FEC_API_KEY", "API_DATA_GOV_KEY"), "congress": ("CONGRESS_API_KEY", "API_DATA_GOV_KEY"), "lda": ("LDA_API_KEY",), "hud": ("HUD_API_TOKEN",)}`. - - `api_key(source: str, env: Mapping[str, str] | None = None) -> str | None`: the first non-empty variable. - - `key_headers(source: str, key: str | None) -> dict[str, str]`: tiingo/lda `Authorization: Token k`; fec/congress `X-Api-Key: k`; hud `Authorization: Bearer k`; `{}` for None. + - `api_key(source: str, env: Mapping[str, str] | None = None) -> str | None`. + - `key_headers(source: str, key: str | None) -> dict[str, str]`: tiingo/lda `Authorization: Token k`; fec/congress `X-Api-Key: k`; hud `Authorization: Bearer k`. - `client(source: str, key: str | None = None) -> httpx.Client`: UA, key headers, timeout 120, follow_redirects. - - `class Pacer(min_interval: float, *, clock=time.monotonic, sleep=time.sleep)` with `.wait() -> None`. - - `class Refused(Exception)` with `source: str`, `status: int`, `reason: str`; its `str()` never contains a query string. - - `get(cl, url, *, params=None, pacer: Pacer, source: str, sleep=time.sleep, retries=3, max_retry_after=600.0, ok_404=False) -> httpx.Response`: - - 5xx retried with a `2·2**n` s backoff; + - `class Pacer(min_interval: float, *, clock=time.monotonic, sleep=time.sleep)` with `.wait()`. + - `class Refused(Exception)` with `source`, `status`, `reason`; its `str()` never contains a query string. + - `get(cl, url, *, params=None, pacer, source, sleep=time.sleep, retries=3, max_retry_after=600.0, ok_404=False) -> httpx.Response`: + - 5xx is retried with a `2·2**n` s backoff; - 429 with `Retry-After ≤ max_retry_after` waits once and retries, otherwise `Refused`; - 401/403 → `Refused`; - 404 is returned when `ok_404`, else `Refused`. - - `hoard_root() -> Path` (`conf.path("db.aco").parent / "market"`); `hoard(*parts: str) -> Path`. - - `save = pfs.seats.save_snapshot`; `newest = pfs.psps.newest_cache`. - - `slug(text: str) -> str`: lower-case alnum and `-`, at most 60 chars, plus `-` when truncated. - - `market.seed`: - - `@dataclass(frozen=True) Cite(url: str = "", title: str = "", accessed: str = "", adsh: str = "", filename: str = "", quote: str = "", item_key: str = "")`. Exactly one of `url` / `adsh+filename` / `item_key` is set; `quote` ≤ 400 chars. - - `SeedSecurity(ticker: str, exchange: str, valid_from: date | None, valid_to: date | None)`. - - `SeedLink(scope: str, target: str, cite: Cite)` with scope ∈ `("family", "specialty", "program")`. - - `SeedCompany(key, name, cik, status, ended_on: date | None, successor: str, aliases: tuple[str, ...], securities: tuple[SeedSecurity, ...], links: tuple[SeedLink, ...], cite: Cite)` with status ∈ `("listed", "acquired", "delisted", "bankrupt")`. - - `SeedOrg(key, name, org_type, specialty, ruc_seat, aliases, fec: tuple[str, ...], cite: Cite)` with org_type ∈ `("specialty_society", "trade_group", "professional_assoc")`. - - `load_seed(path: Path | None = None) -> tuple[tuple[SeedCompany, ...], tuple[SeedOrg, ...]]` raises `ValueError(f"{key}: no citation")` for any uncited entry or link. - - `EXCLUDED = ("HARMONYCARES", "WAKELY", "WNOT")`: a seed name or alias containing one raises. - - `norm_org(name: str) -> str`: upper-case, `&`→`AND`, punctuation dropped, spaces collapsed, leading `THE ` and trailing `INC|INCORPORATED|CORP|CORPORATION|CO|LLC|L L C|LP|LTD|PLC` dropped. - - `alias_match(name: str, aliases: Sequence[str], *, prefix: bool) -> bool`: `norm_org` equality; with `prefix`, also `norm(name).startswith(alias + " ")` for aliases ≥ 5 chars. - - Tables: - - `EntityRow(key, name, kind, cik, sic, sic_desc, status, ended_on: date | None, successor, specialty, org_type, source_url, item_key)` → `market.entity`; - - `SecurityRow(entity_key, ticker, exchange, valid_from: date | None, valid_to: date | None, source)` → `market.security`; - - `LinkRow(entity_key, scope, target, basis, strength, cite_kind, cite_ref, passage_id, item_key)` → `market.entity_link`, with: - - scope ∈ `("family", "specialty", "program", "sector")`; - - basis ∈ `("curated", "sic", "fts")`; - - strength ∈ `("curated", "sector", "revenue_exposure", "risk_factor", "mention")`; - - `FilingRow(cik, adsh, form, filed: date, accepted: str, report_date: str, items: str, primary_doc: str)` → `market.filing`; - - `GapRow(source, note, recorded_on: date)` → `market.source_gap`. - - Functions: - - `ensure_tables(con)`; - - `write_entities(con, rows)`, which deletes by key; - - `write_securities(con, entity_key, rows)`; - - `write_links(con, basis, rows)`, which replaces that basis; - - `write_filings(con, cik, rows)`; - - `write_gap(con, source, note)` and `clear_gap(con, source)`; - - readers `read_entities(con, kind=None)`, `read_securities(con, entity_key=None)`, `read_links(con, *, scope=None, target=None, basis=None)`, `read_filings(con, cik, forms=None)`, `read_gaps(con)`. - - `market.edgar`: - - URLs: `TICKERS = "https://www.sec.gov/files/company_tickers.json"`, `SUBMISSIONS = "https://data.sec.gov/submissions/CIK{cik10}.json"`, `SIC_ATOM = "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&SIC={sic}&type=10-K&owner=include&count=100&start={start}&output=atom"`. - - Pacer 0.15 s (SEC limit 10 req/s). + - This is the market tier's one retry loop. The `pfs.seats` and `pfs.psps` loops stay as they are (H1). + - `hoard_root() -> Path` (`conf.path("db.aco").parent / "market"`); `hoard(*parts) -> Path`; `save = pfs.seats.save_snapshot`; `newest = pfs.psps.newest_cache`; `slug(text) -> str`. + - **`market.seed`** + - `Cite(url="", title="", accessed="", adsh="", filename="", quote="", item_key="")`: exactly one locator is set; `quote` ≤ 400 chars. + - `SeedSecurity(ticker, exchange, valid_from: date | None, valid_to: date | None)`. + - `SeedLink(scope, target, cite)`. + - `SeedCompany(handle, name, cik, status, ended_on: date | None, successor_cik, aliases, securities, links, fec_committee_ids: tuple[str, ...], cite)`, with status ∈ `("listed", "acquired", "delisted", "bankrupt")`. + - `SeedOrg(key, name, org_type, specialty, aliases, fec_committee_ids, links, cite)`, with org_type ∈ `("specialty_society", "trade_group", "professional_assoc")` and `specialty` a specialty-scope target or `""`. + - `load_seed(path=None) -> tuple[tuple[SeedCompany, ...], tuple[SeedOrg, ...]]` is pure. It raises `ValueError(f"{key}: no citation")`, or raises on any `EXCLUDED = ("HARMONYCARES", "WAKELY", "WNOT")` name. + - `PROGRAMS = ("PFS", "QPP-MIPS", "MSSP", "APM", "TELEHEALTH", "HOME-BASED-CARE", "CARE-MANAGEMENT")`; `SEAT_LABEL_KEYS = ("PRIMARY CARE", "ONCOLOGY")`. + - `validate_seed(companies, orgs, *, families: set[str], specialties: set[str]) -> list[str]` returns errors, one per invalid `(scope, target)`. The CLI calls it with `specialty_vocab(con)` keyed by `specialty_key`, plus `SEAT_LABEL_KEYS`. + - `seat_key(row: SeatRow) -> str | None`: + - `None` for `holder_kind == "category"`; + - else `row.specialty` when set; + - else `pfs.seats._key(row.organization).upper()` when that `_key` maps to `""` in `RUC_SEAT_SPECIALTY` (primary care, oncology); + - else `None` (AMA, AOA). + - `seat_specialties(seats: Sequence[SeatRow], *, years: Iterable[int]) -> set[str]`: the `seat_key`s of RUC rows with `holder_kind == "specialty"` in those years. + - `norm_org(name) -> str`: upper-case, `&`→`AND`, punctuation dropped, spaces collapsed, leading `THE ` dropped, and a trailing `INC|INCORPORATED|CORP|CORPORATION|CO|LLC|L L C|LP|LTD|PLC` dropped. + - `alias_match(name, aliases, *, prefix: bool) -> bool`: true on `norm_org` equality; with `prefix`, also when `norm_org(name).startswith(alias + " ")` for an alias of at least 5 characters. + - **Tables** (district helpers live here too): + - Rows: + - `EntityRow(key, name, kind, cik, sic, sic_desc, status, ended_on, successor, specialty, org_type, aliases: str, source_url, item_key)` → `market.entity`; + - `SecurityRow(entity_key, ticker, exchange, valid_from, valid_to, source)` → `market.security`; + - `LinkRow(entity_key, scope, target, basis, strength, cite_kind, cite_ref, passage_id, item_key)` → `market.entity_link`; + - `FilingRow(cik, adsh, form, filed: date, accepted, report_date, items, primary_doc)` → `market.filing`; + - `GapRow(source, note, recorded_on: date)` → `market.source_gap`. + - Writers: `write_entities(con, rows)` (delete by key); `write_securities(con, entity_key, rows)`; `write_links(con, rows)` (upsert by `(basis, entity_key, scope, target)`); `retire_links(con, basis, entity_keys, keep: set[tuple[str, str, str]]) -> int`; `write_filings(con, cik, rows)`; `write_gap(con, source, note)`; `clear_gap(con, source)`. + - Readers: `read_entities(con, kind=None)`, `read_securities`, `read_links(con, *, scope=None, target=None, basis=None)`, `read_filings(con, cik, forms=None)`, `read_gaps`. + - District helpers: `state_codes(con) -> dict[str, tuple[str, str, str]]` (fips, postal or name → `(fips, postal, name)`, from `reference_data.ansi_fips_state`); `district_id(postal: str, number: int | str | None) -> str` (`None`, `0`, `"00"` and `"At Large"` all give `-00`). + - **`market.edgar`** + - URLs: + - `TICKERS = "https://www.sec.gov/files/company_tickers.json"`; + - `SUBMISSIONS = "https://data.sec.gov/submissions/{name}"`, where `name` is `CIK##########.json` or a `filings.files[].name`; + - `SIC_ATOM = "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&SIC={sic}&type=10-K&owner=include&count=100&start={start}&output=atom"`. + - Pacer 0.15 s (the SEC limit is 10 requests a second). - `HEALTH_SICS = ("2834", "2835", "2836", "3841", "3842", "3845", "5047", "5122", "6324", "8000", "8011", "8050", "8051", "8060", "8062", "8071", "8082", "8090", "8093")`. - - Functions: - - `cik10(cik: str | int) -> str`; - - `parse_tickers(payload) -> dict[str, list[str]]`, mapping cik10 to tickers; - - `parse_sic_atom(xml: bytes) -> list[str]` (cik10s); - - `entity_from_submissions(sub: dict, seed: SeedCompany | None) -> EntityRow` (the seed fixes status, ended_on and successor); - - `securities_for(sub, seed, tickers) -> list[SecurityRow]` (seed windows win; current tickers from submissions get `valid_from=None`); - - `filings_from_submissions(sub: dict) -> list[FilingRow]` (recent plus each `filings.files` page); - - `sic_links(entity: EntityRow) -> list[LinkRow]` (scope `sector`, target the SIC, basis `sic`, strength `sector`, only for `HEALTH_SICS`); - - `curated_links(seed) -> list[LinkRow]` (basis/strength `curated`); - - `cite_registry(store) -> str` (bib Source, tags `module:market`, `table:market.entity`, `source:sec-edgar`). - - CLI: - - `stack market entities [--sic-sweep] [--refresh] [--offline] [--write]`. It loads the seed and the tickers, then submissions for the seed CIKs (plus every `HEALTH_SICS` CIK with `--sic-sweep`). Caches live at `data/market/sec/{company_tickers.json, submissions/CIK…json, sic/..atom}`. It writes entities, securities, filings, and the curated and sic links. Organizations from the seed become `EntityRow(kind="organization")`. - - `stack market gaps` prints `market.source_gap`. - - Module indirections `_batch`, `_read`, `_store`, `_publish`, `_asker` (as in `cli/pfs.py`) and `_gap(source, note, *, write)`, which echoes `gap: {source}: {note}` and, with write, calls `write_gap` in a batch. - - Seed content (planner's call; the implementer resolves each CIK with `https://efts.sec.gov/LATEST/search-index?keysTyped=` and cites every link): - - Companies, each with ≥ 1 cited link: - - home-based and value-based primary care: OSH (CIK 1564406, 2020-08-06..2023-05-02, acquired), SGFY (1828182, 2021-02-11..2023-04-03, acquired), AMED (896262, 1994-08-18..2025-08-14, acquired), LHCG (2005-06-09..2023-02-22), ONEM (2020-01-31..2023-02-22), AGL (1831097, from 2021-04-15), PRVA (1759655, from 2021-04-28), CANO, ALHC (1832466), CLOV, EVH (1628908), ADUS (1468328), EHAB (2022-07-01..2026-05-14), CHE (19584); - - physician and outpatient services: MD (893949), RDNT (790526), USPH (885978), LFST; - - dialysis: DVA (927066); - - payers: UNH (731766), HUM, CVS, CI, ELV; - - labs: LH, DGX; - - virtual care: TDOC, AMWL; - - hospitals: HCA (860730, from 2011-03-10), THC, UHS, CYH; - - post-acute: ENSG (1125376). - - Organizations: - - one specialty society per 2024 `pfs.governance_seat` RUC label, with `ruc_seat` set to that label (e.g. `Family Medicine` → American Academy of Family Physicians, FEC `C00411553`); - - trade groups AMA, AOA, AHA, AHIP, MGMA, AMGA, NAACOS, America's Physician Groups, the home-care association, AAHCM and ATA; - - cite: a public page naming the organization's role. + - `cik10(cik) -> str`; `parse_tickers(payload) -> dict[str, list[str]]`, mapping cik10 to tickers; `parse_sic_atom(xml: bytes) -> list[str]`, giving cik10s. + - `submission_pages(sub) -> list[str]` (the `filings.files` names); `filings_from_submissions(sub, pages: Sequence[dict] = ()) -> list[FilingRow]` (recent plus each page). + - `entity_from_submissions(sub, seed: SeedCompany | None) -> EntityRow` (key = cik10; the seed's aliases, status and ended_on win); `securities_for(sub, seed, tickers)`; `sic_links(entity)` (scope `sector`, only `HEALTH_SICS`); `curated_links(companies, orgs) -> list[LinkRow]`; `cite_registry(store)`. + - The cache is `data/market/sec/submissions/`. + - **`cli.market`** + - Indirections `_batch`, `_read`, `_store`, `_publish`. + - `_require_key(provider: str, gap_source: str, *, write: bool) -> str | None` prints `gap: {gap_source}: {ENV names} not in .env` and, with `write`, records the gap in a batch. + - `_gap(source, note, *, write)`. + - `stack market entities [--sic-sweep] [--refresh] [--offline] [--write]`: + - loads and validates the seed (errors abort); + - reads tickers, submissions and their pages for the seed CIKs (plus every `HEALTH_SICS` CIK with `--sic-sweep`); + - writes entities, securities and filings, upserts the curated and sic links, retires stale curated links, and clears `sec-edgar:entities`. + - Organizations become `EntityRow(kind="organization", key=)`. + - `stack market gaps`. + - **Seed content:** + - Companies (planner's call), each with ≥ 1 cited link. The implementer resolves any missing CIK with `https://efts.sec.gov/LATEST/search-index?keysTyped=`. + - Home-based and value-based primary care: + - OSH (CIK 1564406, 2020-08-06..2023-05-02, acquired), SGFY (1828182, 2021-02-11..2023-04-03, acquired), AMED (896262, 1994-08-18..2025-08-14, acquired); + - LHCG (2005-06-09..2023-02-22), ONEM (2020-01-31..2023-02-22); + - AGL (1831097, from 2021-04-15), PRVA (1759655, from 2021-04-28), CANO, ALHC (1832466), CLOV, EVH (1628908); + - ADUS (1468328), EHAB (2022-07-01..2026-05-14), CHE (19584). + - Physician and outpatient services: MD (893949), RDNT (790526), USPH (885978), LFST. + - Dialysis: DVA (927066). + - Payers: UNH (731766), HUM, CVS, CI, ELV. + - Labs: LH, DGX. + - Virtual care: TDOC, AMWL. + - Hospitals: HCA (860730, from 2011-03-10), THC, UHS, CYH. + - Post-acute: ENSG (1125376). + - The windows come from Tiingo metadata and EDGAR 8-K items 2.01/3.01, verified 2026-11-12 for OSH, SGFY and AMED. + - Organizations: one specialty society per element of `seat_specialties(read_seats(con, "RUC"), years=range(2024, 2027))`, its `specialty` set to that key. Category rows, AMA and AOA are filtered out by `seat_key` (M8). Societies named on the 2019–23 organization rows are used where they exist; for example, AAFP has FEC `C00411553` in `fec_committee_ids`. + - Trade groups: AMA, AOA, AHA, AHIP, MGMA, AMGA, NAACOS, America's Physician Groups, the home-care association, AAHCM and ATA. - [ ] **Step 1: Write the failing tests** - `test_api_key_falls_back_to_api_data_gov`: - env `{"API_DATA_GOV_KEY": "k1"}` → `api_key("fec") == api_key("congress") == "k1"`; - - with `FEC_API_KEY="k2"` → `"k2"`; + - `FEC_API_KEY="k2"` → `"k2"`; - `api_key("tiingo", {}) is None`. - `test_key_sent_in_header_never_in_url`: MockTransport records the request; `headers["X-Api-Key"] == "k1"` and `"k1" not in str(request.url)`. - - `test_user_agent_has_contact`: `USER_AGENT == "stack-pfs-dossier/1.0 (ci@mail.fhirworx.io)"` and `client("fec","k").headers["user-agent"] == USER_AGENT`. - - `test_pacer_spaces_requests`: fake clock at 0.0 then 1.0, `Pacer(4.2)` → the second `wait()` sleeps `pytest.approx(3.2)`. - - `test_get_retries_5xx_then_ok`: 503, 200 → sleeps `[2.0]`. - - `test_get_429_retry_after_waits_once`: 429 `Retry-After: 30` then 200 → `sleep(30.0)`. - - `test_get_429_twice_raises_refused`. - - `test_refused_message_has_no_query_string`: 403 on `…/x?api_key=secret` → `Refused.status == 403` and `"secret" not in str(exc)` and `"?" not in str(exc)`. - - `test_get_404_ok_when_allowed`. - - `test_slug`: `slug("Medicare Physician Fee Schedule") == "medicare-physician-fee-schedule"`. - - `test_seed_loads_and_every_entry_cited`. - - `test_seed_entry_without_cite_rejected`: a tmp YAML without a cite → `ValueError` naming the key. - - `test_seed_has_no_excluded_names`. - - `test_security_windows_parse`: OSH `valid_from == date(2020,8,6)` and `valid_to == date(2023,5,2)`. - - `test_norm_org`: - - `"Amedisys Holding, L.L.C."` → `"AMEDISYS HOLDING"`; - - `"The American College of Physicians, Inc."` → `"AMERICAN COLLEGE OF PHYSICIANS"`. + - `test_user_agent_has_contact`: `USER_AGENT == "stack-pfs-dossier/1.0 (ci@mail.fhirworx.io)"` and `client("fec", "k").headers["user-agent"] == USER_AGENT`. + - `test_pacer_spaces_requests`: clock 0.0 and 1.0, `Pacer(4.2)` → sleeps `approx(3.2)`. + - `test_get_retries_5xx_then_ok`: sleeps `[2.0]`. + - `test_get_429_retry_after_waits_once`: `sleep(30.0)`. + - `test_get_429_twice_raises_refused`: 429, 429 → `Refused.status == 429`. + - `test_refused_message_has_no_query_string`: 403 on `…/x?api_key=secret` → `Refused.status == 403`, `"secret" not in str(exc)`, `"?" not in str(exc)`. + - `test_get_404_ok_when_allowed`: `ok_404=True` → the 404 response is returned; `ok_404=False` → `Refused`. + - `test_slug`: `slug("Medicare Physician Fee Schedule") == "medicare-physician-fee-schedule"`; a 70-char input → 60 chars plus `-`. + - `test_seed_loads_and_every_entry_cited`: the shipped `seed.yaml` loads; every company, org and link has exactly one locator. + - `test_seed_entry_without_cite_rejected`: a tmp YAML company without `cite` → `ValueError` whose message starts with its handle. + - `test_seed_has_no_excluded_names`: a tmp YAML alias `HARMONYCARES LLC` → `ValueError`; the shipped seed has no `EXCLUDED` word. + - `test_security_windows_parse`: OSH 2020-08-06..2023-05-02. + - `test_validate_seed_targets`: family `NOPE`, specialty `FAMILY MEDICINE` (a label, not a key) and program `ACO` give 3 errors; `FAMILY PRACTICE`, `PRIMARY CARE`, `PFS` and `HOME-VISITS` give 0. + - `test_seat_key_and_seat_specialties`: synthetic SeatRows: + - `("RUC", 2024, "Family Medicine", "FAMILY PRACTICE", "permanent", …, "specialty")` → `"FAMILY PRACTICE"`; + - `("RUC", 2022, "Primary Care Rotating Seat", "", "rotating", …, "organization")` → `"PRIMARY CARE"`; + - `"RUC Chair"` (category) → None; + - `"American Medical Association"` → None; + - `seat_specialties(..., years=[2024])` → `{"FAMILY PRACTICE"}`. + - `test_norm_org`: `"Amedisys Holding, L.L.C."` → `"AMEDISYS HOLDING"`; `"The American College of Physicians, Inc."` → `"AMERICAN COLLEGE OF PHYSICIANS"`. - `test_alias_match_exact_and_prefix`: - - `"AMERICAN ACADEMY OF FAMILY PHYSICIANS FOUNDATION"` does not match alias `AMERICAN ACADEMY OF FAMILY PHYSICIANS` with `prefix=False`; + - `"AMERICAN ACADEMY OF FAMILY PHYSICIANS FOUNDATION"` does not match `AMERICAN ACADEMY OF FAMILY PHYSICIANS` with `prefix=False`; - `"AMEDISYS HOLDING LLC"` matches `AMEDISYS` with `prefix=True`, but `"AMEDISYSTEMS"` does not; - - `"HCA SOMETHING"` does not match the 3-char alias `HCA` with `prefix=True`. - - `test_parse_tickers`. - - `test_parse_sic_atom`: fixture → `["0001725255", "0001468328"]`. - - `test_entity_from_osh_submissions`: sic `"8000"`, status `"acquired"`, ended_on `date(2023,5,2)` from the seed. - - `test_delisted_seed_entity_kept_without_current_ticker`: AMED is absent from the tickers fixture → one `EntityRow` and `SecurityRow("AMEDISYS","AMED","NASDAQ",date(1994,8,18),date(2025,8,14),"seed")`. - - `test_filings_from_submissions_include_8k_items`: `0001193125-23-132570` has items `"1.01,1.02,2.01,3.01,3.03,5.01,5.02,5.03,8.01,9.01"` and accepted `"2023-05-02T20:42:01.000Z"`. - - `test_sic_link_only_for_health_sics`: 8082 → one sector link; 7372 → none. - - `test_ensure_tables_idempotent`. + - `"HCA SOMETHING"` does not match `HCA` with `prefix=True`. + - `test_parse_tickers`: the fixture → `{"0000790526": ["RDNT"], …}`; three keys. + - `test_parse_sic_atom`: `["0001725255", "0001468328"]`. + - `test_entity_from_osh_submissions`: key `"0001564406"`, sic `"8000"`, status `"acquired"`, `ended_on == date(2023,5,2)`, aliases from the seed. + - `test_delisted_seed_entity_kept_without_current_ticker`: AMED is absent from the tickers fixture → `EntityRow` key `"0000896262"` and `SecurityRow("0000896262","AMED","NASDAQ",date(1994,8,18),date(2025,8,14),"seed")`. + - `test_filings_follow_files_pages`: recent (3) plus page (2) → 5 rows; `0001193125-23-132570` has items `"1.01,1.02,2.01,3.01,3.03,5.01,5.02,5.03,8.01,9.01"` and accepted `"2023-05-02T20:42:01.000Z"`. + - `test_sic_link_only_for_health_sics`: SIC 8082 → one `LinkRow(scope="sector", target="8082", basis="sic", strength="sector")`; SIC 7372 → none. + - `test_write_links_upserts_and_never_wipes_other_basis`: write the sic links, then curated, then the sic link again with a new strength → 2 rows, the sic row updated. + - `test_retire_curated_links`: two curated links stored, `keep` holds one → 1 deleted; sic links untouched. + - `test_district_id`: `("MI", 12)` → `"MI-12"`; `("AK", 0)`, `("AK", None)` and `("AK", "At Large")` → `"AK-00"`; `state_codes` maps `"26"`, `"MI"` and `"Michigan"` to `("26", "MI", "Michigan")` (in-memory reference table). + - `test_ensure_tables_idempotent`: `ensure_tables` twice on `:memory:` raises nothing; every `market.*` table exists. - `test_roundtrip[entity|security|link|filing|gap]`. - - `test_entities_offline_writes_and_logs` (CLI): fixture caches → entity, security and link rows, one `cms.ingest_log` row per source file, `_publish` called once. - - `test_entities_refusal_prints_gap` (CLI): 403 → output has `gap: sec-edgar:` and nothing is written. - - `test_gaps_command_lists_rows`. - - In `test_cli_import_without_optional_deps.py`: - - add `cli.market` to `_PROBE`; - - new `test_cli_market_imports_without_numpy` (the probe also blocks `numpy`, `duckdb` and `yaml` and imports `cli.market`). -- [ ] **Step 2: Run** `uv run --no-sync pytest -q -p no:cacheprovider -n auto tests/market tests/cli/test_market_cli.py tests/cli/test_cli_import_without_optional_deps.py` → FAIL (`ModuleNotFoundError: market`). -- [ ] **Step 3: Implement** the files above. In `cli/market.py`, each command fetches everything first, then opens one `_batch()` for the writes and `log_ingest(module="market.edgar", table_name=…, rows=n, source_file=, pincite_key=)`, then calls `_publish()`. -- [ ] **Step 4: Run** the Step 2 command → PASS; run the coverage gate and ruff. -- [ ] **Step 5: Live** (network, no GPU): - - Run `ENV stack market entities --write`, then `ENV stack market entities --sic-sweep --write` (≈ 19 SIC atoms plus their submissions at 0.15 s). - - Report the entity counts by kind and status. Confirm that OSH, SGFY and AMED rows exist with tickers from the seed, and give the count of `sic` links per SIC. + - `test_market_modules_import_without_numpy_pandas` (`test_imports.py`): a subprocess probe blocks `numpy` and `pandas` and imports each module in `MODULES = ["market.http", "market.tables", "market.seed", "market.edgar"]`. Later tasks append their modules. + - CLI: + - `test_entities_offline_writes_and_logs`; + - `test_entities_no_write_prints_counts`; + - `test_entities_refusal_prints_gap` (403 → `gap: sec-edgar:entities:`, nothing written); + - `test_entities_seed_errors_abort`; + - `test_require_key_missing_writes_only_gap` (a direct call with `write=True` → one gap row, nothing else); + - `test_gap_cleared_after_success`; + - `test_gaps_command_lists_rows`. +- [ ] **Step 2: Run** `uv run --no-sync pytest -q -p no:cacheprovider -n auto tests/market tests/cli/test_market_cli.py` → FAIL (`ModuleNotFoundError: market`). +- [ ] **Step 3: Implement.** Each command fetches first, then opens one `_batch()` for the writes, `log_ingest(module="market.edgar", …)`, `clear_gap`, then calls `_publish()`. +- [ ] **Step 4: Run** → PASS; run the coverage gate, then ruff; also run `tests/cli/test_cli_import_without_optional_deps.py`, which imports `cli` and so `cli.market`. +- [ ] **Step 5: Live:** + - Run `ENV stack market entities --write`, then `ENV stack market entities --sic-sweep --write`. + - Report the entity counts by kind and status. Confirm that OSH, SGFY and AMED exist with seed tickers, that sic links appear per SIC, and that the seat-specialty societies cover `seat_specialties(..., 2024–2026)`. - [ ] **Step 6: Commit**: ```bash - git add src/market/__init__.py src/market/http.py src/market/tables.py src/market/seed.yaml src/market/seed.py src/market/edgar.py src/cli/market.py src/cli/__init__.py tests/market tests/cli/test_market_cli.py tests/cli/test_cli_import_without_optional_deps.py + git add src/market/__init__.py src/market/http.py src/market/tables.py src/market/seed.yaml src/market/seed.py src/market/edgar.py src/cli/market.py src/cli/__init__.py tests/market/__init__.py tests/market/test_http.py tests/market/test_seed.py tests/market/test_edgar.py tests/market/test_tables.py tests/market/test_imports.py tests/market/fixtures/company_tickers.json tests/market/fixtures/submissions_CIK0001564406.json tests/market/fixtures/CIK0001564406-submissions-001.json tests/market/fixtures/sic_8082.atom tests/cli/test_market_cli.py git commit -m "feat(market): EDGAR entity registry, cited seed of exposed companies and organizations, market CLI group" ``` @@ -275,147 +404,170 @@ Each item below is a planner's call. Nobody was available to ask; each is made h ### Task 2: EDGAR full-text search, exposure passages, optional local classification — `stack market fts` **Files:** -- Modify: `src/market/edgar.py`, `src/market/tables.py`, `src/cli/market.py` -- Create: `tests/market/fixtures/efts_mpfs_2023.json` (the RadNet hit), `tests/market/fixtures/filing_snippet.htm` -- Test: `tests/market/test_fts.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py` +- Modify: `src/market/edgar.py`, `src/market/tables.py`, `src/cli/market.py`. +- Create: `tests/market/fixtures/efts_mpfs_2023.json` (the RadNet hit), `tests/market/fixtures/filing_snippet.htm`. +- Test: `tests/market/test_fts.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`. **Interfaces:** -- Consumes: Task 1, `pfs.families.FAMILIES`/`refresh_from`, `pfs.origin.shares_run`, `pfs.families.CODE_RE`, `pfs.codetables.read_cpt_codes`, `pfs.runlog.RunLog`/`run_units`/`input_hash`/`ledger_path`, `cli.pfs._asker`. +- Consumes: + - Task 1 (`write_links` upsert, `sic_links`, `entity_from_submissions`, `HEALTH_SICS`, `PROGRAMS`); + - read-only imports, the documented exception under H1: `pfs.families.FAMILIES`/`refresh_from`, `pfs.codetables.cpt_years`/`read_cpt_codes`, `pfs.runlog.RunLog`/`run_units`/`input_hash`/`ledger_path`. - Produces: - - Search setup: - - `EFTS = "https://efts.sec.gov/LATEST/search-index"`; a page is 100 hits; `from` is the offset. - - `PROGRAM_TERMS = ("Medicare Physician Fee Schedule", "physician fee schedule", "RBRVS", "Relative Value Scale Update Committee", "CPT Editorial Panel")`. - - `family_terms(fam: Family) -> tuple[str, ...]`: the family name plus every multi-word synonym. - - `fts_params(term: str, form: str, year: int, offset: int) -> dict` gives `{"q": f'"{term}"', "forms": form, "dateRange": "custom", "startdt": f"{year}-01-01", "enddt": f"{year}-12-31", "from": offset}`. One query per (term, form, year) keeps each under the 10,000-hit cap. - - `HitRow(term, scope, target, cik, adsh, filename, form, file_date: date, period_ending: str, sics: str, display_name: str)` → `market.filing_hit`. `parse_hits(payload, *, term, scope, target) -> list[HitRow]`: `_id` is `"{adsh}:{filename}"`; cik is the first of `ciks`; `sics` are comma-joined. - - `doc_url(cik10: str, adsh: str, filename: str) -> str` returns `https://www.sec.gov/Archives/edgar/data/{int(cik)}/{adsh without dashes}/{filename}`. Docs are cached at `data/market/sec/docs///`. - - `PassageRow(passage_id, cik, adsh, filename, term, char_start, passage, label, label_version, item_key)` → `market.filing_passage`, where `passage_id = f"{adsh}:{filename}:{char_start}"`. - - `passages(text: str, term: str, *, max_chars=400, max_n=3) -> list[tuple[int, str]]`: a case-insensitive match, widened to sentence bounds within `max_chars`. - - `ama_safe(passage: str, descriptors: Sequence[str]) -> bool`. - - Classification: - - `EXPOSURE_LABELS = ("revenue_exposure", "risk_factor", "incidental")` and `EXPOSURE_PROMPT_VERSION = "exposure-v1"`. - - Schema `{"label": enum, "reason": str ≤ 200}`. - - `classify_passage(passage: str, term: str, ask) -> str | None`. The ledger is `ledger_path("market-exposure", "all")`. - - `fts_links(hits, passages, entities: Mapping[str, EntityRow], seed_ciks: set[str]) -> list[LinkRow]`, with basis `fts`: + - `EFTS = "https://efts.sec.gov/LATEST/search-index"`. A page is 100 hits and `from` is the offset. + - `fts_params(term, form, year, offset) -> dict` returns `{"q": f'"{term}"', "forms": form, "dateRange": "custom", "startdt": f"{year}-01-01", "enddt": f"{year}-12-31", "from": offset}`. One query per (term, form, year) keeps each under the 10,000-hit cap. + - `PROGRAM_TERMS: dict[str, tuple[str, ...]]`: + - `PFS`: "Medicare Physician Fee Schedule", "physician fee schedule", "RBRVS", "Relative Value Scale Update Committee", "CPT Editorial Panel"; + - `QPP-MIPS`: "Merit-based Incentive Payment System", "Quality Payment Program"; + - `MSSP`: "Medicare Shared Savings Program"; + - `APM`: "Advanced Alternative Payment Model"; + - `TELEHEALTH`: "Medicare telehealth"; + - `HOME-BASED-CARE`: "home-based primary care", "home-based care"; + - `CARE-MANAGEMENT`: "chronic care management", "transitional care management". + - `family_terms(fam) -> tuple[str, ...]`: the family name plus every multi-word synonym. + - `HitRow(term, scope, target, cik, adsh, filename, form, file_date: date, period_ending, sics, display_name)` → `market.filing_hit`, with `target` a program or family key. + - `parse_hits(payload, *, term, scope, target) -> list[HitRow]`; + - `_id` is `"{adsh}:{filename}"`, `cik` is the first of `ciks`, and `sics` are comma-joined. + - `doc_url(cik10, adsh, filename) -> str` returns `https://www.sec.gov/Archives/edgar/data/{int(cik)}/{adsh without dashes}/{filename}`. + - `PassageRow(passage_id, cik, adsh, filename, term, char_start, passage, label, label_version, item_key)` → `market.filing_passage`, with `passage_id = f"{adsh}:{filename}:{char_start}"`. + - `passages(text, term, *, max_chars=400, max_n=3) -> list[tuple[int, str]]` makes a case-insensitive match, widened to sentence bounds within `max_chars`. + - **AMA guard (D3):** + - `cpt_grams(descriptors: Iterable[str], n: int = 8) -> frozenset[tuple[str, ...]]` tokenizes like `pfs.origin.shares_run` (`re.findall(r"[a-z0-9]+", text.lower())`, per descriptor, no cross-descriptor runs). + - `ama_safe(passage, grams) -> bool` checks **every** passage against the grams of **all** descriptors (every edition from `cpt_years`). + - **Classifier:** + - `EXPOSURE_LABELS = ("revenue_exposure", "risk_factor", "incidental")`; `EXPOSURE_PROMPT_VERSION = "exposure-v1"`; + - schema `{"label": enum, "reason": str ≤ 200}`; + - `classify_passage(passage, term, ask) -> str | None`; + - the ledger is `ledger_path("market-exposure", "all")`. + - **`cli.market._asker() -> Any`** (the single home, M3) returns `json_extractor(cfg, HostPool.from_config(cfg))` with `cfg = llm.config.load()`. + - `fts_links(hits, passages, entities, seed_ciks) -> list[LinkRow]`, with basis `fts`: - a link needs the hit's SIC in `HEALTH_SICS` or the CIK in the seed; - strength is the passage label, or `mention` when the passage is unlabelled; - `incidental` makes no link; - - a hit without a kept passage links with `passage_id=""` and `cite_ref=adsh:filename`. - - `cite_filing(store, hit, path) -> str`: a bib Download titled `"{display_name} Form {form} filed {file_date} (accession {adsh})"`, website_title `"U.S. Securities and Exchange Commission"`, tags `module:market`, `table:market.filing_passage`, `source:sec-edgar`, `year:`, with the document attached (public record). - - CLI `stack market fts [--family KEY …] [--program] [--forms 10-K] [--from 2001] [--to ] [--limit N] [--dry-run] [--classify] [--write]`: - - FTS pages are cached at `data/market/sec/fts//
/..json`; - - a saved page or document is never refetched (resumable); - - `--classify` runs the local model through the ledger; - - with `--write`: hits, passages, and the fts links are recomputed from all passages, with bib items for the cited filings. - - Hits for CIKs not yet in `market.entity` get a submissions fetch and an `EntityRow` in the same run, so delisted filers enter the registry with no ticker. + - a hit without a kept passage links with `passage_id=""` and `cite_ref=f"{adsh}:{filename}"`. + - `cite_filing(store, hit, path) -> str`: a bib Download titled `"{display_name} Form {form} filed {file_date} (accession {adsh})"`, website_title `"U.S. Securities and Exchange Commission"`, tags `module:market`, `table:market.filing_passage`, `source:sec-edgar`, `year:`, with the document attached. + - CLI `stack market fts [--family KEY …] [--program KEY …] [--forms 10-K] [--from 2001] [--to ] [--limit N] [--dry-run] [--classify] [--write]`: + - the cache is `data/market/sec/fts///..json`, and documents are cached under `sec/docs/…`; nothing is refetched; + - a hit's CIK that is new to `market.entity` gets its submissions read and an `EntityRow` plus its `sic_links`, upserted; + - it clears `sec-edgar:fts`. - [ ] **Step 1: Failing tests:** - - `test_fts_params_one_year`: term `"Medicare Physician Fee Schedule"`, 2023 → `q == '"Medicare Physician Fee Schedule"'`, `startdt == "2023-01-01"`, `enddt == "2023-12-31"`, `from == 0`. - - `test_parse_hits`: the RadNet fixture → `HitRow(cik="0000790526", adsh="0000790526-23-000003", filename="rdnt-20221231.htm", form="10-K", file_date=date(2023,3,1), sics="8071", …)`. - - `test_fts_pages_until_total`: total 150 → offsets `[0, 100]`. - - `test_doc_url` → `"https://www.sec.gov/Archives/edgar/data/790526/000079052623000003/rdnt-20221231.htm"`. - - `test_family_terms`: HOME-VISITS includes `"house calls"` and `"home or residence"`, and excludes `"domiciliary"`. - - `test_passages_window_and_cap`: - - term at char 1000 → `char_start` ≤ 1000, length ≤ 400, the term is included; - - 5 occurrences → 3 passages. - - `test_passage_sharing_cpt_descriptor_withheld`: the hit is kept, the passage is dropped, the link has `passage_id == ""`. - - `test_fts_links_require_health_sic_or_seed`: SIC 8071 links; SIC 6798 outside the seed gives no link but the hit is stored. + - `test_fts_params_one_year`: term `"Medicare Physician Fee Schedule"`, 2023, offset 0 → `q == '"Medicare Physician Fee Schedule"'`, `startdt == "2023-01-01"`, `enddt == "2023-12-31"`, `from == 0`. + - `test_parse_hits`: `HitRow(cik="0000790526", adsh="0000790526-23-000003", filename="rdnt-20221231.htm", form="10-K", file_date=date(2023,3,1), sics="8071", …)`. + - `test_fts_pages_until_total`: `[0, 100]`. + - `test_doc_url`: → `"https://www.sec.gov/Archives/edgar/data/790526/000079052623000003/rdnt-20221231.htm"`. + - `test_family_terms`: HOME-VISITS includes `"house calls"` and `"home or residence"`, excludes `"domiciliary"`. + - `test_program_terms_keys_are_programs`: `set(PROGRAM_TERMS) == set(PROGRAMS)`. + - `test_passages_window_and_cap`: term at char 1000 → `char_start` ≤ 1000, length ≤ 400, term included; 5 occurrences → 3 passages. + - `test_ama_guard_matches_shares_run`: for 3 (passage, descriptor) pairs, `ama_safe(p, cpt_grams([d])) == (not shares_run(p, d))`. + - `test_passage_quoting_descriptor_without_code_withheld` (D3): the passage quotes 9 descriptor words and names no code → withheld, hit kept, link `passage_id == ""`. + - `test_fts_links_require_health_sic_or_seed`: SIC 8071 → link; SIC 6798 not in seed → no link, hit stored. - `test_link_strength_from_label`: None → `mention`; `incidental` → no link; `revenue_exposure` → `revenue_exposure`. - - `test_classify_passage_closed_labels`: the fake ask returns `{"label":"revenue_exposure"}` → that label; `{"label":"bogus"}` → None. + - `test_classify_passage_closed_labels`: fake ask `{"label":"revenue_exposure"}` → that label; `{"label":"bogus"}` → None. - `test_roundtrip[filing_hit|filing_passage]`. - - `test_fts_cli_dry_run_lists_queries`: prints one line per (term, form, year) and makes no request. - - `test_fts_cli_classify_resumes_from_ledger`: a second run makes 0 new model calls. -- [ ] **Step 2: Run** `tests/market/test_fts.py tests/cli/test_market_cli.py` → FAIL. -- [ ] **Step 3: Implement.** -- [ ] **Step 4: Run** → PASS; coverage gate; ruff. + - CLI: + - `test_fts_cli_dry_run_lists_queries`; + - `test_fts_cli_write_adds_new_cik_entity_and_sic_link` (an existing curated link survives); + - `test_fts_cli_refusal_prints_gap`; + - `test_fts_cli_classify_resumes_from_ledger`; + - `test_asker_builds_json_extractor` (monkeypatched `llm.config.load`, `HostPool.from_config` and `json_extractor` → a sentinel). +- [ ] **Step 2: Run** `tests/market/test_fts.py tests/cli/test_market_cli.py` → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff. - [ ] **Step 5: Live:** - - Run `ENV stack market fts --program --from 2001 --dry-run | tail -3`, then `--write` as a long run (`.state/market/fts-program.log`). + - Run `ENV stack market fts --program PFS --dry-run | tail -3`, then the long run `ENV stack market fts --write` (all programs; `.state/market/fts.log`). - Run `ENV stack market fts --family HOME-VISITS --family CCM --family TCM --family APCM --write`. - - Optional GPU run: check `ENV stack llm hosts`, then `ENV stack market fts --program --classify --limit 200 --write`. - - Report hits per term, the passages withheld by the AMA guard, and new fts links per family. -- [ ] **Step 6: Commit** `git add src/market/edgar.py src/market/tables.py src/cli/market.py tests/market/test_fts.py tests/market/test_tables.py tests/market/fixtures/efts_mpfs_2023.json tests/market/fixtures/filing_snippet.htm tests/cli/test_market_cli.py && git commit -m "feat(market): EDGAR full-text exposure passages per program term and family, cited and AMA-guarded"` + - Optional GPU run: check `ENV stack llm hosts`, then `ENV stack market fts --program PFS --classify --limit 200 --write`. + - Report hits per program and term, passages withheld by the guard, new entities, and fts links per target. +- [ ] **Step 6: Commit** `git add src/market/edgar.py src/market/tables.py src/cli/market.py tests/market/test_fts.py tests/market/test_tables.py tests/market/fixtures/efts_mpfs_2023.json tests/market/fixtures/filing_snippet.htm tests/cli/test_market_cli.py && git commit -m "feat(market): EDGAR full-text exposure passages per program and family, cited and AMA-guarded"` --- ### Task 3: Daily prices from Tiingo and the monthly series (#797) — `stack market prices` **Files:** -- Create: `src/market/tiingo.py`, `tests/market/fixtures/supported_tickers_head.csv` (5 lines) -- Modify: `src/market/tables.py`, `src/cli/market.py` -- Test: `tests/market/test_tiingo.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py` +- Create: `src/market/tiingo.py`, `tests/market/fixtures/supported_tickers_head.csv`. +- Modify: `src/market/tables.py`, `src/cli/market.py`, `tests/market/test_imports.py` (add `"market.tiingo"`). +- Test: `tests/market/test_tiingo.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`. **Interfaces:** -- Consumes: Task 1 (`client("tiingo", key)`, `Pacer`, `get`, `hoard`, `save`, `read_securities`, `read_links`). +- Consumes: Task 1 (`client("tiingo", key)`, `Pacer`, `get`, `hoard`, `save`, `read_securities`, `read_links`, `read_entities`). - Produces: - Endpoints: - `EOD = "https://api.tiingo.com/tiingo/daily/{ticker}/prices"` with params `startDate`, `endDate`, `format=json`, `resampleFreq=daily`; - `META = "https://api.tiingo.com/tiingo/daily/{ticker}"`; - `SUPPORTED = "https://apimedia.tiingo.com/docs/tiingo/daily/supported_tickers.zip"` (keyless, no budget). - - `BENCHMARKS = {"SPY": ("market", date(1993,1,29)), "XLV": ("sector", date(1998,12,22)), "IHF": ("providers", date(2006,5,5))}`. Start dates are from Tiingo metadata read 2026-11-12. XLV is the sector index (planner's call); IHF is a providers sensitivity index. - - `@dataclass Budget(month: str, symbols: set[str], day_counts: dict[str, int])`: - - stored in `.state/market/tiingo-.json`; - - caps `MONTHLY_SYMBOLS = 450`, `DAILY_REQUESTS = 950`, pacer 75.0 s (48 per hour). The Starter limits are 500 symbols a month, 1,000 requests a day and 50 an hour. - - `Budget.allow(ticker: str, today: date) -> bool`; `Budget.spend(ticker, today) -> None`. - - `PriceRow(ticker, day: date, open, high, low, close, volume, adj_close, div_cash, split_factor, ret: float | None, ret_raw: float | None, pulled_on: date, source_file: str)` → `market.price_daily`. + - `BENCHMARKS = {"SPY": ("market", date(1993,1,29)), "XLV": ("sector", date(1998,12,22)), "IHF": ("providers", date(2006,5,5))}`. + - **Budget:** `Budget(month, symbols, day_counts)` is kept in `.state/market/tiingo-.json`, with `MONTHLY_SYMBOLS = 450`, `DAILY_REQUESTS = 950` and a 75.0 s pacer. `allow(ticker, today)` and `spend(ticker, today)`. + - **Universe (M4):** `universe(con, *, used: set[str], cap: int = MONTHLY_SYMBOLS) -> list[tuple[str, str]]` returns `(ticker, entity_key)` pairs in rank order: + 1. the benchmarks; + 2. the seed securities, in seed order; + 3. the securities of entities with ≥ 1 `fts` link, ranked by best strength (`revenue_exposure` > `risk_factor` > `mention`), then the number of distinct link targets (descending), then the newest 10-K `filed` (descending). + - Entities whose only links are `sic` are excluded. + - Tickers absent from the supported list are excluded. + - Symbols in `used` do not count against `cap`. + - `parse_supported(zip_bytes: bytes) -> dict[str, tuple[date | None, date | None]]` reads `supported_tickers.csv` and keeps USD rows. The zip is cached at `data/market/tiingo/supported_tickers.zip` and is free of budget. + - `PriceRow(ticker, day: date, open, high, low, close, volume, adj_close, div_cash, split_factor, ret: float | None, ret_raw: float | None, pulled_on: date, source_file)` → `market.price_daily`. - `to_prices(ticker, records, *, pulled_on, source_file, overlap: date | None) -> list[PriceRow]`: - - `ret = adjClose_t / adjClose_{t-1} − 1` within this pull only; + - `ret = adjClose_t / adjClose_{t-1} − 1`, within this pull only; - `ret_raw = (close_t + divCash_t) · splitFactor_t / close_{t-1} − 1`; - - the `overlap` day (the last stored day, re-requested as the pull's first row) is dropped, so stored rows keep their returns. - - `clip(rows, valid_from, valid_to) -> list[PriceRow]`. - - `window_gap(meta: dict, sec: SecurityRow, tol_days=7) -> str | None`: Tiingo's `startDate`/`endDate` against the security window. - - `MonthlyRow(series_id, symbol, source, year, month, value, item_key)` → `market.series`, the #797 shape. `tri_monthly(ticker, rows, *, item_key) -> list[MonthlyRow]`: - - `series_id = f"tiingo:{ticker}:tri"`; - - the value is a total-return index chained from `ret`, base 100.0 on the first stored day, taken at each month's last trading day. - - `cite(store, ticker) -> str`: a bib Source with url `EOD` formatted, **no attachment**, tags `module:market`, `table:market.price_daily`, `source:tiingo`. - - Readers in tables: - - `read_returns(con, tickers, start: date, end: date) -> dict[str, dict[date, float]]`; - - `trading_days(con, ticker="SPY") -> list[date]`; - - `read_series(con, symbols) -> list[MonthlyRow]`. - - Writer `write_prices(con, ticker, rows)` deletes that ticker's rows in `[min(day), max(day)]`, then inserts. - - CLI `stack market prices [--ticker T …] [--linked] [--benchmarks] [--limit N] [--refresh] [--dry-run] [--write]`: - - `--linked` covers every security of an entity with any link. - - A first pull asks for full history from `max(valid_from, 1993-01-29)`. Later pulls start at the last stored day. - - Each pull is cached at `data/market/tiingo/eod//_.json`, metadata at `tiingo/meta/.json`. - - No `TIINGO_API_KEY` → `gap: tiingo: TIINGO_API_KEY not in .env`. + - the `overlap` day is dropped, so stored rows keep their returns. + - `clip(rows, valid_from, valid_to)`. + - `window_gap(meta, sec, tol_days=7) -> str | None` compares Tiingo's `startDate`/`endDate` with the security window. + - `MonthlyRow(series_id, symbol, source, year, month, value, item_key)` → `market.series`. `tri_monthly(ticker, rows, *, item_key)`, where `series_id = f"tiingo:{ticker}:tri"`. + - `cite(store, ticker)` is a bib Source with **no attachment**. + - Readers: `read_returns(con, tickers, start, end)`, `trading_days(con, ticker="SPY")`, `read_series(con, symbols)`. Writer: `write_prices(con, ticker, rows)` (delete-in-range, then insert). + - CLI `stack market prices [--ticker T …] [--universe] [--benchmarks] [--limit N] [--refresh] [--dry-run] [--write]`: + - a first pull starts at `max(valid_from, 1993-01-29)`, or at the benchmark start; later pulls start at the last stored day; + - pulls are cached at `data/market/tiingo/eod//_.json`; + - `--dry-run` prints the ranked universe and the remaining budget; + - it clears `tiingo:prices`. - [ ] **Step 1: Failing tests** (synthetic prices only): - - `test_to_prices_returns_within_pull`: adj 100, 110, 99 → ret `[None, 0.1, -0.1]` (approx). - - `test_ret_raw_with_split_and_dividend`: close 100 → 50 with split 2.0 gives 0.0; close 100 → 99 with div 1.0 gives 0.0. - - `test_incremental_pull_rebased_adjclose_keeps_returns`: pull 1 is days d1, d2 at adj 100, 110. Pull 2 is d2, d3 at adj 55, 60.5 (rebased by ×0.5) with overlap d2. Result: the stored d2 row is pull 1's; d3 ret = 0.1. - - `test_prices_clipped_to_security_window`: AGL rows dated 2015-06-01 and 2021-05-03 with `valid_from=2021-04-15` → only 2021 is kept. - - `test_window_gap`: metadata `startDate` 2021-04-15 vs `valid_from` 2021-04-15 → None; `endDate` 2023-05-02 vs `valid_to` 2023-06-30 → a note naming both dates. - - `test_budget_monthly_symbols_and_daily_cap`: with 450 symbols used, a new symbol is refused and a known one allowed; at 950 requests today, everything is refused. - - `test_monthly_tri_month_end`: rets on 2024-01-02 None, 2024-01-03 0.1, 2024-02-01 −0.1 → rows `(2024,1)=110.0`, `(2024,2)=99.0`. - - `test_benchmarks_pinned`. - - `test_tiingo_cite_has_no_attachment`: the fake store records `upsert` and `attach_file` is never called; the url contains no `token`. + - `test_to_prices_returns_within_pull`: adj 100, 110, 99 → ret `[None, approx(0.1), approx(-0.1)]`. + - `test_ret_raw_with_split_and_dividend`: close 100 → 50 with split 2.0 → `ret_raw == 0.0`; close 100 → 99 with div 1.0 → `0.0`. + - `test_incremental_pull_rebased_adjclose_keeps_returns`: pull 1 d1, d2 adj 100, 110; pull 2 d2, d3 adj 55, 60.5 (rebased ×0.5) with overlap d2 → stored d2 row is pull 1's; d3 `ret == approx(0.1)`. + - `test_prices_clipped_to_security_window`: AGL rows 2015-06-01 and 2021-05-03, `valid_from=2021-04-15` → only 2021-05-03 kept. + - `test_window_gap`: meta `startDate` 2021-04-15 vs `valid_from` 2021-04-15 → None; `endDate` 2023-05-02 vs `valid_to` 2023-06-30 → a note naming both dates. + - `test_budget_monthly_symbols_and_daily_cap`: 450 symbols used → a new symbol refused, a used one allowed; 950 requests today → all refused. + - `test_monthly_tri_month_end`: `(2024,1)=110.0`, `(2024,2)=99.0`. + - `test_universe_ranked_capped_excludes_sic_only`: + - the benchmarks come first; + - a seed security comes before an fts entity; + - a `revenue_exposure` entity outranks a `mention` entity; + - a sic-only entity is absent; + - `cap=5` truncates; + - a `used` symbol does not consume the cap; + - a ticker missing from the supported list is absent. + - `test_parse_supported`: the fixture `supported_tickers_head.csv`, zipped in the test → `OSH` gives `(date(2020,8,6), date(2023,5,2))`; a CNY row is dropped. + - `test_benchmark_first_pull_starts_at_listing`: the first EOD request for XLV has `startDate=1998-12-22`, and for SPY `1993-01-29` (MockTransport). + - `test_tiingo_cite_has_no_attachment`: fake store records `upsert`; `attach_file` never called; url has no `token`. - `test_roundtrip[price_daily|series]`. - - `test_read_returns_and_trading_days`. - - `test_prices_without_key_records_gap` (CLI): exit 0, output starts with `gap: tiingo:`. - - `test_prices_dry_run_reports_budget` (CLI). -- [ ] **Step 2: Run** → FAIL. -- [ ] **Step 3: Implement.** -- [ ] **Step 4: Run** → PASS; coverage; ruff. + - `test_read_returns_and_trading_days`: two tickers in `:memory:` → `{ticker: {date: ret}}` with None rets dropped; `trading_days` = SPY dates sorted. + - CLI: + - `test_prices_cli_fetch_write` (MockTransport, two tickers, rows plus ingest_log, publish once); + - `test_prices_dry_run_reports_budget`; + - `test_prices_without_key_records_gap` (`gap: tiingo:prices:`, nothing else written); + - `test_prices_refusal_prints_gap`. +- [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff. - [ ] **Step 5: Live:** - - Run `ENV stack market prices --benchmarks --write` (3 symbols). Report the row counts and first/last days per symbol: SPY must start 1993-01-29 and XLV 1998-12-22. - - Then run `ENV stack market prices --linked --dry-run`, followed by the long run `--linked --write` (`.state/market/prices.log`, about 48 symbols an hour). - - Report symbols fetched, the window gaps, and the days where `|ret − ret_raw| > 0.001`. Report counts only. -- [ ] **Step 6: Commit** `git add src/market/tiingo.py src/market/tables.py src/cli/market.py tests/market/test_tiingo.py tests/market/test_tables.py tests/market/fixtures/supported_tickers_head.csv tests/cli/test_market_cli.py && git commit -m "feat(market): Tiingo daily prices with total returns, benchmarks SPY/XLV/IHF, monthly series (#797)"` + - Run `ENV stack market prices --benchmarks --write`. Report rows and the first/last day per symbol: SPY must start 1993-01-29 and XLV 1998-12-22. + - Run `ENV stack market prices --universe --dry-run`, then the long run `--universe --write` (`.state/market/prices.log`). + - Report symbols fetched against the budget, the window gaps, and the number of days where `|ret − ret_raw| > 0.001`. +- [ ] **Step 6: Commit** `git add src/market/tiingo.py src/market/tables.py src/cli/market.py tests/market/test_tiingo.py tests/market/test_tables.py tests/market/test_imports.py tests/market/fixtures/supported_tickers_head.csv tests/cli/test_market_cli.py && git commit -m "feat(market): Tiingo daily prices for a ranked, capped universe; benchmarks SPY/XLV/IHF; monthly series (#797)"` --- ### Task 4: Lobbying disclosures (LDA) — `stack market lda` **Files:** -- Create: `src/market/lda.py`, `tests/market/fixtures/lda_aafp_2024_q1.json` (one filing, lobbyists removed) -- Modify: `src/market/tables.py`, `src/cli/market.py` -- Test: `tests/market/test_lda.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py` +- Create: `src/market/lda.py`, `tests/market/fixtures/lda_aafp_2024_q1.json` (one filing, lobbyists removed). +- Modify: `src/market/tables.py`, `src/cli/market.py`, `tests/market/test_imports.py`. +- Test: `tests/market/test_lda.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`. **Interfaces:** -- Consumes: Task 1 (`client("lda", key)`, `api_key("lda")`, `alias_match(prefix=False)`, organization and company entities with `aliases`). +- Consumes: Task 1 (`client("lda", key)`, `api_key("lda")`, `alias_match(prefix=False)`, `EntityRow.aliases`). - Produces: - - Endpoint `LDA_API = "https://lda.gov/api/v1/"`. `lda.senate.gov` was sunset on 2026-07-31 and returns a 503 maintenance page. - - `page_size = 25` (the API maximum). Pacer 4.2 s anonymous (the limit is 15/min) and 0.55 s with `LDA_API_KEY` (120/min). - - `FIRST_YEAR = 2008` (planner's call: LD-203 starts in 2008). + - `LDA_API = "https://lda.gov/api/v1/"` (`lda.senate.gov` was sunset 2026-07-31 and returns 503). + - `page_size = 25`; pacer 4.2 s anonymous, 0.55 s keyed. + - `FIRST_YEAR = 2008`. - `LDA_TERMS = ("physician fee schedule", "conversion factor", "relative value", "evaluation and management", "home-based")`, sent as `filing_specific_lobbying_issues=""`. - `HEALTH_ISSUES = ("MMM", "HCR", "MED", "INS", "PHA", "BUD", "TAX")`. - Rows: @@ -425,518 +577,573 @@ Each item below is a planner's call. Nobody was available to ask; each is made h - `amount = income` when income is not None, else `expenses`; - registrations (`RR`, `RA`) have `amount=None`; - decimal strings become floats. - - `supersede(rows) -> set[str]`: within each (registrant_id, client_id, filing_year, filing_period), every report-type filing except the latest `dt_posted` is superseded. + - `supersede(rows: Sequence[LdaFilingRow]) -> set[str]` is the rule stated in Decisions and its only implementation (D4). + - `write_lda(con, rows, issues)` reads the stored rows of the (registrant, client, year, period) groups it touches, runs `supersede()` over stored ∪ new, deletes those groups, and inserts them with the flags set. - `query_pages(cl, params, *, pacer) -> Iterator[dict]` follows `next`. - `spend_by_year(con, entity_key) -> dict[int, float]` sums non-superseded amounts once per filing. `filings_by_issue(con, entity_key, code) -> list[str]`. - - `write_lda(con, rows, issues)` deletes by `filing_uuid`; then `mark_superseded(con)` recomputes the flag in SQL. - - CLI `stack market lda [--org KEY …] [--terms] [--from 2008] [--to ] [--limit N] [--write]`: - - tracked entities are queried by `client_name` and `registrant_name` per year, and the API's fuzzy name matches are post-filtered with `alias_match(prefix=False)`; - - `--terms` adds the discovery queries, with `entity_key=""` and `matched_by="term:"`; + - CLI `stack market lda [--entity KEY …] [--terms] [--from 2008] [--to ] [--limit N] [--write]`: + - tracked entities are organizations and seed companies, using their `aliases`; - the cache is `data/market/lda//..json`; - - one bib Source per filing year (url `https://lda.gov/api/v1/filings/?filing_year=`), with no attachments. + - one bib Source per filing year; + - it clears `lda:lda`. - [ ] **Step 1: Failing tests:** - - `test_lda_base_is_lda_gov`. - - `test_lda_pacer_anonymous_and_keyed`: no key → 4.2; key → 0.55 and the header `Authorization: Token k`. - - `test_to_filing_amounts`: fixture uuid `df5f17c4-62a7-4d7f-8ebe-aeb15147f396` → filing_type `"Q1"`, amount `825016.0`, income None, and the issues include `"BUD"`. - - `test_amendment_supersedes_original`: a Q1 posted 2024-04-15 and a `1A` posted 2024-05-01 for the same pair and period → Q1 superseded, 1A not. - - `test_registration_has_no_amount`. - - `test_issue_codes_never_split_amount`: a filing with 3 issues and amount 300.0 → `spend_by_year == {2024: 300.0}`, and `filings_by_issue` returns it for each code. - - `test_name_post_filter_drops_foundation`. - - `test_query_pages_follows_next`. + - `test_lda_requests_go_to_lda_gov`: the MockTransport records host `"lda.gov"`, never `"lda.senate.gov"`. + - `test_lda_pacer_anonymous_and_keyed`: no key → `Pacer(4.2)`; key → `Pacer(0.55)` and header `Authorization: Token k`. + - `test_to_filing_amounts`: uuid `df5f17c4-62a7-4d7f-8ebe-aeb15147f396`, `"Q1"`, amount 825016.0, issue `"BUD"`. + - `test_amendment_supersedes_original`: Q1 posted 2024-04-15 and `1A` posted 2024-05-01, same registrant/client/year/period → Q1 in `supersede()`, 1A not. + - `test_registration_has_no_amount`: `RR` with expenses 100.00 → `amount is None`. + - `test_issue_codes_never_split_amount`: `{2024: 300.0}`. + - `test_write_lda_resupersedes_with_stored_rows`: Q1 is stored first, then a 1A is written alone → the stored Q1 is now superseded. + - `test_name_post_filter_drops_foundation`: API returns `AMERICAN ACADEMY OF FAMILY PHYSICIANS FOUNDATION` for alias `AMERICAN ACADEMY OF FAMILY PHYSICIANS` → not kept. + - `test_query_pages_follows_next`: two pages linked by `next` → 2 requests, all results yielded. - `test_roundtrip[lda_filing|lda_issue]`. - - `test_lda_cli_terms_offline` (CLI). + - CLI: `test_lda_cli_entity_write`, `test_lda_cli_terms_offline`, `test_lda_cli_refusal_prints_gap`. - [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff. -- [ ] **Step 5: Live:** `ENV stack market lda --from 2024 --to 2024 --limit 50` (anonymous), then the long run `ENV stack market lda --terms --write` (`.state/market/lda.log`; about 14 requests a minute). Report filings per organization type, the superseded count, and the top 10 clients by 2024 spend. -- [ ] **Step 6: Commit** `git add src/market/lda.py src/market/tables.py src/cli/market.py tests/market/test_lda.py tests/market/test_tables.py tests/market/fixtures/lda_aafp_2024_q1.json tests/cli/test_market_cli.py && git commit -m "feat(market): LDA lobbying filings and issue codes for tracked organizations and PFS terms (lda.gov)"` +- [ ] **Step 5: Live:** + - Run `ENV stack market lda --from 2024 --to 2024 --limit 50`, then the long run `ENV stack market lda --terms --write`. + - Report filings per organization type, the superseded count, and the top 10 clients by 2024 spend. +- [ ] **Step 6: Commit** `git add src/market/lda.py src/market/tables.py src/cli/market.py tests/market/test_lda.py tests/market/test_tables.py tests/market/test_imports.py tests/market/fixtures/lda_aafp_2024_q1.json tests/cli/test_market_cli.py && git commit -m "feat(market): LDA lobbying filings and issue codes for tracked organizations and PFS terms (lda.gov)"` --- ### Task 5: FEC committees, totals and contributions — `stack market fec` **Files:** -- Create: `src/market/fec.py`, `tests/market/fixtures/fec_committees_aafp.json`, `fec_totals_C00411553_2024.json`, `fec_schedule_b_page1.json` (synthetic, 3 rows incl. one memo) -- Modify: `src/market/tables.py`, `src/cli/market.py` -- Test: `tests/market/test_fec.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py` +- Create: `src/market/fec.py`, `tests/market/fixtures/fec_committees_aafp.json`, `tests/market/fixtures/fec_totals_C00411553_2024.json`, `tests/market/fixtures/fec_schedule_b_page1.json` (synthetic; 3 rows including one memo row and one individual recipient). +- Modify: `src/market/tables.py`, `src/cli/market.py`, `tests/market/test_imports.py`. +- Test: `tests/market/test_fec.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`. **Interfaces:** -- Consumes: Task 1 (`api_key("fec")`, `client("fec", key)`, `alias_match`, seed `fec` ids). +- Consumes: Task 1 (`api_key("fec")`, which resolves to `API_DATA_GOV_KEY`; `client("fec", key)`; `alias_match(prefix=False)`; seed `fec_committee_ids`). - Produces: - - `FEC_API = "https://api.open.fec.gov/v1/"`. The key goes in the `X-Api-Key` header. Pacer 3.7 s (api.data.gov allows 1,000 an hour). `CYCLES = tuple(range(2008, 2028, 2))`. + - `FEC_API = "https://api.open.fec.gov/v1/"`; the key goes in `X-Api-Key`; pacer 3.7 s; `CYCLES = tuple(range(2008, 2028, 2))`. - Rows: - - `FecCommitteeRow(committee_id, name, committee_type, designation, entity_key, first_file_date, last_file_date, source_file)` → `market.fec_committee`; + - `FecCommitteeRow(committee_id, name, committee_type, designation, entity_key, basis, cite_url, first_file_date, last_file_date, source_file)`, with basis `curated` or `exact-name` → `market.fec_committee`; - `FecTotalsRow(committee_id, cycle, receipts, disbursements, individual_contributions, fed_candidate_committee_contributions, independent_expenditures, coverage_start: date | None, coverage_end: date | None, source_file)` → `market.fec_totals`; the field names were verified 2026-11-12; - - `FecContributionRow(sub_id, committee_id, cycle, day: date | None, amount, recipient_committee_id, recipient_name, candidate_id, candidate_name, candidate_office, candidate_state, candidate_district, memo_code, source_file)` → `market.fec_contribution`. - - `resolve(cl, org: SeedOrg | SeedCompany, *, pacer) -> tuple[list[FecCommitteeRow], list[dict]]`: - - uses `/committees/?q=&committee_type=Q&committee_type=N&per_page=20`; - - auto-accepts a committee when a normalized alias occurs in its normalized name, and returns the rest as candidates; - - seed `fec` ids override. + - `FecContributionRow(sub_id, committee_id, cycle, day: date | None, amount, recipient_committee_id, recipient_name, candidate_id, candidate_name, candidate_office, candidate_state, candidate_district, memo_code, source_file)` → `market.fec_contribution`, storing every Schedule B row as filed, recipients' names included (D2). + - `resolve(cl, entity: EntityRow, seed_ids: Sequence[str], *, pacer) -> tuple[list[FecCommitteeRow], list[dict]]` implements the one rule in Decisions: + - curated ids are fetched with `/committee/{id}/`; + - otherwise a `/committees/?q=` result is accepted only when `alias_match(result.name, aliases, prefix=False)`, with `cite_url = f"https://www.fec.gov/data/committee/{id}/"`; + - every other result is a candidate, printed and not stored. - `totals(cl, cid, cycle, *, pacer) -> FecTotalsRow | None` uses `/committee/{cid}/totals/?cycle=`. - `contributions(cl, cid, cycle, *, pacer) -> list[FecContributionRow]` uses `/schedules/schedule_b/?committee_id=…&two_year_transaction_period=…&per_page=100&sort=-disbursement_date`. Keyset paging takes `last_index` and `last_disbursement_date` from `pagination.last_indexes`. - `contributions_by_cycle(con, entity_key) -> dict[int, float]` excludes `memo_code == "X"`. - - CLI `stack market fec [--org KEY …] [--cycles 2020,2022] [--limit N] [--write]`: - - with no key: `gap: fec: API_DATA_GOV_KEY (or FEC_API_KEY) not in .env — key pending (ci@mail.fhirworx.io)`, plus a gap row with `--write`, exit 0; - - the cache is `data/market/fec/{committees/.json, totals//.json, schedule_b///.json}`; - - candidates the code did not accept are printed for curation into `seed.yaml`. + - CLI `stack market fec [--entity KEY …] [--cycles 2020,2022] [--limit N] [--write]`: + - the cache is `data/market/fec/{committees/.json, committee/.json, totals//.json, schedule_b///.json}`; + - it clears `fec:fec`. - [ ] **Step 1: Failing tests:** - - `test_fec_without_key_records_gap`. - - `test_committee_auto_accept_contains_alias`: `C00411553` "AMERICAN ACADEMY OF FAMILY PHYSICIANS POLITICAL ACTION COMMITTEE (FAMMEDPAC)" is accepted for AAFP; an unrelated name becomes a candidate. - - `test_seed_fec_ids_override_search`. - - `test_totals_row`: cycle 2024, receipts 513798.01, disbursements 696685.91, fed_candidate_committee_contributions 683300.0, coverage `date(2023,1,1)`..`date(2024,12,31)`. - - `test_schedule_b_keyset_paging`: the second request carries `last_index=123` and `last_disbursement_date=2024-03-01`. - - `test_memo_rows_excluded_from_totals`: rows 100 + 50 (memo X) + 25 → 125.0. - - `test_fec_error_message_has_no_key`. + - `test_committee_curated_ids_win`: seed id `C00411553` → `basis == "curated"` without a search request. + - `test_committee_exact_name_fallback_cited`: a result named "AMERICAN ACADEMY OF FAMILY PHYSICIANS POLITICAL ACTION COMMITTEE (FAMMEDPAC)" is a **candidate** for the alias `AMERICAN ACADEMY OF FAMILY PHYSICIANS`. It is accepted, with `cite_url` `https://www.fec.gov/data/committee/C00411553/`, when the aliases include `AMERICAN ACADEMY OF FAMILY PHYSICIANS POLITICAL ACTION COMMITTEE FAMMEDPAC`. + - `test_short_alias_never_matches_inside_names`: the alias `AMA` does not accept "AMALGAMATED … PAC". + - `test_totals_row`: receipts 513798.01, disbursements 696685.91, fed_candidate_committee_contributions 683300.0, coverage 2023-01-01..2024-12-31. + - `test_schedule_b_keyset_paging`: second request carries `last_index=123` and `last_disbursement_date=2024-03-01`. + - `test_memo_rows_excluded_from_totals`: 125.0. + - `test_schedule_b_rows_stored_as_filed`: the individual recipient row is kept with its name (D2). + - `test_fec_error_message_has_no_key`: 403 → `Refused`; the key string is in no exception text and no requested URL. - `test_roundtrip[fec_committee|fec_totals|fec_contribution]`. - - `test_fec_cli_offline_from_cache` (CLI). + - CLI: `test_fec_cli_write_from_cache`, `test_fec_cli_no_write_prints_candidates`, `test_fec_cli_refusal_prints_gap`. - [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff. -- [ ] **Step 5: Live:** - - If the key is present: `ENV stack market fec --org AAFP --cycles 2024 --write`, then a long run without `--org`. Report the committees resolved and unresolved, and the 2024 receipts per org type. - - If not: `ENV stack market fec --write` must print the gap line and leave one `market.source_gap` row. -- [ ] **Step 6: Commit** `git add src/market/fec.py src/market/tables.py src/cli/market.py tests/market/test_fec.py tests/market/test_tables.py tests/market/fixtures/fec_*.json tests/cli/test_market_cli.py && git commit -m "feat(market): FEC PAC committees, cycle totals and committee contributions (key-gated, gap until present)"` +- [ ] **Step 5: Live:** run `ENV stack market fec --entity AAFP --cycles 2024 --write`, then the long run without `--entity`. Report committees resolved (curated or exact-name) and the candidates left, receipts and candidate contributions per organization type for 2024, and Schedule B rows per cycle. +- [ ] **Step 6: Commit** `git add src/market/fec.py src/market/tables.py src/cli/market.py tests/market/test_fec.py tests/market/test_tables.py tests/market/test_imports.py tests/market/fixtures/fec_committees_aafp.json tests/market/fixtures/fec_totals_C00411553_2024.json tests/market/fixtures/fec_schedule_b_page1.json tests/cli/test_market_cli.py && git commit -m "feat(market): FEC PAC committees (curated ids, exact-name fallback), cycle totals and Schedule B"` --- ### Task 6: Members of Congress and committee assignments — `stack market members`, `stack market committees` **Files:** -- Create: `src/market/congress.py`, `tests/market/fixtures/congress_members_118.json` (2 members), `legislators_snippet.yaml`, `membership_snippet.yaml` -- Modify: `src/market/tables.py`, `src/cli/market.py` -- Test: `tests/market/test_congress.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py` +- Create: `src/market/congress.py`, `tests/market/fixtures/congress_members_118.json` (2 members, full state names), `tests/market/fixtures/legislators_snippet.yaml`, `tests/market/fixtures/membership_snippet.yaml`. +- Modify: `src/market/tables.py`, `src/cli/market.py`, `tests/market/test_imports.py`. +- Test: `tests/market/test_congress.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`. **Interfaces:** -- Consumes: Task 1. +- Consumes: Task 1 (`client("congress", key)`, `state_codes`, `district_id`). - Produces: - Endpoints: - - `CONGRESS_API = "https://api.congress.gov/v3/"`, with `X-Api-Key`, pacer 0.8 s (the limit is 5,000 an hour), `limit=250` and `offset`. - - `LEGISLATORS = "https://raw.githubusercontent.com/unitedstates/congress-legislators/{ref}/{name}"` (public domain, keyless) for `legislators-current.yaml`, `legislators-historical.yaml` and `committee-membership-current.yaml`. - - `GH_COMMITS = "https://api.github.com/repos/unitedstates/congress-legislators/commits"`, pacer 61 s (60 an hour unauthenticated). - - `congress_of(year: int) -> int` = `(year − 1789) // 2 + 1`. `CONGRESSES = range(110, 120)`. - - `HEALTH_COMMITTEES = ("HSWM", "HSWM02", "HSIF", "HSIF14", "SSFI", "SSFI10", "SSHR", "SSHR12")`, verified 2026-11-12 in `committees-current.yaml`: W&M Health, E&C Health, Finance Health Care, HELP Primary Health and Retirement Security. + - `CONGRESS_API = "https://api.congress.gov/v3/"`, with `X-Api-Key`, pacer 0.8 s (the limit is 5,000 an hour), `limit=250` and `offset`; + - `LEGISLATORS = "https://raw.githubusercontent.com/unitedstates/congress-legislators/{ref}/{name}"` (public domain, keyless), for `legislators-current.yaml`, `legislators-historical.yaml` and `committee-membership-current.yaml`; + - `GH_COMMITS = "https://api.github.com/repos/unitedstates/congress-legislators/commits"`, with `path`, `until`, `per_page=1` and pacer 61 s. + - `congress_of(year) -> int`; `first_year(congress) -> int` (`1789 + 2·(congress − 1)`); `snapshot_cutoff(congress) -> date` (April 1 of `first_year`). + - `CONGRESSES = range(110, 120)`; `COMMITTEE_CONGRESSES = range(113, 120)`. + - `HEALTH_COMMITTEES = ("HSWM", "HSWM02", "HSIF", "HSIF14", "SSFI", "SSFI10", "SSHR", "SSHR12")`. - Rows: - - `MemberRow(congress, bioguide_id, name, party, state, district: int | None, chamber, fec_ids: str, source)` → `market.congress_member`; chamber is `house` or `senate`; source is `congress.gov` or `congress-legislators`; - - `AssignmentRow(congress, snapshot: date, committee_id, bioguide_id, rank: int | None, title, party, source_sha)` → `market.committee_assignment`. - - Parsers: - - `members_from_api(payload, congress) -> list[MemberRow]` takes the term that covers the congress; - - `members_from_yaml(docs, congresses) -> list[MemberRow]`, one row per (congress, bioguide) whose term overlaps it; - - `fec_ids_by_bioguide(docs) -> dict[str, tuple[str, ...]]`; - - `membership_rows(doc: dict, *, snapshot: date, sha: str) -> list[AssignmentRow]` keeps `HEALTH_COMMITTEES` only. - - Committee assignments come from congress-legislators snapshots (planner's call): the Congress.gov API has no committee membership. One snapshot per year, the last commit before March 1, for 2013 through this year; earlier years are a gap. + - `MemberRow(congress, bioguide_id, name, party, state, district_id, chamber, fec_ids, source)` → `market.congress_member`, where `state` is postal and `district_id` is `"MI-12"` for the House and `""` for the Senate; + - `AssignmentRow(congress, snapshot: date, committee_id, bioguide_id, rank, title, party, source_sha)` → `market.committee_assignment`, one snapshot per congress (M7). + - `members_from_api(payload, congress, states) -> list[MemberRow]`: full state names map to postal codes through `states`. `members_from_yaml(docs, congresses, states)`; `fec_ids_by_bioguide(docs)`; `membership_rows(doc, *, congress, snapshot, sha)`. - CLI: - - `stack market members [--from 110] [--to 119] [--write]` uses the YAML always (keyless, FEC crosswalk) and the Congress.gov API when keyed; without a key it prints the gap and keeps the YAML rows. The reader prefers `congress.gov` rows. - - `stack market committees [--from 2013] [--to ] [--write]`. - - Caches: `data/market/congress/members//.json` and `data/market/legislators/..yaml`. + - `stack market members [--from 110] [--to 119] [--write]` uses Congress.gov (key) plus the legislators YAML (FEC ids, and members of congresses the API lacks). A missing key gives a gap only (M11). It clears `congress:members`. + - `stack market committees [--from 113] [--to 119] [--write]` (keyless) clears `legislators:committees`. - [ ] **Step 1: Failing tests:** - `test_congress_of_year`: 2013 → 113, 2026 → 119, 2027 → 120. - - `test_members_from_api_page`: chamber `house`, state `MI`, district 12. - - `test_members_from_legislators_yaml`. - - `test_fec_ids_by_bioguide`. - - `test_membership_snapshot_rows`: SSAF and HSWM02 in the fixture → only HSWM02 rows, with rank and title kept. - - `test_github_commit_lookup_params`: `until == "2015-03-01T00:00:00Z"`, `path == "committee-membership-current.yaml"`, `per_page == 1`. - - `test_members_without_key_uses_yaml_and_records_gap` (CLI). + - `test_snapshot_cutoff`: 119 → `date(2025,4,1)`; 113 → `date(2013,4,1)`. + - `test_members_from_api_page`: "Michigan", district 12 → state `"MI"`, `district_id == "MI-12"`, chamber `house`. + - `test_at_large_member_district`: Alaska at-large (district None) → `"AK-00"`. + - `test_members_from_legislators_yaml`: a synthetic rep (state `VA`, term start 1789-03-04, `district: 9`) with congresses `[1]` → `district_id == "VA-09"`; a senator → `district_id == ""`. + - `test_fec_ids_by_bioguide`: fixture `id.fec: [H0XX00001]` → `{bioguide: ("H0XX00001",)}`. + - `test_membership_snapshot_rows`: only HSWM02 is kept. + - `test_github_commit_lookup_params`: `until == "2025-04-01T00:00:00Z"` for 119. - `test_roundtrip[congress_member|committee_assignment]`. + - CLI: `test_members_cli_write`, `test_members_cli_no_write_prints_counts`, `test_members_cli_missing_key_gap_only` (no rows, one gap), `test_members_cli_refusal_prints_gap`, `test_committees_cli_write`, `test_committees_cli_refusal_prints_gap`. - [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff. -- [ ] **Step 5: Live:** `ENV stack market members --write`, then `ENV stack market committees --write` (a long run of 14 snapshots at 61 s). Report members per congress (≈ 540) and the HSWM02 and HSIF14 member counts for 2025. -- [ ] **Step 6: Commit** `git add src/market/congress.py src/market/tables.py src/cli/market.py tests/market/test_congress.py tests/market/test_tables.py tests/market/fixtures/congress_members_118.json tests/market/fixtures/legislators_snippet.yaml tests/market/fixtures/membership_snippet.yaml tests/cli/test_market_cli.py && git commit -m "feat(market): members of Congress by district and health-committee assignments over time"` +- [ ] **Step 5: Live:** run `ENV stack market members --write` and `ENV stack market committees --write` (7 snapshots at 61 s). Report members per congress, the at-large count (all `-00`), and the HSWM02/HSIF14 member counts for the 119th Congress. +- [ ] **Step 6: Commit** `git add src/market/congress.py src/market/tables.py src/cli/market.py tests/market/test_congress.py tests/market/test_tables.py tests/market/test_imports.py tests/market/fixtures/congress_members_118.json tests/market/fixtures/legislators_snippet.yaml tests/market/fixtures/membership_snippet.yaml tests/cli/test_market_cli.py && git commit -m "feat(market): members of Congress by canonical district and health-committee assignments per congress"` --- ### Task 7: Bills and actions on Medicare physician payment — `stack market bills` **Files:** -- Modify: `src/market/congress.py`, `src/market/tables.py`, `src/cli/market.py` -- Create: `tests/market/fixtures/congress_bills_page.json`, `congress_bill_detail.json`, `congress_bill_actions.json`, `congress_bill_cosponsors.json` (synthetic) -- Test: `tests/market/test_congress.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py` +- Modify: `src/market/congress.py`, `src/market/tables.py`, `src/cli/market.py`. +- Create: `tests/market/fixtures/congress_bills_page.json`, `tests/market/fixtures/congress_bill_detail.json`, `tests/market/fixtures/congress_bill_actions.json`, `tests/market/fixtures/congress_bill_cosponsors.json` (synthetic). +- Test: `tests/market/test_congress.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`. **Interfaces:** -- Consumes: Task 6 (`CONGRESS_API`, client, pacer). +- Consumes: Task 6. - Produces: - - `BILL_RE = re.compile(r"(?i)\b(physician fee schedule|physician payment|conversion factor|sustainable growth rate|relative value|evaluation and management|telehealth|home-based|house call|care management|primary care)\b")`. A matched title must also have detail `policyArea.name == "Health"` to be kept. + - `BILL_RE = re.compile(r"(?i)\b(physician fee schedule|physician payment|conversion factor|sustainable growth rate|relative value|evaluation and management|telehealth|home-based|house call|care management|primary care)\b")`. A matched bill is kept only when its detail has `policyArea.name == "Health"`. - Rows: - `BillRow(congress, bill_type, number: int, title, introduced: date | None, policy_area, latest_action_date: date | None, latest_action, matched_term, url, item_key)` → `market.bill`; - `BillActionRow(congress, bill_type, number, action_date: date, text, action_code, committee_codes)` → `market.bill_action`; - `BillMemberRow(congress, bill_type, number, bioguide_id, role, since: date | None)` → `market.bill_member`, with role `sponsor` or `cosponsor`. - Helpers: - - `ordinal(n: int) -> str`; - - `bill_url(congress, bill_type, number)` gives `https://www.congress.gov/bill/{ordinal}-congress/{house-bill|senate-bill|house-resolution|senate-resolution|house-joint-resolution|senate-joint-resolution|house-concurrent-resolution|senate-concurrent-resolution}/{n}`; + - `ordinal(n) -> str`; + - `bill_url(congress, bill_type, number)`: `https://www.congress.gov/bill/{ordinal}-congress/{house-bill|senate-bill|house-resolution|senate-resolution|house-joint-resolution|senate-joint-resolution|house-concurrent-resolution|senate-concurrent-resolution}/{n}`; - `match_title(title) -> str | None`. - CLI `stack market bills [--from 110] [--to 119] [--limit N] [--write]`: - - it lists `/bill/{congress}?limit=250&offset=N`, filters locally, then fetches `/bill/{c}/{type}/{n}`, `/actions` and `/cosponsors` for each match; - - one bib Source per kept bill (url = `bill_url`); - - the key is required (gap otherwise). + - it lists `/bill/{congress}?limit=250&offset=N`, filters locally, then reads `/bill/{c}/{type}/{n}`, `/actions` and `/cosponsors` for each match; + - one bib Source per kept bill; + - it clears `congress:bills`. - [ ] **Step 1: Failing tests:** - `test_bill_title_filter`: "Medicare Physician Fee Schedule Update Act" → `"physician fee schedule"`; "Telehealth Modernization Act" → `"telehealth"`; "Farm Bill" → None. - - `test_bill_kept_only_health_policy_area`. + - `test_bill_kept_only_health_policy_area`: a title match with `policyArea.name == "Taxation"` → dropped. - `test_ordinal`: 111 → "111th", 112 → "112th", 121 → "121st", 122 → "122nd", 123 → "123rd". - - `test_bill_url`: (118, "HR", 1234) → `"https://www.congress.gov/bill/118th-congress/house-bill/1234"`. - - `test_actions_and_cosponsors_rows`. - - `test_bills_without_key_records_gap` (CLI). + - `test_bill_url`: `"https://www.congress.gov/bill/118th-congress/house-bill/1234"`. + - `test_actions_and_cosponsors_rows`: fixtures → 3 `BillActionRow` with dates; 1 sponsor + 2 cosponsor `BillMemberRow`. - `test_roundtrip[bill|bill_action|bill_member]`. + - CLI: `test_bills_cli_offline_write`, `test_bills_cli_no_write_lists_matches`, `test_bills_cli_refusal_prints_gap`. - [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff. -- [ ] **Step 5: Live:** with the key, `ENV stack market bills --from 118 --to 119 --write`, then all congresses. Report matched and kept bills per congress and the share sponsored by HSWM02/HSIF14/SSFI10 members. Without the key, report the gap row. -- [ ] **Step 6: Commit** `git add src/market/congress.py src/market/tables.py src/cli/market.py tests/market/test_congress.py tests/market/test_tables.py tests/market/fixtures/congress_bill*.json tests/cli/test_market_cli.py && git commit -m "feat(market): Medicare physician-payment bills with actions, sponsors and cosponsors (Congress.gov)"` +- [ ] **Step 5: Live:** run `ENV stack market bills --from 118 --to 119 --write`, then all congresses. Report matched and kept bills per congress, and the share sponsored by members of HSWM02, HSIF14 or SSFI10. +- [ ] **Step 6: Commit** `git add src/market/congress.py src/market/tables.py src/cli/market.py tests/market/test_congress.py tests/market/test_tables.py tests/market/fixtures/congress_bills_page.json tests/market/fixtures/congress_bill_detail.json tests/market/fixtures/congress_bill_actions.json tests/market/fixtures/congress_bill_cosponsors.json tests/cli/test_market_cli.py && git commit -m "feat(market): Medicare physician-payment bills with actions, sponsors and cosponsors (Congress.gov)"` --- -### Task 8: District crosswalks — `stack market districts` +### Task 8: District crosswalks — `stack market districts`, `stack market hud` **Files:** -- Create: `src/market/districts.py`, `tests/market/fixtures/tab20_cd11920_zcta520_head.txt` (header plus 5 synthetic rows), `hud_zip_cd.json` (synthetic) -- Modify: `src/market/tables.py`, `src/cli/market.py` -- Test: `tests/market/test_districts.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py` +- Create: `src/market/districts.py`, `tests/market/fixtures/tab20_cd11920_zcta520_head.txt` (header plus 5 synthetic rows), `tests/market/fixtures/hud_zip_cd.json` (synthetic). +- Modify: `src/market/tables.py`, `src/cli/market.py`, `tests/market/test_imports.py`. +- Test: `tests/market/test_districts.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`. **Interfaces:** -- Consumes: Task 1. +- Consumes: Task 1 (`state_codes`, `district_id`, `api_key("hud")`). - Produces: - - Census source: `CENSUS_REL = "https://www2.census.gov/geo/docs/maps-data/data/rel2020/cd-sld/tab20_cd{c}20_{geo}_natl.txt"` for `c ∈ (118, 119)` and `geo ∈ ("zcta520", "county20")`. Files are pipe-delimited with a BOM; checked 2026-11-12. - - HUD source: `HUD_API = "https://www.huduser.gov/hudapi/public/usps"` with params `type=5` (zip-cd), `query=All`, `year` and `quarter`, and header `Authorization: Bearer `. The huduser.gov download pages sit behind an AWS WAF challenge (HTTP 202 `x-amzn-waf-action: challenge`, 2026-11-12), so they are never scraped. + - `CENSUS_REL = "https://www2.census.gov/geo/docs/maps-data/data/rel2020/cd-sld/tab20_cd{c}20_{geo}_natl.txt"` for `c ∈ (118, 119)` and `geo ∈ ("zcta520", "county20")`. Files are pipe-delimited with a BOM. + - `HUD_API = "https://www.huduser.gov/hudapi/public/usps"`, with params `type=5` (zip-cd), `query=All`, `year` and `quarter`, and the header `Authorization: Bearer `. The huduser.gov pages sit behind an AWS WAF challenge and are never scraped. - Rows: - - `ZipDistrictRow(congress, zip, state_fips, district, weight, method, vintage, source_file)` → `market.zip_district`, with method ∈ `("census-zcta-land", "hud-res", "hud-bus")`; - - `CountyDistrictRow(congress, county_fips, district, weight, source_file)` → `market.county_district`. + - `ZipDistrictRow(congress, zip, district_id, weight, method, vintage, source_file)` → `market.zip_district`, with method ∈ `("census-zcta-land", "hud-res", "hud-bus")`; + - `CountyDistrictRow(congress, county_fips, district_id, weight, source_file)` → `market.county_district`. - Parsers: - - `parse_census_zcta(text, congress) -> list[ZipDistrictRow]`: - - `weight = AREALAND_PART / AREALAND_ZCTA5_20`; - - rows with a blank ZCTA are skipped; - - `GEOID_CD{c}_20[:2]` is the state and `[2:]` the district (`"00"` at-large); district `"ZZ"` is skipped. - - `parse_census_county(text, congress)`, with weights by `AREALAND_PART` over the county's land area. - - `parse_hud(payload, congress) -> list[ZipDistrictRow]` gives two rows per result, `hud-res` = `res_ratio` and `hud-bus` = `bus_ratio`. - - `zip_weights(con, congress, *, prefer=("hud-bus", "census-zcta-land")) -> dict[str, list[tuple[str, str, float]]]` maps a ZIP to `(state_fips, district, weight)` from the first method that has the ZIP. - - CLI `stack market districts [--congress 118 --congress 119] [--hud-year Y --hud-quarter Q --hud-congress C] [--write]`: - - Census files are cached at `data/market/geo/census/`, each with one bib Download and the file attached (public); - - HUD is cached at `data/market/geo/hud/zip_cd/q.json`; without a token the CLI prints `gap: hud: HUD_API_TOKEN not in .env — key pending (ci@mail.fhirworx.io)`. + - `parse_census_zcta(text, congress, states)`: `weight = AREALAND_PART / AREALAND_ZCTA5_20`. A blank ZCTA is skipped. The GEOID's state FIPS maps to postal and the district number is zero-padded (`"0100"` → `"AL-00"`); `ZZ` is skipped. + - `parse_census_county(text, congress, states)`. + - `parse_hud(payload, congress, states) -> list[ZipDistrictRow]` (two rows per result). + - `zip_weights(con, congress, *, prefer=("hud-bus", "census-zcta-land")) -> dict[str, list[tuple[str, float]]]` maps each ZIP to `(district_id, weight)` pairs. + - CLI: + - `stack market districts [--congress 118 --congress 119] [--write]` (keyless, Census) gives one bib Download per file and clears `census:districts`. + - `stack market hud --year Y --quarter Q --congress C [--write]` is key-gated. A missing token writes only the gap (M11). It clears `hud:hud`. - [ ] **Step 1: Failing tests:** - - `test_census_zcta_weights`: ZCTA 10001 split 3:1 → weights 0.75/0.25; a blank-ZCTA row is skipped; GEOID `"0100"` → state `"01"`, district `"00"`; `"ZZ"` is skipped. - - `test_census_real_header_row`: the real header line plus the row `…|0101|…|36009|…|417245447|…|4938464|0` → weight `pytest.approx(0.0118358727)`. - - `test_hud_rows_two_methods`: res 0.6, bus 0.9 → two rows. - - `test_zip_weights_prefer_hud_bus_then_census`. - - `test_hud_without_token_records_gap` (CLI). + - `test_census_zcta_weights`: 0.75/0.25; the blank ZCTA is skipped; `"0100"` → `"AL-00"`; `ZZ` is skipped. + - `test_census_real_header_row`: weight `approx(0.0118358727)`, `district_id == "AL-01"`. + - `test_hud_rows_two_methods`: result zip 36009, geoid `"0102"`, res 0.6, bus 0.9 → `("36009","AL-02",0.6,"hud-res")` and `(…,0.9,"hud-bus")`. + - `test_zip_weights_prefer_hud_bus_then_census`: a ZIP with both → hud-bus weights; a ZIP with census only → census weights. - `test_roundtrip[zip_district|county_district]`. + - CLI: + - `test_districts_cli_census_offline_write`; + - `test_districts_cli_no_write_prints_counts`; + - `test_districts_cli_refusal_prints_gap`; + - `test_hud_cli_missing_token_writes_only_gap`: a gap row only, no Census rows touched; + - `test_hud_cli_write_from_cache`. - [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff. -- [ ] **Step 5: Live:** run `ENV stack market districts --congress 118 --congress 119 --write`; add the `--hud-*` flags once the token lands, recording which CD vintage HUD returns. Report ZIPs per congress and the share of ZIPs whose weights sum to 1 ± 0.001. -- [ ] **Step 6: Commit** `git add src/market/districts.py src/market/tables.py src/cli/market.py tests/market/test_districts.py tests/market/test_tables.py tests/market/fixtures/tab20_cd11920_zcta520_head.txt tests/market/fixtures/hud_zip_cd.json tests/cli/test_market_cli.py && git commit -m "feat(market): ZCTA/county to congressional district crosswalks (Census rel files; HUD USPS when keyed)"` +- [ ] **Step 5: Live:** run `ENV stack market districts --congress 118 --congress 119 --write`, then `ENV stack market hud --year 2025 --quarter 2 --congress 119 --write` (expected now: the gap `hud:hud`). Report ZIPs per congress and the share of ZIPs whose weights sum to 1 ± 0.001. +- [ ] **Step 6: Commit** `git add src/market/districts.py src/market/tables.py src/cli/market.py tests/market/test_districts.py tests/market/test_tables.py tests/market/test_imports.py tests/market/fixtures/tab20_cd11920_zcta520_head.txt tests/market/fixtures/hud_zip_cd.json tests/cli/test_market_cli.py && git commit -m "feat(market): ZCTA/county to canonical district crosswalks (Census rel files; HUD USPS key-gated)"` --- ### Task 9: District footprint of exposed companies and family billers — `stack market footprint`, `stack market district-view` **Files:** -- Create: `src/market/footprint.py`, `tests/market/fixtures/hha_all_owners_head.csv` (synthetic, 4 rows), `prov_svc_head.csv` (synthetic, 5 rows, real header) -- Modify: `src/market/tables.py`, `src/cli/market.py` -- Test: `tests/market/test_footprint.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py` +- Create: `src/market/footprint.py`, `tests/market/fixtures/hha_all_owners_head.csv` (synthetic, 5 rows), `tests/market/fixtures/prov_svc_head.csv` (synthetic, 5 rows, real header). +- Modify: `src/market/tables.py`, `src/cli/market.py`, `tests/market/test_imports.py`. +- Test: `tests/market/test_footprint.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`. **Interfaces:** - Consumes: - - Tasks 1, 6 and 8; - - `cms.{hospital,home_health_agency,hospice,skilled_nursing_facility,federally_qualified_health_center,rural_health_clinic}_enrollments` (`enrollment_id`, `ccn`, `organization_name`, `doing_business_as_name`, `zip_code`); + - Task 1 (`EntityRow.aliases`, `alias_match(prefix=True)`); + - Task 6 (`market.congress_member`, `market.committee_assignment`); + - Task 8 (`zip_weights`); + - the enrollments tables `cms.{hospital,home_health_agency,hospice,skilled_nursing_facility,federally_qualified_health_center,rural_health_clinic}_enrollments` (`enrollment_id`, `ccn`, `organization_name`, `doing_business_as_name`, `zip_code`); - the hoarded All Owners CSVs `data/public/lead_enrichment/cms__all_owners//*_All_Owners_*.csv` (`ENROLLMENT ID`, `ORGANIZATION NAME - OWNER`, `ROLE TEXT - OWNER`, `PERCENTAGE OWNERSHIP`); - - `terminology.provider` (Tuva 2025-09 NPPES cut; `entity_type_code = '2'`, `provider_organization_name`, `parent_organization_name`, `practice_zip_code`); - - the hoarded `data/public/lead_enrichment/cms_medicare_physician_other_practitioners_by_provider_and_service/*/PHY_R26_P05_V10_D24_Prov_Svc.csv` (data year 2024); - - `pfs.families` codes. + - `terminology.provider` (`entity_type_code = '2'`, `provider_organization_name`, `parent_organization_name`, `practice_zip_code`); + - the hoarded `data/public/lead_enrichment/cms_medicare_physician_other_practitioners_by_provider_and_service/*/PHY_R26_P05_V10_D24_Prov_Svc.csv` (`Rndrng_NPI`, `Rndrng_Prvdr_Zip5`, `HCPCS_Cd`, `Tot_Srvcs`, `Avg_Mdcr_Pymt_Amt`); + - `pfs.families`. - Produces: - - Rows: - - `SiteRow(entity_key, site_kind, site_id, name, zip, state, note, source, source_file)` → `market.entity_site`, with site_kind ∈ `("facility", "org_npi")`; - - `FootprintRow(level, key, congress, state_fips, district, year, n_providers, n_sites, services, payment, method, source)` → `market.footprint`, with level ∈ `("entity", "family")`. - - Site and biller extraction: - - `owner_sites(owners_csv: Path, enrollments: Mapping[str, dict], entities) -> list[SiteRow]` matches the owner organization, or the facility's own name or DBA, with `alias_match(prefix=True)`. One site per (entity, enrollment), with the owner role kept in `note`. - - `npi_sites(rows, entities) -> list[SiteRow]`. - - `family_billers(con, csv: Path, codes) -> list[tuple[str, str, float, float]]` returns `(npi, zip5, services, payment)` with `payment = Tot_Srvcs × Avg_Mdcr_Pymt_Amt`. It is read by DuckDB `read_csv(all_varchar=true)` filtered on `HCPCS_Cd`. - - `rollup(items, weights, *, level, key, congress, year, method) -> list[FootprintRow]`: - - allocation is fractional by weight, so `n_providers` may be fractional; - - a ZIP with no weight goes to `district="??"`, never dropped. - - `@dataclass(frozen=True) DistrictView(state_fips, district, bioguide_id, member, party, health_committees: tuple[str, ...], n_providers, n_sites, services, payment, agi_k: float | None)`. `district_view(con, *, congress, level, key, year) -> list[DistrictView]` joins `market.footprint`, `market.congress_member` (house) and the `market.committee_assignment` snapshot for that congress. `agi_k` is always None here; Task 14 fills it from `econ.soi_zip`. + - `SiteRow(entity_key, site_kind, site_id, name, zip, state, note, source, source_file)` → `market.entity_site`. + - `FootprintRow(level, key, congress, district_id, year, n_providers, n_sites, services, payment, method, source)` → `market.footprint`. + - `owner_sites(owners_csv, enrollments, entities)` and `npi_sites(rows, entities)` match on `EntityRow.aliases` with `alias_match(prefix=True)`. + - `family_billers(con, csv, codes) -> list[tuple[str, str, float, float]]` returns `(npi, zip5, services, payment)` with `payment = Tot_Srvcs × Avg_Mdcr_Pymt_Amt`, read by DuckDB `read_csv(all_varchar=true)` filtered on `HCPCS_Cd`. + - `rollup(items, weights, *, level, key, congress, year, method)`: allocation is fractional, and an unmatched ZIP goes to `"??"`. + - `DistrictView(district_id, bioguide_id, member, party, health_committees, n_providers, n_sites, services, payment, agi_k: float | None)`. `district_view(con, *, congress, level, key, year)` joins on `district_id` with the House members of that congress and that congress's committee snapshot. `agi_k` is None here; Task 14 fills it. - CLI: - - `stack market footprint (--entities | --family KEY) [--congress 119] [--write]`; the cross-section is year 2024, CD119 (planner's call; earlier years and districts are a gap note); + - `stack market footprint (--entities | --family KEY) [--congress 119] [--write]`; - `stack market district-view (--entity KEY | --family KEY) [--congress 119] [--top 20]`. - [ ] **Step 1: Failing tests:** - `test_owner_sites_from_owners_csv`: 3 Amedisys owner rows over 2 enrollments → 2 sites. - - `test_short_alias_exact_only`: the alias `HCA` does not prefix-match `HCA SOMETHING`. - - `test_npi_sites_from_nppes`. - - `test_family_billers_from_prov_svc`: codes `{"99347","99348"}` → 3 of 5 rows; `payment == 12 * 85.5`. - - `test_rollup_fractional_and_unmatched_kept`: ZIP split 0.75/0.25 with services 100 → 75.0/25.0; an unknown ZIP → `"??"` row. - - `test_district_view_joins_member_and_committees`: the member is on HSWM02 → `health_committees == ("HSWM02",)`. + - `test_owner_sites_short_alias_needs_long_form`: an entity with aliases `HCA` and `HCA HEALTHCARE` → owner "HCA HEALTHCARE INC" matches through the long alias; owner "HCA SOMETHING LLC" does not match. + - `test_npi_sites_from_nppes`: two type-2 provider rows, one `parent_organization_name` `AMEDISYS HOLDING LLC` → 1 `org_npi` site with zip5. + - `test_family_billers_from_prov_svc`: `payment == 12 * 85.5`. + - `test_rollup_fractional_and_unmatched_kept`: 75.0/25.0 and `"??"`. + - `test_district_view_joins_member_and_committees`: `district_id "MI-12"` → member and `("HSWM02",)`. - `test_roundtrip[entity_site|footprint]`. - - `test_footprint_cli_family_offline` (CLI). + - CLI: `test_footprint_cli_family_offline_write`, `test_footprint_cli_entities_no_write`, `test_district_view_cli_prints`, `test_footprint_cli_missing_crosswalk_says_run_districts`. - [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff. -- [ ] **Step 5: Live:** run `ENV stack market footprint --entities --write`, then `ENV stack market footprint --family HOME-VISITS --write`, then `ENV stack market district-view --family HOME-VISITS --top 10`. Report the sites per entity (top 10), the share of HOME-VISITS payment that fell to `??`, and the top-10 districts with their members' health committees. -- [ ] **Step 6: Commit** `git add src/market/footprint.py src/market/tables.py src/cli/market.py tests/market/test_footprint.py tests/market/test_tables.py tests/market/fixtures/hha_all_owners_head.csv tests/market/fixtures/prov_svc_head.csv tests/cli/test_market_cli.py && git commit -m "feat(market): district footprint of exposed companies and family billers, joined to members and health committees"` +- [ ] **Step 5: Live:** run `ENV stack market footprint --entities --write`, `ENV stack market footprint --family HOME-VISITS --write` and `ENV stack market district-view --family HOME-VISITS --top 10`. Report the sites per entity (top 10), the `??` share, and the top-10 districts with their members' health committees. +- [ ] **Step 6: Commit** `git add src/market/footprint.py src/market/tables.py src/cli/market.py tests/market/test_footprint.py tests/market/test_tables.py tests/market/test_imports.py tests/market/fixtures/hha_all_owners_head.csv tests/market/fixtures/prov_svc_head.csv tests/cli/test_market_cli.py && git commit -m "feat(market): district footprint of exposed companies and family billers, joined to members and health committees"` --- ### Task 10: Event windows — `stack market events` **Files:** -- Create: `src/market/events.py`, `tests/market/fixtures/fr_pi_2026-22743.json`, `fr_pi_2022-23873.json`, `ama_schedule_snippet.html` (synthetic) -- Modify: `src/market/tables.py`, `src/cli/market.py` -- Test: `tests/market/test_events.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py` +- Create: `src/market/events.py`, `tests/market/fixtures/fr_pi_2026-22743.json`, `tests/market/fixtures/fr_pi_2022-23873.json`, `tests/market/fixtures/ama_schedule_snippet.html` (synthetic). +- Modify: `src/market/tables.py`, `src/cli/market.py`, `tests/market/test_imports.py`. +- Test: `tests/market/test_events.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`. **Interfaces:** - Consumes: - - `pfs.rounds.rule_items(store, [])`, which gives `item_key`, `date_published` and `kind`; - - `fr_anchor_docs` (item_key → document_number); - - `fr_anchors`, `pfs.codetables.read_all_events`, `pfs.codetables.read_seats`; - - `pfs.seats.cdx_snapshots`/`fetch_snapshot`/`html_text`/`is_refusal`; - - Task 1. + - read-only `pfs.*`: `pfs.rounds.rule_items(store, [])` (kinds `proposed`, `final`, `rule`), `fr_anchor_docs`, `fr_anchors`, `pfs.codetables.read_exposures`/`read_all_events`/`read_seats`/`read_specialty_impacts`, `pfs.seats.seats_by_specialty`/`cdx_snapshots`/`fetch_snapshot`/`html_text`/`is_refusal`, `pfs.commenters.specialty_key`; + - Task 1 (`seat_key`). - Produces: - - `FR_PI = "https://www.federalregister.gov/api/v1/public-inspection-documents/{doc}.json"` (`filed_at`, ISO with offset), pacer 0.5 s, cached at `data/market/fr/pi/.json`. A 404 means no public-inspection record (pre-2013 rules). - - `EventRow(event_id, kind, event_date: date, after_close: bool, precision, scope, target, item_key, p_id: int, document_number, filed_at, note)` → `market.event`: + - `FR_PI = "https://www.federalregister.gov/api/v1/public-inspection-documents/{doc}.json"`: + - `filed_at` is ISO with an offset; + - pacer 0.5 s; + - cache `data/market/fr/pi/.json`; + - a 404 means no public-inspection record (rules before 2013). + - `MarketEvent(event_id, kind, event_date: date, after_close: bool, precision, scope, target, item_key, p_id, document_number, filed_at, note)` → `market.event`. The name avoids `pfs.codetables.EventRow`. - kind ∈ `("rule-proposed", "rule-final", "ruc-meeting", "panel-meeting")`; - - precision ∈ `("day", "month")`; - - scope ∈ `("program", "family")`. - - `release(filed_at: str | None, published: date) -> tuple[date, bool, str]`: - - `(local ET date of filed_at, time ≥ 16:00 ET, "")`; - - without `filed_at`: `(published, False, "event date is FR publication; public-inspection time unknown")`. - - `rule_events(rules: Sequence[dict], docs: Mapping[str, str], pi: Mapping[str, dict]) -> list[EventRow]`: - - scope `program`; - - `event_id = f"{kind}:{document_number}"`. - - `family_rule_events(events, fr_events_by_code, codes, family) -> list[EventRow]`: copies with scope `family` for rules whose `item_key` anchors a `pfs.code_event` FR row (`source == "fr"`) of a family code. + - scope ∈ `("program", "family", "specialty")`; + - `target` follows the vocabularies in Decisions. + - `release(filed_at: str | None, published: date) -> tuple[date, bool, str]` returns `(ET date of filed_at, ET time ≥ 16:00, "")`. Without `filed_at` it returns `(published, False, "event date is FR publication; public-inspection time unknown")`. + - `rule_events(rules, docs, pi) -> tuple[list[MarketEvent], int]` returns the program events (target `PFS`, `event_id = f"{kind}:{document_number}"`) and the count of skipped `rule`-kind items. + - `family_rule_events(events, *, exposure_item_keys: set[str], lineage_item_keys: set[str], family) -> list[MarketEvent]`: copies with scope `family`, for rules whose `item_key` anchors a `pfs.code_exposure` row of the family, or an FR `pfs.code_event` row of a family code. + - `rule_year(title: str) -> int | None` (`CY\s?(\d{4})`). + - `specialty_rule_events(events, impacts: Sequence[ImpactRow], titles: Mapping[str, str], specialty: str, *, min_abs: float = 0.01) -> list[MarketEvent]` (M6): + - copies with scope `specialty`; + - the matching impact row has the event's rule year, `vintage` equal to the event's kind (proposed/final), `setting == "TOTAL"`, and `specialty_key(row.specialty) == specialty`; + - `|impact_combined| ≥ min_abs`; + - `note = f"CMS estimated impact {impact_combined:+.0%}"`. - `MEETING_RE` matches `" (RUC|CPT Editorial Panel|CPT Panel) meeting"` and `"(RUC|CPT Editorial Panel|Panel) (at|during|in) (its|their) meeting"`. - - `meeting_mentions(paragraphs: Iterable[tuple[str, int, str]]) -> list[EventRow]`: - - month precision, `event_date` = the 1st of the month; + - `meeting_mentions(paragraphs: Iterable[tuple[str, int, str]]) -> list[MarketEvent]`: + - month precision, with `event_date` the 1st of the month; - `event_id = f"{kind}:{YYYY-MM}"`; + - scope `program`, target `PFS`; - the first cite is kept (lowest item_key, then p_id). - - `ama_schedule(text: str, body: str) -> list[EventRow]`: day precision from `" D–D, YYYY"` ranges, with the start day as `event_date`. - - `@dataclass(frozen=True) Tenure(body, specialty, organization, seat_type, first_year, last_year, left_censored: bool, right_censored: bool)`. `tenures(seats, *, body="RUC") -> list[Tenure]` gives runs of consecutive roster years per specialty; specialty `""` is skipped; censoring is at the min and max roster year. - - `day0(event: EventRow, calendar: Sequence[date]) -> date | None`: the first trading day ≥ `event_date`, or the next one when `after_close`; None for month precision. - - CLI `stack market events [--refresh] [--ama-url URL --body RUC|CPT_PANEL] [--write]`. AMA pages go through the seats Wayback path; a refusal or bot page is a gap. + - `ama_schedule(text, body) -> list[MarketEvent]`: day precision from `" D–D, YYYY"` ranges, with the start day as the event date. + - `Tenure(body, key, holders: tuple[str, ...], seat_types: tuple[str, ...], first_year, last_year, left_censored, right_censored)` (M8). `tenures(seats, *, body="RUC") -> list[Tenure]`: + - per roster year, the held keys are the `seats_by_specialty(seats, y)` keys with permanent + rotating > 0 (RUC), plus the `seat_key` label keys (`PRIMARY CARE`, `ONCOLOGY`) of that year's rows; + - runs of consecutive roster years per key; + - `holders` and `seat_types` are collected across the run. + - `day0(event, calendar: Sequence[date]) -> date | None`: the first trading day ≥ `event_date`, or the next one when `after_close`; None for month precision. + - CLI `stack market events [--refresh] [--ama-url URL --body RUC|CPT_PANEL] [--write]`: + - writes the program, family (every family with exposure or lineage anchors) and specialty (every `specialty_vocab` key plus the seat-label keys) events; + - prints the skipped `rule` count; + - clears `fr:events`, and `ama:events` when `--ama-url` succeeds. - [ ] **Step 1: Failing tests:** - - `test_after_close_filing_moves_day0`: doc `2026-22743`, filed `"2026-11-04T17:00:00.000-05:00"` → `event_date == date(2026,11,4)`, `after_close` True; `day0` on a calendar of weekdays → `date(2026,11,5)`. - - `test_day0_after_close_friday_is_monday`: `after_close` on Friday 2026-11-06 → `day0` is Monday 2026-11-09 on a weekday calendar. - - `test_before_close_same_day`: `"2026-07-14T15:59:00.000-04:00"` → day0 2026-07-14. - - `test_cy2023_final_release_not_publication`: doc `2022-23873`, filed `"2022-11-02T16:15:00.000-04:00"`, published 2022-11-18 → event_date 2022-11-02, after_close True. - - `test_publication_fallback_caveat`: 404 → `event_date` = published and `note == "event date is FR publication; public-inspection time unknown"`. - - `test_rule_events_from_rule_items`. - - `test_family_rule_events`: a HOME-VISITS FR code_event in `PETM225D` → a family event for `rule-final:2022-23873`. - - `test_meeting_mentions_from_fr`: `("5ITGVDJV",193,"…April 2022 RUC meeting…")`, `("N6MTT7VK",584,"…RUC at its January 2021 meeting…")`, `("5ITGVDJV",291,"…February 2025 CPT Editorial Panel meeting…")` → ids `ruc-meeting:2022-04`, `ruc-meeting:2021-01`, `panel-meeting:2025-02`, all month precision. - - `test_meeting_mentions_dedupe_keep_first_cite`. - - `test_ama_schedule_day_dates`: `"April 24–27, 2024"` → `date(2024,4,24)`, day precision. - - `test_tenures_runs_and_censoring`: FAMILY PRACTICE 2019–2026 → 1 tenure, censored on both sides; NEPHROLOGY 2019 and 2021–2022 → 2 tenures. - - `test_day0_month_precision_is_none`. + - `test_after_close_filing_moves_day0`: `2026-22743` filed `"2026-11-04T17:00:00.000-05:00"` → 2026-11-04 after close; `day0` → 2026-11-05. + - `test_day0_after_close_friday_is_monday`: 2026-11-06 → 2026-11-09. + - `test_before_close_same_day`: `"2026-07-14T15:59:00.000-04:00"` → `event_date` 2026-07-14, `after_close` False, day0 2026-07-14. + - `test_cy2023_final_release_not_publication`: `2022-23873` → 2022-11-02, after close. + - `test_publication_fallback_caveat`: 404 → `event_date` = published, `note == "event date is FR publication; public-inspection time unknown"`. + - `test_rule_events_skip_undecidable_kind`: one `rule`-kind item → skipped count 1. + - `test_family_rule_events_from_exposure_and_lineage`: a `PETM225D` exposure anchor → `rule-final:2022-23873` in scope `family`, target `HOME-VISITS`; a lineage-only anchor also qualifies. + - `test_specialty_rule_events_threshold`: FAMILY PRACTICE CY2027 final, combined +0.02 → event, note `"CMS estimated impact +2%"`; ALLERGY/IMMUNOLOGY 0.004 → none; a proposed impact never attaches to a final event. + - `test_meeting_mentions_from_fr`: ids `ruc-meeting:2022-04`, `ruc-meeting:2021-01`, `panel-meeting:2025-02` (`5ITGVDJV` ¶193 and ¶291, `N6MTT7VK` ¶584). + - `test_meeting_mentions_dedupe_keep_first_cite`: two mentions of April 2022 RUC → one event citing the lower (item_key, p_id). + - `test_ama_schedule_day_dates`: `"April 24–27, 2024"` → `event_date == date(2024,4,24)`, precision `day`. + - `test_tenures_span_holder_change` (M8): + - AAFP organization rows 2019–2023 plus "Family Medicine" specialty rows 2024–2026 → one FAMILY PRACTICE tenure 2019–2026, holders `("American Academy of Family Physicians", "Family Medicine")`, censored both sides; + - INTERNAL MEDICINE permanent 2019–2022, then rotating 2023 → one tenure, `seat_types ("permanent", "rotating")`; + - "Primary Care Rotating Seat" 2019–2023 and "Primary Care—additional rotating seat" 2026 → two `PRIMARY CARE` tenures; + - "RUC Chair" and AMA → none. + - `test_day0_month_precision_is_none`: a month-precision event → `day0(...) is None`. - `test_roundtrip[event]`. - - `test_events_cli_offline` (CLI). + - CLI: `test_events_cli_offline_write`, `test_events_cli_no_write_prints_counts`, `test_events_cli_ama_refusal_gap`. - [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff. - [ ] **Step 5: Live:** - - Run `ENV stack market events --write`. Report rule events by kind with and without public-inspection times, meeting events per body, and the tenure count per specialty. - - Print the RUC `(organization, specialty)` pairs and flag the known mislabels `Radiology → RADIATION ONCOLOGY` and `Cardiothoracic Surgery → CARDIOLOGY` (2024 rows) for a seats fix. Do not patch them here. - - Try the AMA RUC and CPT Panel meeting-schedule pages with `--ama-url`, recording the URLs or the gap. -- [ ] **Step 6: Commit** `git add src/market/events.py src/market/tables.py src/cli/market.py tests/market/test_events.py tests/market/test_tables.py tests/market/fixtures/fr_pi_*.json tests/market/fixtures/ama_schedule_snippet.html tests/cli/test_market_cli.py && git commit -m "feat(market): event windows — rule releases at public inspection, RUC/Panel meetings, seat tenures"` + - Run `ENV stack market events --write`. + - Report: + - rule events by kind, with and without public-inspection times, and the skipped `rule` count; + - family events for HOME-VISITS, CCM, TCM and APCM; + - specialty events for FAMILY PRACTICE and INTERNAL MEDICINE; + - meeting events per body; + - every tenure with its holders (FAMILY PRACTICE must be one tenure). + - Try the AMA RUC and CPT Panel schedule pages with `--ama-url`, recording the URLs or the gap. +- [ ] **Step 6: Commit** `git add src/market/events.py src/market/tables.py src/cli/market.py tests/market/test_events.py tests/market/test_tables.py tests/market/test_imports.py tests/market/fixtures/fr_pi_2026-22743.json tests/market/fixtures/fr_pi_2022-23873.json tests/market/fixtures/ama_schedule_snippet.html tests/cli/test_market_cli.py && git commit -m "feat(market): event windows — rule releases at public inspection (program/family/specialty), RUC/Panel meetings, seat tenures"` --- ### Task 11: Abnormal-return engine (market model, CAR, intervals) **Files:** -- Create: `src/market/abnormal.py` -- Modify: `src/pfs/event_study.py` (extract `bootstrap_draws`, `percentile_interval`; `did_effect` uses them) -- Test: `tests/market/test_abnormal.py`, `tests/pfs/test_event_study.py` +- Create: `src/market/abnormal.py`. +- Modify: `src/pfs/event_study.py` (the only `src/pfs` edit this plan makes; H1), `tests/market/test_imports.py` (add `"market.abnormal"`). +- Test: `tests/market/test_abnormal.py`, `tests/pfs/test_event_study.py`. **Interfaces:** -- Consumes: Task 1 only. Inputs are plain mappings, so all tests are synthetic. +- Consumes: Task 1 only. Inputs are plain mappings and the trading calendar; the caller supplies `d0` (Task 12 gets it from `events.day0`). - Produces: - - In `pfs.event_study`: - - `bootstrap_draws(values: Sequence[float], *, n_boot: int = _BOOT, seed: int = _SEED) -> list[float]` returns resampled means in draw order: `random.Random(seed)`, then for each draw `sum(rng.choice(values) for _ in range(n)) / n`. - - `percentile_interval(draws: Sequence[float]) -> tuple[float, float]` = `(sorted[int(0.025·B)], sorted[int(0.975·B) − 1])`. - - `did_effect` builds `[t_change − m for m in bootstrap_draws(changes)]` and returns the identical interval. + - In `pfs.event_study`: `bootstrap_draws(values, *, n_boot=_BOOT, seed=_SEED) -> list[float]` and `percentile_interval(draws) -> tuple[float, float]` (`sorted[int(0.025·B)]`, `sorted[int(0.975·B) − 1]`). `did_effect` uses them with identical output. - In `market.abnormal`: - `MODELS = {"market": ("SPY",), "sector": ("XLV",), "two-factor": ("SPY", "XLV")}`, `EST = (-250, -31)`, `MIN_EST = 120`, `WINDOWS = ((0, 0), (-1, 1), (0, 1), (-5, 5), (0, 10))`. - - `@dataclass(frozen=True) Fit(alpha: float, betas: tuple[float, ...], sigma2: float, n: int, xtx_inv: tuple[tuple[float, ...], ...])`; `fit(y: Sequence[float], xs: Sequence[Sequence[float]]) -> Fit | None` is OLS with an intercept via `numpy.linalg.lstsq`, `sigma2 = SSR/(n − k)`, None when `n ≤ k`. - - `car_from_arrays(f: Fit, y_win, xs_win) -> tuple[float, float]` (car, se), where `AR_t = y_t − α − Σβ x_t` and: - - ``` - Var(CAR) = σ² · (L + ιᵀ X* (XᵀX)⁻¹ X*ᵀ ι) # Campbell–Lo–MacKinlay, X* = [1, x_win] - ``` - - `@dataclass(frozen=True) Car(event_id, unit, ticker, model, window: tuple[int, int], car: float | None, se: float | None, ci_low, ci_high, n_est: int, n_members: int, caveats: tuple[str, ...])`. The CI is `car ± 1.96·se`. - - `event_car(returns: Mapping[date, float], factors: Mapping[str, Mapping[date, float]], calendar: Sequence[date], d0: date, *, model="market", est=EST, window=(-1, 1), min_est=MIN_EST, event_id="", unit="", ticker="", other_events: Sequence[date] = ()) -> Car`. Offsets count trading days in `calendar`. Exact caveat copy: - - `f"short estimation window (n={n})"`, which sets car to None; - - `"missing returns in event window"`, which sets car to None; + - `Fit(alpha, betas: tuple[float, ...], sigma2, n, xtx_inv)`; `fit(y, xs) -> Fit | None` is OLS with an intercept via `numpy.linalg.lstsq` (numpy imported inside), with `sigma2 = SSR/(n − k)` and None when `n ≤ k`. + - `car_from_arrays(f, y_win, xs_win) -> tuple[float, float]` gives `(car, se)`, with `AR_t = y_t − α − Σβx_t` and `Var(CAR) = σ²·(L + ιᵀX*(XᵀX)⁻¹X*ᵀι)`, where `X* = [1, x_win]` (Campbell, Lo and MacKinlay). + - `Car(event_id, unit, ticker, model, window, car, se, ci_low, ci_high, n_est, n_members, caveats)`, with the CI `car ± 1.96·se`. + - `event_car(returns, factors, calendar, d0, *, model="market", est=EST, window=(-1, 1), min_est=MIN_EST, event_id="", unit="", ticker="", other_events=()) -> Car`. Offsets count trading days. The caveats, verbatim: + - `f"short estimation window (n={n})"` → car None; + - `"missing returns in event window"` → car None; - `f"estimation window overlaps event on {d}"`; - `"sector index absorbs sector-wide effects"` for the `sector`/`two-factor` models. - - `@dataclass(frozen=True) Pooled(unit, model, window, mean_car, ci_low, ci_high, n_events, caveats)`; `pooled(cars: Sequence[Car]) -> Pooled` gives the mean of non-None CARs and a bootstrap CI via `bootstrap_draws` + `percentile_interval` (None when n < 2); caveat `"fewer than 5 events"` when n < 5. - - Every Car and Pooled includes the caveat `"association, not a causal estimate"`. + - `Pooled(unit, model, window, mean_car, ci_low, ci_high, n_events, caveats)`; `pooled(cars) -> Pooled` gives the mean of the non-None CARs and a bootstrap CI (None when n < 2), with the caveat `"fewer than 5 events"` when n < 5. + - Every Car and Pooled includes `"association, not a causal estimate"`. - [ ] **Step 1: Failing tests:** - `test_fit_recovers_alpha_beta`: `y = 0.001 + 1.5x` exactly over 130 points → α 0.001, β 1.5, σ² 0 (abs 1e-12). - - `test_car_variance_clm`: fit on x `[-1,0,1]`, y `[-0.9,-0.2,1.1]` → α 0.0, β 1.0, σ² 0.06; window x* `[1,2]`, y* `[1.5,2.0]` → car 0.5, se `pytest.approx(0.6855654600)`. + - `test_car_variance_clm`: fit on x `[-1,0,1]`, y `[-0.9,-0.2,1.1]` → α 0.0, β 1.0, σ² 0.06; window x* `[1,2]`, y* `[1.5,2.0]` → car 0.5, se `approx(0.6855654600)` (`min_est=3`). - `test_car_window_sum`: firm = market + 0.02 on day 0 → CAR(0,0) = CAR(−1,1) = 0.02. - - `test_short_estimation_window_caveat`: n = 100 → car None, caveat `"short estimation window (n=100)"`. - - `test_missing_window_return`. - - `test_offsets_count_trading_days`: with `d0` Monday 2026-11-09 on a weekday calendar, window (−1,1) covers Friday 2026-11-06, Monday and Tuesday 2026-11-10, and the estimation offsets skip weekends. - - `test_two_factor_and_sector_models`: betas of length 2 and 1; the sector caveat is present. - - `test_estimation_overlap_caveat`. - - `test_pooled_bootstrap_deterministic`: two calls give equal CIs; n = 3 → caveat `"fewer than 5 events"`. + - `test_short_estimation_window_caveat`: 100 estimation returns → car None, caveat `"short estimation window (n=100)"`. + - `test_missing_window_return`: a missing firm return inside (−1,1) → car None, caveat `"missing returns in event window"`. + - `test_offsets_count_trading_days`: `d0` Monday 2026-11-09 → (−1,1) covers 2026-11-06, 2026-11-09 and 2026-11-10. + - `test_two_factor_and_sector_models`: betas of length 2 and 1; caveat `"sector index absorbs sector-wide effects"` present for both. + - `test_estimation_overlap_caveat`: another event 40 trading days before d0 → caveat `"estimation window overlaps event on "`. + - `test_pooled_bootstrap_deterministic`: two calls → equal CIs; n = 3 → caveat `"fewer than 5 events"`; n = 1 → CI None. - In `tests/pfs/test_event_study.py`: - - `test_percentile_interval_indices`: draws `range(500)` → `(12, 486)`; + - `test_percentile_interval_indices`: `(12, 486)`; - `test_bootstrap_draws_reproducible`; - - `test_did_effect_interval_unchanged_after_refactor`: before editing, record `(effect, ci_low, ci_high)` of `did_effect` on the module's existing fixture inputs as literals, and assert them after. -- [ ] **Step 2: Run** `tests/market/test_abnormal.py tests/pfs/test_event_study.py` → FAIL. **Step 3: Implement.** **Step 4: Run** `tests/market/test_abnormal.py tests/pfs/test_event_study.py tests/notebooks/test_event_study_nb.py` → PASS; coverage; ruff. -- [ ] **Step 5: Commit** `git add src/market/abnormal.py src/pfs/event_study.py tests/market/test_abnormal.py tests/pfs/test_event_study.py && git commit -m "feat(market): market-model abnormal returns with CLM intervals; shared bootstrap helpers in pfs.event_study"` + - `test_did_effect_interval_unchanged_after_refactor`: literals recorded before the edit. +- [ ] **Step 2: Run** `tests/market/test_abnormal.py tests/pfs/test_event_study.py` → FAIL. **Step 3: Implement.** **Step 4: Run** `tests/market tests/pfs/test_event_study.py tests/notebooks/test_event_study_nb.py` → PASS; coverage; ruff. +- [ ] **Step 5: Commit** `git add src/market/abnormal.py src/pfs/event_study.py tests/market/test_abnormal.py tests/market/test_imports.py tests/pfs/test_event_study.py && git commit -m "feat(market): market-model abnormal returns with CLM intervals; shared bootstrap helpers in pfs.event_study"` --- ### Task 12: Portfolios, stored results, seat-tenure contrast — `stack market car`, `stack market tenure` **Files:** -- Create: `src/market/portfolio.py` -- Modify: `src/market/tables.py`, `src/cli/market.py` -- Test: `tests/market/test_portfolio.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py` +- Create: `src/market/portfolio.py`. +- Modify: `src/market/tables.py`, `src/cli/market.py`, `tests/market/test_imports.py`. +- Test: `tests/market/test_portfolio.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`. **Interfaces:** -- Consumes: Task 1 (links, securities, filings: 8-K items), Task 2 (fts links), Task 3 (`read_returns`, `trading_days`), Task 10 (`EventRow`, `day0`, `tenures`), Task 11. +- Consumes: Task 1 (links, securities, `EntityRow.ended_on`, `FilingRow`), Task 3 (`read_returns`, `trading_days`), Task 10 (`MarketEvent`, `day0`, `tenures`), Task 11. - Produces: - - `@dataclass(frozen=True) Member(entity_key, ticker)`. - - `members(con, *, scope, target, on: date, strict=False) -> tuple[list[Member], list[str]]` returns the members and the linked entity keys with no usable price series (the gap): - - a member is linked by scope/target, has a security valid on `on`, and is not ended before `on`; - - `strict` keeps only `curated` and `revenue_exposure` strengths. + - `Member(entity_key, ticker)`. + - `members(con, *, scope, target, on: date, strict=False) -> tuple[list[Member], list[str]]` returns the members and the gap: the linked entity keys with no usable price series. + - A member is linked by scope and target, has a security valid on `on`, and has not ended before `on`. + - `strict` keeps only the `curated` and `revenue_exposure` strengths. + - Entities with only `sic` links are neither members nor gaps. - `portfolio_returns(returns: Mapping[str, Mapping[date, float]], members) -> dict[date, float]`: equal weight across the members that have a return that day. - - `earnings_days(filings: Sequence[FilingRow]) -> set[date]`: 8-K filings whose items include `2.02`, dated by `accepted` (UTC → ET; at or after 16:00 counts as the next day). - - `CarRow(event_id, scope, target, unit, ticker, model, win_lo, win_hi, car, se, ci_low, ci_high, n_est, n_members, n_gap, caveats, computed_on: date)` → `market.abnormal_return`, with caveats as a JSON list. `write_cars(con, scope, target, model, rows)` and `read_cars(con, scope, target, model=None) -> list[CarRow]`. - - `run_events(con, events, *, scope, target, model, windows, strict) -> list[CarRow]`: - - for each day-precision event, one row per member ticker (`unit="firm"`), plus one portfolio row (`unit=f"portfolio:{scope}:{target}"`) whose market model runs on the portfolio series; - - plus one pooled row per window (`event_id="pooled"`); - - extra caveat `"earnings release (8-K Item 2.02) in window"` when a member's earnings day falls in the window. - - `year_bhar(port: Mapping[date, float], mkt: Mapping[date, float], year: int, *, min_days=200) -> float | None` = `Π(1+r_p) − Π(1+r_m)` over that year's common trading days. - - `@dataclass(frozen=True) TenureContrast(specialty, mean_on, mean_off, diff, ci_low, ci_high, n_on, n_off, caveats)`; `contrast(rows: Sequence[TenureRow]) -> TenureContrast`: + - `earnings_days(filings: Sequence[FilingRow]) -> set[date]`: 8-K filings whose items include `2.02`, dated by `accepted` converted from UTC to ET; at or after 16:00 counts as the next day. + - `CarRow(event_id, scope, target, unit, ticker, model, win_lo, win_hi, car, se, ci_low, ci_high, n_est, n_members, n_gap, caveats, computed_on: date)` → `market.abnormal_return`, with caveats as a JSON list. `write_cars(con, scope, target, model, rows)` replaces those rows; `read_cars(con, scope, target, model=None) -> list[CarRow]`. + - `run_events(con, events, *, scope, target, model, windows, strict) -> list[CarRow]` emits: + - for each day-precision event, one row per member ticker (`unit="firm"`); + - one portfolio row (`unit=f"portfolio:{scope}:{target}"`), whose market model runs on the portfolio series; + - one pooled row per window (`event_id="pooled"`); + - the extra caveat `"earnings release (8-K Item 2.02) in window"` when a member's earnings day falls in the window. + + It takes: + - for `--family K`, the family events and family-linked members; + - for `--specialty S`, the specialty events and specialty-linked members; + - for `--program P`, the program events (target `PFS`) and the members linked to `P`. + - `year_bhar(port, mkt, year, *, min_days=200) -> float | None` = `Π(1+r_p) − Π(1+r_m)` over that year's common trading days. + - `TenureRow(key, year, on_ruc, bhar, n_members)` → `market.tenure_return`, where `key` is a tenure key and `on_ruc` is true when that calendar year's roster holds the key (Decisions). + - `TenureContrast(key, mean_on, mean_off, diff, ci_low, ci_high, n_on, n_off, caveats)`; `contrast(rows) -> TenureContrast`: - the CI pairs `bootstrap_draws(on, seed=_SEED)` with `bootstrap_draws(off, seed=_SEED + 1)`; - None when either group has fewer than 2 years; - - caveats always include `"association, not a causal estimate"` and `"roster years 2019–2026 only"`. - - `TenureRow(specialty, year, on_ruc: bool, bhar: float | None, n_members)` → `market.tenure_return`. + - the caveats always include `"association, not a causal estimate"` and `"roster years 2019–2026 only"`. - CLI: - - `stack market car (--family KEY | --specialty S | --program) [--kind rule-final --kind rule-proposed …] [--model market] [--window -1,1 …] [--strict] [--write]` prints the portfolio and pooled rows; the defaults are `model=market` and all `WINDOWS`; - - `stack market tenure (--specialty S | --all) [--write]`. + - `stack market car (--family KEY | --specialty S | --program P) [--kind …] [--model market] [--window -1,1 …] [--strict] [--write]`; + - `stack market tenure (--key K | --all) [--write]`. - [ ] **Step 1: Failing tests:** - - `test_members_as_of_day0_include_later_delisted`: OSH linked to HOME-VISITS is a member on 2022-11-03 but not on 2024-11-05. - - `test_no_price_series_counted_as_gap`. - - `test_equal_weight_daily_rebalanced`: returns 0.1 and 0.0 → 0.05; when one is missing → the other's return. - - `test_strict_members_only_curated_or_revenue_exposure`. - - `test_earnings_in_window_caveat`: 8-K 2.02 accepted `2022-11-03T13:00:00.000Z` → 2022-11-03, inside (−1,1) around 2022-11-03. - - `test_month_precision_event_skipped`. - - `test_year_bhar`: `[0.1, −0.1]` vs `[0, 0]` with `min_days=2` → `pytest.approx(-0.01)`. + - `test_members_as_of_day0_include_later_delisted`: OSH linked to HOME-VISITS → member on 2022-11-03, not on 2024-11-05. + - `test_no_price_series_counted_as_gap`: a linked entity with no security → returned in the gap list, not a member. + - `test_sic_only_entities_never_members`: an entity whose only link is `sic` → neither member nor gap. + - `test_equal_weight_daily_rebalanced`: rets 0.1 and 0.0 → 0.05; one missing → the other's. + - `test_strict_members_only_curated_or_revenue_exposure`: `strict=True` keeps `curated` and `revenue_exposure`, drops `mention` and `risk_factor`. + - `test_earnings_in_window_caveat`: 8-K 2.02 accepted `2022-11-03T13:00:00.000Z` → 2022-11-03 inside (−1,1) around 2022-11-03 → caveat `"earnings release (8-K Item 2.02) in window"`. + - `test_month_precision_event_skipped`: a month-precision event → no CarRow. + - `test_specialty_run_uses_specialty_events`: `--specialty "FAMILY PRACTICE"` → only events with scope `specialty`, target `FAMILY PRACTICE`; members linked to that specialty. + - `test_program_run_uses_pfs_events_and_program_members`: `--program MSSP` → events with target `PFS`; members linked to `MSSP`. + - `test_year_bhar`: `[0.1, −0.1]` vs `[0, 0]` with `min_days=2` → `approx(-0.01)`. - `test_tenure_contrast`: on `[0.1, 0.3]`, off `[0.0, 0.2]` → diff 0.1, n_on 2, n_off 2; off `[0.0]` alone → CI None. - `test_roundtrip[abnormal_return|tenure_return]`. - - `test_car_cli_writes_rows` (CLI): synthetic SPY and 2 tickers in `:memory:` → firm, portfolio and pooled rows, publish once. - - `test_tenure_cli_prints_contrast` (CLI). + - CLI: `test_car_cli_writes_rows`, `test_car_cli_no_write_prints`, `test_car_cli_no_prices_says_run_prices`, `test_tenure_cli_prints_contrast`, `test_tenure_cli_write`. - [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff. - [ ] **Step 5: Live:** - - Run `ENV stack market car --family HOME-VISITS --kind rule-final --kind rule-proposed --write`, then `ENV stack market car --program --write`, then `ENV stack market tenure --all --write`. - - Report per family: events, members per event (and the gap count), and caveat frequencies. Give the CY2023 final (`rule-final:2022-23873`, day 0 2022-11-03) portfolio CAR(−1,1) with its CI, internally only. -- [ ] **Step 6: Commit** `git add src/market/portfolio.py src/market/tables.py src/cli/market.py tests/market/test_portfolio.py tests/market/test_tables.py tests/cli/test_market_cli.py && git commit -m "feat(market): family/specialty/program portfolios, stored CARs with caveats, RUC-tenure return contrast"` + - Run `ENV stack market car --family HOME-VISITS --kind rule-final --kind rule-proposed --write`, `ENV stack market car --program PFS --write`, `ENV stack market car --specialty "FAMILY PRACTICE" --write` and `ENV stack market tenure --all --write`. + - Report per run: events, members per event and the gap, and the caveat frequencies. Give the CY2023 final (`rule-final:2022-23873`, day 0 2022-11-03) portfolio CAR(−1,1) with its CI. Derived analytics may be shown inside the stack (D5). +- [ ] **Step 6: Commit** `git add src/market/portfolio.py src/market/tables.py src/cli/market.py tests/market/test_portfolio.py tests/market/test_tables.py tests/market/test_imports.py tests/cli/test_market_cli.py && git commit -m "feat(market): family/specialty/program portfolios, stored CARs with caveats, RUC-tenure return contrast"` --- ### Task 13: FRED, IRS SOI and company revenue — `stack market fred`, `stack market soi`, `stack market revenue` **Files:** -- Modify: `src/econ/fred.py`, `src/econ/irs_soi.py`, `src/market/edgar.py`, `src/market/tables.py`, `src/cli/market.py` -- Create: `tests/market/fixtures/companyfacts_snippet.json` (synthetic) -- Test: `tests/econ/test_fred_irs.py`, `tests/market/test_enrich_sources.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py` +- Modify: `src/econ/fred.py`, `src/econ/irs_soi.py`, `src/market/edgar.py`, `src/market/tables.py`, `src/cli/market.py`. +- Create: `tests/market/fixtures/companyfacts_snippet.json` (synthetic; a 10-K for FY2023 carrying its FY2022 comparative). +- Test: `tests/econ/test_fred_irs.py`, `tests/market/test_enrich_sources.py`, `tests/market/test_tables.py`, `tests/cli/test_market_cli.py`. **Interfaces:** -- Consumes: Task 1; the existing `econ.fred.SeriesRow`. +- Consumes: Task 1; `econ.fred.SeriesRow`. - Produces: - `econ.fred`: - `FREDGRAPH = "https://fred.stlouisfed.org/graph/fredgraph.csv?id={series}"` (keyless; probed 2026-11-12); - - `parse_fredgraph_csv(text: str, series_id: str) -> tuple[SeriesRow, ...]` accepts the header `observation_date,` or `DATE,`, treats `.` or empty as None, and raises `ValueError` on any other header (an HTML page returned with status 200 for an unknown id). - - `market.tables.FRED_SERIES = ("CPIAUCSL", "CPIMEDSL", "CUSR0000SAM2", "PCU621111621111", "PCU621610621610", "VIXCLS", "USREC", "DTB3")`. These are the CPI (all items, medical care, medical care services), the PPI for physician offices and for home health, VIX, recessions and the 3-month T-bill. All were verified 2026-11-12. Table `econ.fred(series_id, year, period, value, item_key)`. + - `parse_fredgraph_csv(text, series_id) -> tuple[SeriesRow, ...]` accepts the header `observation_date,` or `DATE,`, treats `.` or empty as None, and raises `ValueError` on any other header (an HTML page returned with status 200 for an unknown id). - `econ.irs_soi`: - `SOI_ZIP = "https://www.irs.gov/pub/irs-soi/{yy}zpallagi.csv"`; - `ZipIncome(year, zip, state_fips, n_returns: int | None, agi_k: int | None)`; - - `parse_soi_zip(csv_text: str, year: int) -> tuple[ZipIncome, ...]` sums `N1` and `A00100` over `agi_stub`, drops zipcodes `00000` and `99999`, and zero-pads to 5 digits. Table `econ.soi_zip`. + - `parse_soi_zip(csv_text, year) -> tuple[ZipIncome, ...]` sums `N1` and `A00100` over `agi_stub`, drops zipcodes `00000` and `99999`, and zero-pads to 5 digits. + - In `market.tables` (M9, W9; every row carries its citation): + - `FRED_SERIES = ("CPIAUCSL", "CPIMEDSL", "CUSR0000SAM2", "PCU621111621111", "PCU621610621610", "VIXCLS", "USREC", "DTB3")`, all verified 2026-11-12; + - `FredRow(series_id, year, period, value, item_key)` → `econ.fred`; + - `SoiZipRow(year, zip, state_fips, n_returns, agi_k, item_key)` → `econ.soi_zip`; + - `RevenueRow(cik, period_start: date, period_end: date, frame, form, concept, value, filed: date, adsh, item_key)` → `market.revenue`. - `market.edgar`: - - `COMPANYFACTS = "https://data.sec.gov/api/xbrl/companyfacts/CIK{cik10}.json"`; - - `RevenueRow(cik, fy: int, fp, form, concept, value: float, filed: date, adsh)` → `market.revenue`; - - `revenue_rows(facts) -> list[RevenueRow]` uses `us-gaap:Revenues`, falling back to `RevenueFromContractWithCustomerExcludingAssessedTax`, `fp == "FY"` and 10-K forms only, keeping the latest `filed` per fy. + - `COMPANYFACTS`; + - `revenue_rows(facts, *, item_key) -> list[RevenueRow]` (D1): + - concept `us-gaap:Revenues`, falling back to `RevenueFromContractWithCustomerExcludingAssessedTax`; + - annual durations only (`end − start` between 350 and 380 days), forms `10-K` and `10-K/A`; + - keyed on **`end`**, with `frame` kept when present; the latest `filed` per `end` wins; + - the filing's `fy` is never used as the year. - CLI: - - `stack market fred [--series ID …] [--write]`: cache `data/market/econ/fred/.csv`, one bib Download per series with the file attached; - - `stack market soi [--year Y …] [--write]`: defaults to 2013 through the newest year whose URL returns CSV; cache `data/market/econ/soi/zpallagi.csv`; - - `stack market revenue [--linked] [--write]`. + - `stack market fred [--series ID …] [--write]` clears `fred:fred`; + - `stack market soi [--year Y …] [--write]` clears `irs-soi:soi`; + - `stack market revenue [--entity KEY …] [--write]` (default: entities in the price universe) clears `sec-edgar:revenue`. + - Each gets one bib item per source file and passes its key into the rows. - [ ] **Step 1: Failing tests:** - `test_parse_fredgraph_csv`: `"observation_date,CPIMEDSL\n1947-01-01,13.200\n2026-10-01,.\n"` → `(SeriesRow("CPIMEDSL",1947,"1947-01-01",13.2), SeriesRow("CPIMEDSL",2026,"2026-10-01",None))`. - - `test_fredgraph_old_header`. - - `test_fredgraph_html_rejected`. - - `test_soi_zip_sums_agi_stubs`: zip 1001 with stubs 1–6 → `"01001"` and exact sums; `00000`/`99999` dropped. - - `test_companyfacts_revenue`: duplicate FY 2023 rows → the latest filed is kept; 10-Q rows dropped. - - `test_roundtrip[fred|soi_zip|revenue]`. - - `test_fred_cli_offline` and `test_soi_cli_offline` (CLI). + - `test_fredgraph_old_header`: `"DATE,CPIMEDSL\n…"` parses the same. + - `test_fredgraph_html_rejected`: `"…"` → `ValueError`. + - `test_soi_zip_sums_agi_stubs`: zip 1001 with stubs 1–6 → `"01001"`, `n_returns` and `agi_k` the exact sums; `00000` and `99999` dropped. + - `test_companyfacts_revenue_keys_period_end` (D1): the FY2023 10-K's comparative (fy 2023, end 2022-12-31) is stored under `period_end == date(2022,12,31)`, never 2023. A later 10-K restating the 2022-12-31 value wins. 10-Q and quarterly durations are dropped. + - `test_roundtrip[fred|soi_zip|revenue]`, with item_key round-tripped. + - CLI: `test_fred_cli_offline_write`, `test_fred_cli_refusal_prints_gap`, `test_soi_cli_offline_write`, `test_soi_cli_no_write`, `test_revenue_cli_offline_write`, `test_revenue_cli_refusal_prints_gap`. - [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; run the coverage gate with `--cov=src/econ` added; ruff. -- [ ] **Step 5: Live:** run `ENV stack market fred --write`, `ENV stack market soi --write` and `ENV stack market revenue --linked --write`. Report rows per series, the SOI years loaded, and the revenue coverage of linked companies. -- [ ] **Step 6: Commit** `git add src/econ/fred.py src/econ/irs_soi.py src/market/edgar.py src/market/tables.py src/cli/market.py tests/econ/test_fred_irs.py tests/market/test_enrich_sources.py tests/market/test_tables.py tests/market/fixtures/companyfacts_snippet.json tests/cli/test_market_cli.py && git commit -m "feat(market): FRED (fredgraph), IRS SOI ZIP income and EDGAR company revenue for enrichment"` +- [ ] **Step 5: Live:** run `ENV stack market fred --write`, `ENV stack market soi --write` and `ENV stack market revenue --write`. Report rows per series, the SOI years loaded, and the revenue coverage per year. +- [ ] **Step 6: Commit** `git add src/econ/fred.py src/econ/irs_soi.py src/market/edgar.py src/market/tables.py src/cli/market.py tests/econ/test_fred_irs.py tests/market/test_enrich_sources.py tests/market/test_tables.py tests/market/fixtures/companyfacts_snippet.json tests/cli/test_market_cli.py && git commit -m "feat(market): FRED (fredgraph), IRS SOI ZIP income and EDGAR revenue by period end, each cited"` --- ### Task 14: Enrichment joins (readers for the joint displays) — `stack market joint` **Files:** -- Create: `src/market/enrich.py` -- Modify: `src/market/footprint.py` (`district_view` fills `agi_k` from `econ.soi_zip`), `src/cli/market.py` -- Test: `tests/market/test_enrich.py`, `tests/market/test_footprint.py`, `tests/cli/test_market_cli.py` +- Create: `src/market/enrich.py`. +- Modify: `src/market/footprint.py` (`district_view` fills `agi_k`), `src/cli/market.py`, `tests/market/test_imports.py`. +- Test: `tests/market/test_enrich.py`, `tests/market/test_footprint.py`, `tests/cli/test_market_cli.py`. **Interfaces:** -- Consumes: Task 3 (returns, `market.series`), Task 12 (`portfolio_returns`, `year_bhar`), Task 13 (`econ.fred`, `econ.soi_zip`), `bls.oews`, `bls.ces`, `pfs.utilization`, `pfs.event_study.family_series`, `pfs.psps.read_by_specialty`. +- Consumes: + - Task 3 (returns, `market.series`); + - Task 8 (`zip_weights`, to roll SOI ZIPs to districts); + - Task 9 (`district_view`, `FootprintRow`); + - Task 12 (`portfolio_returns`, `year_bhar`, `members`); + - Task 13 (`FredRow`, `SoiZipRow`); + - `bls.oews`, `bls.ces`; + - `pfs.event_study.family_series`, `pfs.psps.read_by_specialty`, `pfs.commenters.specialty_key`, `pfs.families`. - Produces: - - `SPECIALTY_SOC: dict[str, tuple[str, ...]]`: FAMILY PRACTICE 29-1215, INTERNAL MEDICINE 29-1216, PEDIATRIC MEDICINE 29-1221, PSYCHIATRY 29-1223, ANESTHESIOLOGY 29-1211, CARDIOLOGY 29-1212, DERMATOLOGY 29-1213, EMERGENCY MEDICINE 29-1214, NEUROLOGY 29-1217, OBSTETRICS GYNECOLOGY 29-1218, PATHOLOGY 29-1222, DIAGNOSTIC RADIOLOGY 29-1224, OPHTHALMOLOGY 29-1241, ORTHOPEDIC SURGERY 29-1242, GENERAL SURGERY 29-1249, PHYSICAL THERAPIST 29-1123, NURSE PRACTITIONER 29-1171, PHYSICIAN ASSISTANT 29-1071. All are present in `bls.oews`; the specialist codes start in 2019 or 2021. + - `SPECIALTY_SOC: dict[str, tuple[str, ...]]` (OEWS codes): + + | Specialty | SOC | Specialty | SOC | + |---|---|---|---| + | FAMILY PRACTICE | 29-1215 | NEUROLOGY | 29-1217 | + | INTERNAL MEDICINE | 29-1216 | OBSTETRICS GYNECOLOGY | 29-1218 | + | PEDIATRIC MEDICINE | 29-1221 | PATHOLOGY | 29-1222 | + | PSYCHIATRY | 29-1223 | DIAGNOSTIC RADIOLOGY | 29-1224 | + | ANESTHESIOLOGY | 29-1211 | OPHTHALMOLOGY | 29-1241 | + | CARDIOLOGY | 29-1212 | ORTHOPEDIC SURGERY | 29-1242 | + | DERMATOLOGY | 29-1213 | GENERAL SURGERY | 29-1249 | + | EMERGENCY MEDICINE | 29-1214 | PHYSICAL THERAPIST | 29-1123 | + | NURSE PRACTITIONER | 29-1171 | PHYSICIAN ASSISTANT | 29-1071 | + - `FAMILY_INDUSTRY: dict[str, tuple[str, ...]] = {"HOME-VISITS": ("621100", "621600")}`, default `("621100",)` (CES). - - `@dataclass(frozen=True) JointYear(year: int, values: dict[str, float | None], sources: dict[str, str])`. Column keys are exactly: + - `JointYear(year: int, values: dict[str, float | None], sources: dict[str, str])`. The column keys are exactly: - `market:::bhar`; - `util::n_services`; - `psps::allowed`; - `oews::a_median`; - - `ces::employment` (annual mean); - - `fred:` (annual mean); - - `sources` maps each key to its table plus the item_key. - - `joint_annual(con, *, family: str | None = None, specialty: str | None = None, years: Sequence[int]) -> list[JointYear]`. A missing input is None, never 0. - - `JointMonth(year, month, values, sources)`; `joint_monthly(con, *, symbols, ces, fred, start: date, end: date) -> list[JointMonth]`, where `market::tri` comes from `market.series`. - - `annual_tri(con, symbol) -> dict[int, float]`, the mean monthly TRI per year (the #797 notebook input). - - CLI `stack market joint (--family KEY | --specialty S) [--from 2015] [--to 2024] [--monthly]` prints the table (internal). + - `ces::employment` (the annual mean); + - `fred:` (the annual mean). + - `joint_annual(con, *, family=None, specialty=None, years)`: + - `psps::allowed` needs the family's codes and matches `specialty_key(PspsRow.specialty) == specialty_key(S)` (M9); + - with `specialty` alone, the `psps:*` columns are absent and `sources["psps"] == "needs a family"`; + - a missing input is None, never 0; + - `sources` maps every column to its table plus `item_key`. + - `JointMonth`; `joint_monthly`; `annual_tri(con, symbol)`. + - `district_view` gains `agi_k`: `econ.soi_zip` rolled to `district_id` by `zip_weights`. + - CLI `stack market joint (--family KEY | --specialty S) [--from 2015] [--to 2024] [--monthly]`. - [ ] **Step 1: Failing tests:** - - `test_joint_annual_columns_and_sources`: in-memory tables give exact values for each column key. - - `test_joint_missing_inputs_are_none_not_zero`. - - `test_joint_monthly_alignment`. - - `test_annual_tri`. - - `test_district_view_agi_from_soi` (in `test_footprint.py`). - - `test_specialty_soc_codes_in_oews`: live, skipped without the replica. - - `test_joint_cli_prints` (CLI). + - `test_joint_annual_columns_and_sources`: in-memory utilization, PSPS, OEWS 29-1215, CES 621100, FRED CPIMEDSL and market rows → each column key's exact value; `sources[key]` = `":"`. + - `test_joint_psps_uses_codes_and_specialty_key`: PSPS rows "Family Practice" and "Obstetrics/Gynecology" → `psps:FAMILY PRACTICE:allowed` summed over the family codes only. + - `test_joint_specialty_only_has_no_psps`: `specialty="FAMILY PRACTICE"`, no family → no `psps:*` key; `sources["psps"] == "needs a family"`. + - `test_joint_missing_inputs_are_none_not_zero`: no FRED rows for a year → `values["fred:CPIMEDSL"] is None`. + - `test_joint_monthly_alignment`: CES month 2024-03, FRED 2024-03-01 and `market:XLV:tri` (2024,3) → one `JointMonth(2024, 3, …)` with all three. + - `test_annual_tri`: monthly values 100, 110 in 2024 → `{2024: 105.0}`. + - `test_specialty_soc_keys_are_specialty_keys`: every key equals `specialty_key(key)`, and every value matches `^\d{2}-\d{4}$`. + - `test_district_view_agi_from_soi` (in `test_footprint.py`): SOI ZIP `48201` with `agi_k` 1000, weighted 0.75 to `MI-12` and 0.25 to `MI-13` → `agi_k == 750.0` for `MI-12`. + - CLI: `test_joint_cli_prints`, `test_joint_cli_monthly`, `test_joint_cli_requires_family_or_specialty`. - [ ] **Step 2: Run** → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS; coverage; ruff. -- [ ] **Step 5: Live:** run `ENV stack market joint --family HOME-VISITS --from 2015 --to 2024` and `ENV stack market joint --specialty "FAMILY PRACTICE"`. Report which columns are populated by year. -- [ ] **Step 6: Commit** `git add src/market/enrich.py src/market/footprint.py src/cli/market.py tests/market/test_enrich.py tests/market/test_footprint.py tests/cli/test_market_cli.py && git commit -m "feat(market): joint readers — market series beside BLS, FRED, SOI, PSPS and utilization"` +- [ ] **Step 5: Live:** + - Run `ENV stack market joint --family HOME-VISITS --from 2015 --to 2024` and `ENV stack market joint --specialty "FAMILY PRACTICE"`, and report the populated columns by year. + - Check in the replica that every `SPECIALTY_SOC` code is in `bls.oews` and every key in `specialty_vocab`, and report any miss. This was a live-only pytest before; it is now a verification step (D4). +- [ ] **Step 6: Commit** `git add src/market/enrich.py src/market/footprint.py src/cli/market.py tests/market/test_enrich.py tests/market/test_footprint.py tests/market/test_imports.py tests/cli/test_market_cli.py && git commit -m "feat(market): joint readers — market series beside BLS, FRED, SOI, PSPS and utilization"` --- ### Task 15: #797 notebook market panel, acceptance, roll-out **Files:** -- Modify: `notebooks/event_study.py` (a new section "Market series (#797)", placed after the wages/employment section) -- Test: `tests/notebooks/test_event_study_nb.py` +- Modify: `notebooks/event_study.py` (a new section "Market series (#797)", after the wages/employment section). +- Test: `tests/notebooks/test_event_study_nb.py`. **Interfaces:** -- Consumes: `market.enrich.annual_tri` (Task 14), `market.tables.read_series` (Task 3), `market.tables.read_cars` (Task 12), and `pfs.event_study.pre_post`. +- Consumes: `market.enrich.annual_tri` (Task 14), `market.tables.read_series` (Task 3), `market.tables.read_cars` (Task 12), `pfs.event_study.pre_post`. - Produces: - - For the selected exposure, `pre_post` runs on `annual_tri` for XLV, IHF and SPY and on the family portfolio's annual BHAR. There is no control set, and the panel says so in the same words as the wages panel. - - Below it sits a table of stored `market.abnormal_return` portfolio rows for that family's rule events, with their caveats. - - Each panel gets a provenance line built from the bib keys. + - For the selected exposure, `pre_post` runs on `annual_tri` for XLV, IHF and SPY and on the family portfolio's annual BHAR, with "no control set" stated as in the wages panel. + - Below it sits a table of the family's stored portfolio CARs with their caveats, plus a provenance line built from the bib keys. - When the market tables are absent, `not_built("stack market prices --write")`. - - No causal wording: the existing `test_no_causal_language` must still pass. + - No causal wording. - [ ] **Step 1: Failing tests:** - - `test_market_panel_headless_not_built`: with no replica, the output contains `stack market prices --write`. - - `test_market_panel_live_ccm_2015_apcm_2025`: `_live`; the fixtures `("99490", 2015, "becomes-payable")` and `("G0556", 2025, "becomes-payable")` render a market table with an XLV row and provenance. + - `test_market_panel_headless_not_built`: no replica → output contains `stack market prices --write`. + - `test_market_panel_renders_from_memory_tables`: offline. In-memory synthetic `market.series` and `market.abnormal_return` rows for CCM 2015 → a table with an XLV row, the caveat column and a provenance line. + - `test_market_panel_live_ccm_2015_apcm_2025`: `_live`. The nightly notebooks-integration run executes it, which is #797's done-when. + - `test_no_causal_language` still passes. - [ ] **Step 2: Run** `tests/notebooks/test_event_study_nb.py` → FAIL. **Step 3: Implement.** **Step 4: Run** → PASS. -- [ ] **Step 5: Full gate:** `uv run --no-sync pytest tests/ -x --cov=src --cov-report=term-missing --cov-fail-under=99 -q -n auto -p no:cacheprovider` → PASS; ruff on all touched paths. +- [ ] **Step 5: Full gate:** `uv run --no-sync pytest tests/ -x --cov=src --cov-report=term-missing --cov-fail-under=99 -q -n auto -p no:cacheprovider` → PASS; ruff on the touched paths. - [ ] **Step 6: Commit** `git add notebooks/event_study.py tests/notebooks/test_event_study_nb.py && git commit -m "feat(notebooks): market series panel in the event-study notebook (#797)"` - [ ] **Step 7: Acceptance (live):** - 1. Confirm `ENV stack market gaps` lists only the pending-key sources (fec, congress bills, hud, and lda only if a key was expected). - 2. Confirm, in the replica, that every `market.entity_link` row has a non-empty `cite_ref` or `item_key`, and that no `market.*` column contains a key value. Check with `grep -c` of the key values over the replica's `cms.ingest_log`, `data/bib.sqlite` URLs and `data/market` file names; each must be 0. - 3. Confirm that no file under `data/market/tiingo` is attached in bib. - 4. Confirm that `market.price_daily` covers SPY from 1993-01-29. - 5. Confirm that HOME-VISITS rule events have stored portfolio CARs. - 6. Push, and let the nightly `notebooks-integration` run render the panel. - 7. Then comment on #797 (row counts and commit SHAs only, no prices) and close it. Use curl with `-H "User-Agent: curl/8.5.0"` and the `GITEA_TOKEN` from `.env`. + 1. `ENV stack market gaps` lists only `hud:hud` (M10). + 2. Every `market.entity_link` row has a non-empty `cite_ref` or `item_key`, and every `econ.fred`, `econ.soi_zip` and `market.revenue` row has an `item_key`. + 3. Each key value occurs 0 times in the replica's `cms.ingest_log`, in the bib URLs and in the file names under `data/market`. + 4. No bib attachment lies under `data/market/tiingo`. + 5. `market.price_daily` has SPY from 1993-01-29. + 6. The HOME-VISITS, `PFS` and FAMILY PRACTICE runs have stored portfolio CARs. + 7. Push, let the nightly `notebooks-integration` run render the panel, then comment on #797 (row counts and SHAs) and close it. Use curl with `-H "User-Agent: curl/8.5.0"` and `GITEA_TOKEN`. --- @@ -944,21 +1151,34 @@ Each item below is a planner's call. Nobody was available to ask; each is made h - **Spec coverage:** - exposed entities, cited, with delisted firms kept: Tasks 1–2; - - prices with a market and a sector index, hoarded: Task 3; - - rule, RUC and Panel windows and seat tenures: Task 10; - - market-model event studies with intervals and caveats: Task 11; - - portfolios by family and specialty, and descriptive tenure comparisons: Task 12; - - enrichment: BLS and PSPS through readers, FRED and SOI in Task 13, NPPES in Task 9's footprint, SEC filing text in Task 2 and company revenue in Task 13, all joined in Task 14; + - prices with market and sector indices: Task 3; + - rule events from the exposure calendar plus lineage, RUC and Panel meetings, and seat tenures: Task 10; + - event studies: Task 11; + - portfolios by family, specialty and program, and the tenure comparison: Task 12; + - enrichment: BLS/PSPS through readers (Task 14); FRED and SOI (Task 13); NPPES in the Task 9 footprint; SEC filing text (Task 2) and revenue (Task 13); - LDA and FEC: Tasks 4–5; - - Congress members, committees and bills: Tasks 6–7; - - district crosswalks and footprints: Tasks 8–9; - - one api.data.gov key with overrides: the Task 1 `KEY_ENV`; + - Congress: Tasks 6–7; + - districts and footprints: Tasks 8–9; - #797: Tasks 3 and 15. -- **Gaps stated rather than built:** - - NAICS (EDGAR records SIC only); - - segment revenue (companyfacts omits dimensional facts); - - committee assignments before 2013; +- **Gaps stated rather than built (W20):** + - NAICS (EDGAR has SIC only); + - segment revenue (companyfacts has no dimensions); + - RUC recommendation dates (not public before the rule); + - CPT Panel action-summary posting dates (AMA pages, only where `--ama-url` reads them); + - SEC property disclosures for footprints; + - committee assignments before the 113th Congress; - district geography before CD118; - footprint years other than 2024. -- **Type names:** checked across tasks: `EntityRow`, `SecurityRow`, `LinkRow`, `FilingRow`, `PriceRow`, `MonthlyRow`, `EventRow`, `Car`, `CarRow`, `TenureRow`, `JointYear`, `ZipDistrictRow`, `FootprintRow`, `DistrictView`. -- **Function names:** `day0` is defined in Task 10 and used in Tasks 11–12. `bootstrap_draws` and `percentile_interval` are defined in Task 11 and used in Task 12. +- **Rulings applied:** M1–M11, D1–D5 and H1–H2. + - W12's third retry loop (`market.http.get`) stays, because H1 forbids editing the `pfs.seats` and `pfs.psps` loops. + - The supersede and tenure duplication is removed. +- **Names:** + - `MarketEvent` is defined in Task 10 and used in Task 12. + - `day0` is defined in Task 10 and used in Task 12; Task 11 takes `d0` from its caller. + - `bootstrap_draws` and `percentile_interval` come from Task 11 and are used in Task 12. + - `seat_key` comes from Task 1 and is used in Task 10. + - `district_id` and `state_codes` come from Task 1 and are used in Tasks 6, 8 and 9. + - `alias_match`, the one rule, comes from Task 1 and is used in Tasks 4, 5 and 9. + - `read_cars` comes from Task 12 and is used in Task 15. + - Every row dataclass lives in `market/tables.py`. +- **Review Focus:** all five pinned tests exist in their owning tasks, including the new D3 test in Task 2.