From 2e4465084e347ac63808968cc198f390ea335e74 Mon Sep 17 00:00:00 2001 From: kert Date: Sun, 25 Oct 2026 16:36:15 -0400 Subject: [PATCH] =?UTF-8?q?docs(bcda):=20upstream=20review=20=E2=80=94=20b?= =?UTF-8?q?cda-app=207decdcc,=20400=20PRs=20and=20the=20docs=20source=20ag?= =?UTF-8?q?ainst=20what=20we=20built?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eight gaps, two of which change what 'current' means for a claim: error files dropped (a partial job still advances the watermark), and claim identity across control-number changes, DME VMS→NCH digit and Part D natural keys. Also: attribution expiration/January in the checks, adopting expired or cancellable jobs, the narrower _typeFilter surface, runout calendar, vendor manifest coverage, and our own docs quoting the server's (since, transactionTime] window. Each gap has a tracker issue. refs #1139 --- .../specs/2026-10-25-bcda-upstream-review.md | 226 ++++++++++++++++++ 1 file changed, 226 insertions(+) create mode 100644 docs/superpowers/specs/2026-10-25-bcda-upstream-review.md diff --git a/docs/superpowers/specs/2026-10-25-bcda-upstream-review.md b/docs/superpowers/specs/2026-10-25-bcda-upstream-review.md new file mode 100644 index 0000000..033d439 --- /dev/null +++ b/docs/superpowers/specs/2026-10-25-bcda-upstream-review.md @@ -0,0 +1,226 @@ +# Upstream review: CMSgov/bcda-app + bcda.cms.gov vs. what we built (2026-10-25) + +Scope: `CMSgov/bcda-app` at `7decdcc` (r302, 2026-10-21; the commit our +vendor manifest pins), its last 400 pull requests (the repo has issues +enabled but none filed — the team works from Jira tickets named in PR +titles), and the docs source `CMSgov/bcda-static-site` at `cd14bab` +(2026-10-21). Read against `bcda.client`, `bcda.store`, `bcda.sync`, +`bcda.express.v3` / `cclf`, the manager and the two design records. + +Verdict: the client, store and reconciliation match the server's +behaviour where it is documented, and the real pull confirmed the +`_since` contract. Eight things are missing or under-specified. Two of +them change what "current" means for a claim (A, B); the rest are +operational. Each has a tracker issue (listed at the end). + +## What the server actually does (facts we now rely on) + +- **The `_since` window is `(since, transactionTime]`.** The worker + queries BFD with `_lastUpdated=gt` and + `_lastUpdated=le` (`bcda/client/bluebutton.go`, + `updateParamWithLastUpdated`). `transactionTime` is BFD's + `Bundle.meta.lastUpdated` — the instant the last ETL load completed — + with a fallback to `now()` when BFD reports an epoch or pre-2000 value + (PR #1483, "Use correct lastUpdated for v3", merged 2026-09-04). That + is why the sandbox stamps midnight: synthetic data. Our watermark = + previous `transactionTime` with an exclusive lower bound is exactly + the server's contract; `overlap_seconds=0` is correct. +- **Newly attributed enrollees' history** comes only on `Group/all`, + only when the server runs with `BCDA_ENABLE_NEW_GROUP` (SOPS-encrypted + per environment in `ops/`; the docs state the behaviour as fact for + `/Group`). "New" is computed against the attribution file in force at + `_since` (`FindOldCCLFFile`); if `_since` predates the oldest + attribution file the server holds, the request is a **404** + "attribution file not found for given _since parameter" (PR #1236). +- **Service-date window.** Every EOB query carries a claims window: + earliest = the model's lookback (`lookback_period` 2–3 performance + years for CKCC/KCF/DC/ACCESS; none for SSP/CEC/NGACO — hence "data as + early as March 1, 2021", BFD v3's floor), latest = runout claim-through + date or a terminated entity's claims date (`setClaimsDate`, + `setRestrictiveServiceDateWindow`). A caller's `service-date` filter is + intersected with it. GUIDE has fixed windows and **no partially + adjudicated claims** (`v3_no_partial_claims_models`), so + `outcome=partial` is empty for it by design. +- **Suppression.** Enrollees who opted out (bene-prefs file; SAMHSA + preference too) are removed from the attribution set at job time + (`GetSuppressedMBIs`, 60-day lookback) and sensitive claims are + excluded at BFD (`excludeSAMHSA`, `_security:not=42CFRPart2`, PR + #1344). A delta carries no signal for either; only a baseline shows + the enrollee gone. Terminated model entities lose access the day + participation ends. +- **Manifest shape.** `output` lists data files; **`error` lists + OperationOutcome ndjson files**: one `-error.ndjson` per chunk + that hit BFD errors plus a job-level `warnings-and-info.ndjson`. + Blank files are filtered out server-side. EOB files hold 50 resources + each (2024-12 performance change). +- **Job lifecycle.** Nine states; `/jobs` maps Archived/Expired → + completed, FailedExpired → failed, CancelledExpired → cancelled, + Pending → in-progress. Files expire 24 h after completion. + **Completed jobs can be cancelled** (PR #1495, 2026-09): a DELETE + flips Completed → Cancelled and status/data then answer 404 — on the + shared sandbox any other user can do this to a job we are adopting. + Jobs pending more than 6 h trip the server's own health check (PR + #1500). +- **Rate limits.** 3,000 requests per 5 minutes per IP; `/auth/token` + has its own limit (tokens last 20 min — cache them); re-creating a + job identical to one In Progress is a 429 (`rate_limit_config.all: + true` in prod). `Retry-After` is always sent. +- **Refresh cadence.** Patient and Coverage 6×/week; NCH EOB weekly on + Monday (0–7 days after adjudication); Part D 5×/week; SharedSystem + 4×/week. BCDA recommends an NCH pull weekly and the others daily. +- **Re-loads.** "As we continue to refine the v3 dataset, some fields + will occasionally be re-loaded on our end. Corrected claims will show + a more recent `last_updated` date and may result in larger than + expected payloads." Expect `replaced` spikes that are not clinical + changes. +- **January.** `/Group/all` and `/Patient` are disabled per entity while + the new performance year's attribution loads (about a month); runout + keeps working and is refreshed monthly through July. + +## Gaps + +### A · Error files are dropped, and a partial job still advances the watermark + +`Store._download_outputs` reads only `output`; `Client.download` logs +the count of `error` entries and saves them under a different name, but +the manager never uses that path. A job whose manifest lists error +files is a **partial export**: some enrollees' resources are missing for +reasons on the server side. `bcda.sync` then takes its `transactionTime` +as the watermark and the next `_since` skips those enrollees' updates for +good. + +Fix: download `error` entries into the job directory +(`OperationOutcome.N.ndjson`), record them in `files.parquet` with +`resource_type=OperationOutcome`, mark the job `partial` in the manifest +summary, make `watermarks()` not advance past a partial job for its +types, add a `partial_job` check (warn) and show the OperationOutcome +diagnostics on the Store page. Re-pulling with the same `_since` is the +remedy and is idempotent. + +### B · Claim identity across control-number changes, Part D, and DME + +The docs' dedupe rules go beyond "same `CLM-CNTL-NUM`": + +1. **Changed control number.** The new claim carries the previous one in + `ExplanationOfBenefit.related`. We flatten it (`clm_orig_cntl_num`) + and write `bcda.eob_supersession`, but current state still keeps the + predecessor as its own claim and the connector counts both. Fix: in + `_recompute`, a winner whose `clm_orig_cntl_num` names another current + key supersedes it (mark `_superseded_by`, exclude from current counts + and from `_stg_claim_header`); walk the chain (institutional claims + carry only the immediate parent). NCH professional claims never + change number, so this is institutional and SharedSystem→NCH + transitions. +2. **DME (VMS → NCH).** The leading `1` is dropped when a VMS claim is + released to NCH, so the SharedSystem and NCH versions of the same DME + claim have control numbers differing by that digit. Our + SharedSystem-vs-NCH dedupe (`_stg_claim_header`, and `KEY_EXPR` in + `bcda.sync`) compares exact strings. Fix: a normalised key + (`strip leading 1 when system_type = SharedSystem and the claim is + DME`, i.e. `eob_type`/`clm_type_cd` DME) for matching only. +3. **Part D without a control number.** Dedupe by the CCLF Information + Packet §5.1.2 natural key: `CLM_LINE_FROM_DT`, + `PRVDR_SRVC_ID_QLFYR_CD`, `CLM_SRVC_PRVDR_GNRC_ID_NUM`, + `CLM_LINE_RX_SRVC_RFRNC_NUM`, `CLM_LINE_RX_FILL_NUM`. We flatten four + of the five (`prvdr_srvc_id_qlfyr_cd` is not captured) and key PDEs + without a control number by FHIR id, so a re-delivered PDE version + becomes a second claim. Fix: flatten the qualifier, use the natural + key as the PDE fallback in `KEY_EXPR` and the connector's pharmacy + dedupe. + +### C · Attribution state is not part of the temporality checks + +`attribution_status` now returns `expiration_dates` as well as +`ingestion_dates` (PR #1501). Past the expiration, and all of January, +`/Group/all` and `/Patient` answer 404 "No up-to-date attribution +information". `auth_check` already shows both dates; `check_since` only +uses `last_attribution_update`. Fix: `attribution_expired` (block) when +`attribution_access_expiration` is in the past; `attribution_floor` +(warn) when `_since` is earlier than the first attribution file we have +seen (the server will 404); map the server's 404 text to a plain message +in the UI. + +### D · Adopting a job another sandbox user can cancel or that has expired + +`completed_jobs()` keeps every Task whose FHIR status is `completed`, +which includes Archived and Expired jobs whose files are gone (410), and +since PR #1495 a Completed job can turn Cancelled under us (404 during +download). `poll` raises the right errors, but the Exports page offers +jobs we cannot use. Fix: filter `completed_jobs()` to `end` within 24 h, +and let `adopt` finish partial downloads as `failed` with the server's +diagnostics instead of raising mid-way. + +### E · `_typeFilter` surface is narrower than the server's + +Server accepts: comma-separated `_tag` tuples and `outcome=partial,complete` +(PR #1360), several `service-date` parameters with `eq/ne/gt/ge/lt/le` +prefixes and `YYYY`, `YYYY-MM`, `YYYY-MM-DD` forms, with validation that +rejects comma lists and conflicting prefixes (PR #1498). We build +`_tag` lists, one `outcome`, and only `service-date=gt`. Fix: +`service_date_to`, `outcome` as a list, and surface the server's +`ParameterValidationError` text. + +### F · Runout and population bookkeeping + +Runout should be pulled monthly with `_since` through July of the +following year, then stops. Our population split handles the data; the +checks do not know the runout calendar. Fix: a `runout_window` note +(warn after July of PY+1), and the Sync page should label the two +populations. + +### G · Vendored upstream is incomplete for the questions we now ask + +The manifest pins the app source and the v3/v2 dictionaries and samples. +It does not include: `BCDA_Partially_Adjudicated_Data_Dictionary.xlsx`, +`unmapped-fields-between-cclf-and-bcda.xlsx` (CMS's own CCLF↔BCDA gap +list — compare with our `bcda_v3_cclf_absent.csv`), the docs pages whose +text defines behaviour (`filter-claims-data.md`, +`partially-adjudicated-claims-data.md`, `access-claims-data.md`, +`how-to-migrate-v3.md`, `faqs.md`, the announcements), `bcda/fhir/param.go` +(the `_typeFilter` grammar), `bcda/client/bluebutton.go` (the +`_lastUpdated`/service-date translation) or `conf/configs/prod.yml` +(lookbacks, `v3_no_partial_claims_models`). Fix: add them to the manifest +so the weekly drift workflow watches them; the current upstream hashes +for the three files we do pin match. + +### H · Documentation of our own semantics should quote the server + +Our design record says "`_since` filters on `lastUpdated`" and "the +sandbox stamps midnight". It should say the window is +`(since, transactionTime]`, that `transactionTime` is BFD's load +completion instant (synthetic environments fall back to `now()`), that +the lookback window bounds a baseline by service date, that opt-out and +SAMHSA suppression are silent in deltas, and that CMS re-loads will +show as `replaced` without clinical change. + +## Things checked and found fine + +- `Prefer: respond-async` + `Accept: application/fhir+json` on `$export` + and `/jobs`; `Content-Location` job URL; `X-Progress` parsing. +- 401 re-auth, 429 with `Retry-After`, exponential back-off, gzip + (`Accept-Encoding: gzip`) on downloads, 20-minute token cache. +- `_tag` requires the full `system|code` token (PR #1296) — ours does. +- Default `_typeFilter` excludes SharedSystem (PR #1396) — documented in + the UI and the connector's source rank. +- `JobCancelledError` on 404 "Job has been cancelled"; `JobExpiredError` + on 410. +- Job `request` line parsed for `_since` on adopted jobs. +- `/jobs` Task Bundle unwrapping and client-side status filtering (the + server's `_status=Completed` 404 we hit is consistent with its mapping + table). +- Within one snapshot a control number appears once (verified on the + real pull), matching "claims are automatically deduplicated within a + source system". + +## Sources + +- bcda-app `7decdcc`: `bcda/client/bluebutton.go`, `bcda/service/service.go`, + `bcda/api/requests.go`, `bcda/api/v3/api.go`, `bcda/fhir/param.go`, + `bcda/models/models.go`, `bcda/constants/*.go`, `bcdaworker/worker/worker.go`, + `conf/configs/{sandbox,prod}.yml`, `bcda/bene-prefs/*.go`. +- PRs #1483, #1236, #1505, #1498, #1396, #1344, #1360, #1416, #1495, + #1500, #1501, #1367, #1512, #1518, #1511, #1283, #1407, #1376, #1521. +- bcda-static-site `cd14bab`: `src/_pages/api-documentation/{filter-claims-data,access-claims-data,how-to-migrate-v3}.md`, + `src/_pages/bcda-data/partially-adjudicated-claims-data.md`, + `src/_pages/support/faqs.md`, `src/_announcements/*.md`, + `src/assets/downloads/*`.