docs(bcda): upstream review — bcda-app 7decdcc, 400 PRs and the docs source against what we built

Eight gaps, two of which change what 'current' means for a claim:
error files dropped (a partial job still advances the watermark), and
claim identity across control-number changes, DME VMS→NCH digit and
Part D natural keys. Also: attribution expiration/January in the checks,
adopting expired or cancellable jobs, the narrower _typeFilter surface,
runout calendar, vendor manifest coverage, and our own docs quoting the
server's (since, transactionTime] window. Each gap has a tracker issue.

refs #1139

docs/superpowers/specs/2026-10-25-bcda-upstream-review.md

@@ -0,0 +1,226 @@
+# Upstream review: CMSgov/bcda-app + bcda.cms.gov vs. what we built (2026-10-25)
+
+Scope: `CMSgov/bcda-app` at `7decdcc` (r302, 2026-10-21; the commit our
+vendor manifest pins), its last 400 pull requests (the repo has issues
+enabled but none filed — the team works from Jira tickets named in PR
+titles), and the docs source `CMSgov/bcda-static-site` at `cd14bab`
+(2026-10-21). Read against `bcda.client`, `bcda.store`, `bcda.sync`,
+`bcda.express.v3` / `cclf`, the manager and the two design records.
+
+Verdict: the client, store and reconciliation match the server's
+behaviour where it is documented, and the real pull confirmed the
+`_since` contract. Eight things are missing or under-specified. Two of
+them change what "current" means for a claim (A, B); the rest are
+operational. Each has a tracker issue (listed at the end).
+
+## What the server actually does (facts we now rely on)
+
+- **The `_since` window is `(since, transactionTime]`.** The worker
+  queries BFD with `_lastUpdated=gt<since>` and
+  `_lastUpdated=le<transactionTime>` (`bcda/client/bluebutton.go`,
+  `updateParamWithLastUpdated`). `transactionTime` is BFD's
+  `Bundle.meta.lastUpdated` — the instant the last ETL load completed —
+  with a fallback to `now()` when BFD reports an epoch or pre-2000 value
+  (PR #1483, "Use correct lastUpdated for v3", merged 2026-09-04). That
+  is why the sandbox stamps midnight: synthetic data. Our watermark =
+  previous `transactionTime` with an exclusive lower bound is exactly
+  the server's contract; `overlap_seconds=0` is correct.
+- **Newly attributed enrollees' history** comes only on `Group/all`,
+  only when the server runs with `BCDA_ENABLE_NEW_GROUP` (SOPS-encrypted
+  per environment in `ops/`; the docs state the behaviour as fact for
+  `/Group`). "New" is computed against the attribution file in force at
+  `_since` (`FindOldCCLFFile`); if `_since` predates the oldest
+  attribution file the server holds, the request is a **404**
+  "attribution file not found for given _since parameter" (PR #1236).
+- **Service-date window.** Every EOB query carries a claims window:
+  earliest = the model's lookback (`lookback_period` 2–3 performance
+  years for CKCC/KCF/DC/ACCESS; none for SSP/CEC/NGACO — hence "data as
+  early as March 1, 2021", BFD v3's floor), latest = runout claim-through
+  date or a terminated entity's claims date (`setClaimsDate`,
+  `setRestrictiveServiceDateWindow`). A caller's `service-date` filter is
+  intersected with it. GUIDE has fixed windows and **no partially
+  adjudicated claims** (`v3_no_partial_claims_models`), so
+  `outcome=partial` is empty for it by design.
+- **Suppression.** Enrollees who opted out (bene-prefs file; SAMHSA
+  preference too) are removed from the attribution set at job time
+  (`GetSuppressedMBIs`, 60-day lookback) and sensitive claims are
+  excluded at BFD (`excludeSAMHSA`, `_security:not=42CFRPart2`, PR
+  #1344). A delta carries no signal for either; only a baseline shows
+  the enrollee gone. Terminated model entities lose access the day
+  participation ends.
+- **Manifest shape.** `output` lists data files; **`error` lists
+  OperationOutcome ndjson files**: one `<uuid>-error.ndjson` per chunk
+  that hit BFD errors plus a job-level `warnings-and-info.ndjson`.
+  Blank files are filtered out server-side. EOB files hold 50 resources
+  each (2024-12 performance change).
+- **Job lifecycle.** Nine states; `/jobs` maps Archived/Expired →
+  completed, FailedExpired → failed, CancelledExpired → cancelled,
+  Pending → in-progress. Files expire 24 h after completion.
+  **Completed jobs can be cancelled** (PR #1495, 2026-09): a DELETE
+  flips Completed → Cancelled and status/data then answer 404 — on the
+  shared sandbox any other user can do this to a job we are adopting.
+  Jobs pending more than 6 h trip the server's own health check (PR
+  #1500).
+- **Rate limits.** 3,000 requests per 5 minutes per IP; `/auth/token`
+  has its own limit (tokens last 20 min — cache them); re-creating a
+  job identical to one In Progress is a 429 (`rate_limit_config.all:
+  true` in prod). `Retry-After` is always sent.
+- **Refresh cadence.** Patient and Coverage 6×/week; NCH EOB weekly on
+  Monday (0–7 days after adjudication); Part D 5×/week; SharedSystem
+  4×/week. BCDA recommends an NCH pull weekly and the others daily.
+- **Re-loads.** "As we continue to refine the v3 dataset, some fields
+  will occasionally be re-loaded on our end. Corrected claims will show
+  a more recent `last_updated` date and may result in larger than
+  expected payloads." Expect `replaced` spikes that are not clinical
+  changes.
+- **January.** `/Group/all` and `/Patient` are disabled per entity while
+  the new performance year's attribution loads (about a month); runout
+  keeps working and is refreshed monthly through July.
+
+## Gaps
+
+### A · Error files are dropped, and a partial job still advances the watermark
+
+`Store._download_outputs` reads only `output`; `Client.download` logs
+the count of `error` entries and saves them under a different name, but
+the manager never uses that path. A job whose manifest lists error
+files is a **partial export**: some enrollees' resources are missing for
+reasons on the server side. `bcda.sync` then takes its `transactionTime`
+as the watermark and the next `_since` skips those enrollees' updates for
+good.
+
+Fix: download `error` entries into the job directory
+(`OperationOutcome.N.ndjson`), record them in `files.parquet` with
+`resource_type=OperationOutcome`, mark the job `partial` in the manifest
+summary, make `watermarks()` not advance past a partial job for its
+types, add a `partial_job` check (warn) and show the OperationOutcome
+diagnostics on the Store page. Re-pulling with the same `_since` is the
+remedy and is idempotent.
+
+### B · Claim identity across control-number changes, Part D, and DME
+
+The docs' dedupe rules go beyond "same `CLM-CNTL-NUM`":
+
+1. **Changed control number.** The new claim carries the previous one in
+   `ExplanationOfBenefit.related`. We flatten it (`clm_orig_cntl_num`)
+   and write `bcda.eob_supersession`, but current state still keeps the
+   predecessor as its own claim and the connector counts both. Fix: in
+   `_recompute`, a winner whose `clm_orig_cntl_num` names another current
+   key supersedes it (mark `_superseded_by`, exclude from current counts
+   and from `_stg_claim_header`); walk the chain (institutional claims
+   carry only the immediate parent). NCH professional claims never
+   change number, so this is institutional and SharedSystem→NCH
+   transitions.
+2. **DME (VMS → NCH).** The leading `1` is dropped when a VMS claim is
+   released to NCH, so the SharedSystem and NCH versions of the same DME
+   claim have control numbers differing by that digit. Our
+   SharedSystem-vs-NCH dedupe (`_stg_claim_header`, and `KEY_EXPR` in
+   `bcda.sync`) compares exact strings. Fix: a normalised key
+   (`strip leading 1 when system_type = SharedSystem and the claim is
+   DME`, i.e. `eob_type`/`clm_type_cd` DME) for matching only.
+3. **Part D without a control number.** Dedupe by the CCLF Information
+   Packet §5.1.2 natural key: `CLM_LINE_FROM_DT`,
+   `PRVDR_SRVC_ID_QLFYR_CD`, `CLM_SRVC_PRVDR_GNRC_ID_NUM`,
+   `CLM_LINE_RX_SRVC_RFRNC_NUM`, `CLM_LINE_RX_FILL_NUM`. We flatten four
+   of the five (`prvdr_srvc_id_qlfyr_cd` is not captured) and key PDEs
+   without a control number by FHIR id, so a re-delivered PDE version
+   becomes a second claim. Fix: flatten the qualifier, use the natural
+   key as the PDE fallback in `KEY_EXPR` and the connector's pharmacy
+   dedupe.
+
+### C · Attribution state is not part of the temporality checks
+
+`attribution_status` now returns `expiration_dates` as well as
+`ingestion_dates` (PR #1501). Past the expiration, and all of January,
+`/Group/all` and `/Patient` answer 404 "No up-to-date attribution
+information". `auth_check` already shows both dates; `check_since` only
+uses `last_attribution_update`. Fix: `attribution_expired` (block) when
+`attribution_access_expiration` is in the past; `attribution_floor`
+(warn) when `_since` is earlier than the first attribution file we have
+seen (the server will 404); map the server's 404 text to a plain message
+in the UI.
+
+### D · Adopting a job another sandbox user can cancel or that has expired
+
+`completed_jobs()` keeps every Task whose FHIR status is `completed`,
+which includes Archived and Expired jobs whose files are gone (410), and
+since PR #1495 a Completed job can turn Cancelled under us (404 during
+download). `poll` raises the right errors, but the Exports page offers
+jobs we cannot use. Fix: filter `completed_jobs()` to `end` within 24 h,
+and let `adopt` finish partial downloads as `failed` with the server's
+diagnostics instead of raising mid-way.
+
+### E · `_typeFilter` surface is narrower than the server's
+
+Server accepts: comma-separated `_tag` tuples and `outcome=partial,complete`
+(PR #1360), several `service-date` parameters with `eq/ne/gt/ge/lt/le`
+prefixes and `YYYY`, `YYYY-MM`, `YYYY-MM-DD` forms, with validation that
+rejects comma lists and conflicting prefixes (PR #1498). We build
+`_tag` lists, one `outcome`, and only `service-date=gt<date>`. Fix:
+`service_date_to`, `outcome` as a list, and surface the server's
+`ParameterValidationError` text.
+
+### F · Runout and population bookkeeping
+
+Runout should be pulled monthly with `_since` through July of the
+following year, then stops. Our population split handles the data; the
+checks do not know the runout calendar. Fix: a `runout_window` note
+(warn after July of PY+1), and the Sync page should label the two
+populations.
+
+### G · Vendored upstream is incomplete for the questions we now ask
+
+The manifest pins the app source and the v3/v2 dictionaries and samples.
+It does not include: `BCDA_Partially_Adjudicated_Data_Dictionary.xlsx`,
+`unmapped-fields-between-cclf-and-bcda.xlsx` (CMS's own CCLF↔BCDA gap
+list — compare with our `bcda_v3_cclf_absent.csv`), the docs pages whose
+text defines behaviour (`filter-claims-data.md`,
+`partially-adjudicated-claims-data.md`, `access-claims-data.md`,
+`how-to-migrate-v3.md`, `faqs.md`, the announcements), `bcda/fhir/param.go`
+(the `_typeFilter` grammar), `bcda/client/bluebutton.go` (the
+`_lastUpdated`/service-date translation) or `conf/configs/prod.yml`
+(lookbacks, `v3_no_partial_claims_models`). Fix: add them to the manifest
+so the weekly drift workflow watches them; the current upstream hashes
+for the three files we do pin match.
+
+### H · Documentation of our own semantics should quote the server
+
+Our design record says "`_since` filters on `lastUpdated`" and "the
+sandbox stamps midnight". It should say the window is
+`(since, transactionTime]`, that `transactionTime` is BFD's load
+completion instant (synthetic environments fall back to `now()`), that
+the lookback window bounds a baseline by service date, that opt-out and
+SAMHSA suppression are silent in deltas, and that CMS re-loads will
+show as `replaced` without clinical change.
+
+## Things checked and found fine
+
+- `Prefer: respond-async` + `Accept: application/fhir+json` on `$export`
+  and `/jobs`; `Content-Location` job URL; `X-Progress` parsing.
+- 401 re-auth, 429 with `Retry-After`, exponential back-off, gzip
+  (`Accept-Encoding: gzip`) on downloads, 20-minute token cache.
+- `_tag` requires the full `system|code` token (PR #1296) — ours does.
+- Default `_typeFilter` excludes SharedSystem (PR #1396) — documented in
+  the UI and the connector's source rank.
+- `JobCancelledError` on 404 "Job has been cancelled"; `JobExpiredError`
+  on 410.
+- Job `request` line parsed for `_since` on adopted jobs.
+- `/jobs` Task Bundle unwrapping and client-side status filtering (the
+  server's `_status=Completed` 404 we hit is consistent with its mapping
+  table).
+- Within one snapshot a control number appears once (verified on the
+  real pull), matching "claims are automatically deduplicated within a
+  source system".
+
+## Sources
+
+- bcda-app `7decdcc`: `bcda/client/bluebutton.go`, `bcda/service/service.go`,
+  `bcda/api/requests.go`, `bcda/api/v3/api.go`, `bcda/fhir/param.go`,
+  `bcda/models/models.go`, `bcda/constants/*.go`, `bcdaworker/worker/worker.go`,
+  `conf/configs/{sandbox,prod}.yml`, `bcda/bene-prefs/*.go`.
+- PRs #1483, #1236, #1505, #1498, #1396, #1344, #1360, #1416, #1495,
+  #1500, #1501, #1367, #1512, #1518, #1511, #1283, #1407, #1376, #1521.
+- bcda-static-site `cd14bab`: `src/_pages/api-documentation/{filter-claims-data,access-claims-data,how-to-migrate-v3}.md`,
+  `src/_pages/bcda-data/partially-adjudicated-claims-data.md`,
+  `src/_pages/support/faqs.md`, `src/_announcements/*.md`,
+  `src/assets/downloads/*`.