bcda: download manifest error files and stop a partial job from advancing the watermark #1293

New Issue

kert · 2026-10-25T20:36:16Z

kert commented

2026-10-25 20:36:16 +00:00

Upstream review docs/superpowers/specs/2026-10-25-bcda-upstream-review.md §A.

The completed-job manifest lists error files (one <uuid>-error.ndjson per chunk with BFD errors plus warnings-and-info.ndjson, FHIR OperationOutcome ndjson). Store._download_outputs reads only output, so a partial export looks complete and bcda.sync takes its transactionTime as the watermark — the next _since skips the failed enrollees' updates for good.

download error entries into exports/<key>/OperationOutcome[.N].ndjson; file rows with resource_type=OperationOutcome
job marked partial (manifest summary / job row) when error is non-empty
sync.watermarks() does not advance past a partial job for its types; check_since gains partial_job (warn) naming the keys
Store page shows the OperationOutcome diagnostics per job
tests: manifest with error files → partial, watermark unchanged, re-pull with the same _since reconciles to no change

Upstream review docs/superpowers/specs/2026-10-25-bcda-upstream-review.md §A. The completed-job manifest lists `error` files (one `<uuid>-error.ndjson` per chunk with BFD errors plus `warnings-and-info.ndjson`, FHIR OperationOutcome ndjson). `Store._download_outputs` reads only `output`, so a partial export looks complete and `bcda.sync` takes its `transactionTime` as the watermark — the next `_since` skips the failed enrollees' updates for good. - [ ] download `error` entries into `exports/<key>/OperationOutcome[.N].ndjson`; file rows with `resource_type=OperationOutcome` - [ ] job marked `partial` (manifest summary / job row) when `error` is non-empty - [ ] `sync.watermarks()` does not advance past a partial job for its types; `check_since` gains `partial_job` (warn) naming the keys - [ ] Store page shows the OperationOutcome diagnostics per job - [ ] tests: manifest with error files → partial, watermark unchanged, re-pull with the same `_since` reconciles to no change

kert added this to the P52: BCDA v3 migration — client, crosswalk-driven flatteners, vendored upstream fixtures, proving notebook milestone 2026-10-25 20:36:16 +00:00

kert added the cli data labels 2026-10-25 20:36:16 +00:00

kert referenced this issue

2026-10-25 20:36:40 +00:00

bcda: BCDA v3 — client, crosswalk-driven flatteners, vendored upstream, proving notebook (P52) #1140

kert referenced this issue from a commit

2026-10-25 21:11:20 +00:00

feat(bcda): close the upstream-review gaps — partial exports, claim identity across systems, attribution checks, filter grammar, vendored behaviour

kert commented

2026-10-25 21:11:47 +00:00

Done in 2cc1300: Store._download_outputs downloads the manifest error list as OperationOutcome[.N].ndjson, attempts every URL and raises one DownloadError; JobInfo.partial keeps a partial job out of watermarks(); check_since warns partial_job; Store page shows the issues; stack bcda errors KEY.

Done in 2cc1300: `Store._download_outputs` downloads the manifest `error` list as `OperationOutcome[.N].ndjson`, attempts every URL and raises one `DownloadError`; `JobInfo.partial` keeps a partial job out of `watermarks()`; `check_since` warns `partial_job`; Store page shows the issues; `stack bcda errors KEY`.

kert closed this issue

2026-10-25 21:11:47 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#1293

bcda: download manifest error files and stop a partial job from advancing the watermark #1293

bcda: download manifest `error` files and stop a partial job from advancing the watermark #1293