Some checks failed
Infra CI / notebooks (push) Successful in 1m6s
CI / lint (push) Successful in 46s
CI / notebooks-smoke (push) Successful in 1m44s
CI / test (push) Failing after 2m36s
Infra CI / zotero (push) Successful in 21s
Infra CI / llm (push) Successful in 55s
Infra CI / bcda (push) Successful in 1m38s
Infra CI / docs (push) Successful in 1m46s
Infra CI / api (push) Successful in 1m8s
Infra CI / mc (push) Failing after 36s
CI / lint (pull_request) Successful in 43s
CI / test (pull_request) Failing after 2m30s
Infra CI / zotero (pull_request) Successful in 15s
Infra CI / llm (pull_request) Successful in 16s
CI / notebooks-smoke (pull_request) Successful in 1m52s
Infra CI / docs (pull_request) Successful in 15s
Infra CI / api (pull_request) Successful in 15s
Infra CI / notebooks (pull_request) Successful in 49s
Infra CI / bcda (pull_request) Successful in 18s
Infra CI / mc (pull_request) Failing after 35s
Design: docs/superpowers/specs/2026-10-25-bcda-since-reconciliation-design.md. Until now a _since export was loaded like a full one: the loader took the newest export directory per type, so a delta silently replaced the baseline it was meant to extend. bcda.sync makes incremental pulls safe end to end. First order — which row is current. A baseline (no _since) is the authoritative snapshot for its types; deltas of the same population at or after its cut-off extend it. Keys: Patient BENE_SK, Coverage id, EOB claim control number (each claim *version* has a new FHIR id — migration guide). Winner per key: newest meta.lastUpdated, then NCH/DDPS over SharedSystem (the connector's rule), then later transactionTime, later applied sequence, larger id. NotFinalAction winners stay as tombstones; only a baseline retires rows (the API cannot express deletion). eob_item follows its header. Second order — what a job did, per table, in the ledger: new / replaced / unchanged / stale_ignored, backfill (lastUpdated < _since for an enrollee we did not have — the API's promise for newly attributed enrollees on Group/all) vs late (same shape for one we had — should not happen), tombstoned, adjusted (clm_orig_cntl_num → bcda.eob_supersession), retired, orphans (claims/coverage without a current Patient), scope drift. Third order — invariants on every apply before anything is published: unique keys, count conservation (|cur'| = |cur| + new − retired), watermark monotone, every item has a header; replay determinism and idempotency are tested (apply in either order → byte-identical tables; re-apply → no-op). Current tables are re-derived from the applied partitions each time, so the state is a function of the set of applied jobs, not their arrival order. Temporality checks before a _since export starts (block/warn/ok): format, future (server 400s), floor (no v3 lastUpdated before 2026-03-26), no_baseline, gap (since > watermark: a window nobody would ever fetch), overlap (free), endpoint_mix (runout vs current population), backfill_scope (/Patient delta over a Group/all baseline), scope_filter (_typeFilter drift), inflight, unapplied, attribution_newer, stale_baseline. next_since() is the previous job's transactionTime, as BCDA recommends; a combined job takes the earliest watermark of its types. Store: job rows carry transaction_time and request, exports/<key>/ manifest.json keeps the completed-job body, resume() fills both in for jobs completed before this; adopted jobs read _since from their request line; a delta with nothing to deliver (200, empty output) still counts and advances the watermark. flatten_export(out_dir=) writes per-job partitions under flat/history/<key>/; current tables keep _job_key/_transaction_time/ _since/_kind/_applied_seq and the connector's _ensure() drops them before the stage contracts. load_bcda(reconcile=True) applies pending jobs and also loads bcda.sync_ledger, bcda.sync_watermark, bcda.eob_supersession; an explicit ndjson_dir or a store without job rows keeps the legacy path. Surfaces: `stack bcda sync status|check|apply|pull`, `stack bcda export --since auto [--force]`, `stack bcda resume`; the manager's Exports page gets a _since mode (off / watermark / date) with the checks shown and blocks respected, and a Sync page (watermarks, pending, reconcile, ledger); notebook section 8. Verified on the real Large-entity store: applying the three baselines over the legacy flat/ tables reported every row unchanged (0 new, 0 retired) and loaded counts identical to the full load (221,021 medical lines, 11,968 pharmacy, 9,582 eligibility); a real /Patient-endpoint EOB delta with a _typeFilter reconciled as 248 unchanged with the drift on record; the sandbox's transactionTime is midnight UTC of the run day, so a same-day zero-row delta is correctly ignored as predating its baseline. refs #1139
456 lines
15 KiB
Python
456 lines
15 KiB
Python
"""stack bcda — the manager's CLI face, with the service layer stubbed."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from unittest.mock import patch
|
|
|
|
import pytest
|
|
from typer.testing import CliRunner
|
|
|
|
from bcda.ui import service as svc
|
|
from cli import app
|
|
|
|
runner = CliRunner()
|
|
|
|
|
|
class _FakeClient:
|
|
base_url = svc.SANDBOX
|
|
version = "v3"
|
|
|
|
def close(self) -> None:
|
|
pass
|
|
|
|
def __enter__(self): # noqa: ANN204
|
|
return self
|
|
|
|
def __exit__(self, *exc): # noqa: ANN002, ANN204
|
|
return False
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _no_env(monkeypatch):
|
|
monkeypatch.delenv("BCDA_CLIENT_ID", raising=False)
|
|
monkeypatch.delenv("BCDA_CLIENT_SECRET", raising=False)
|
|
monkeypatch.setattr(svc, "make_client", lambda conn, output_dir=None: _FakeClient())
|
|
|
|
|
|
def test_help_lists_commands() -> None:
|
|
result = runner.invoke(app, ["bcda", "--help"])
|
|
assert result.exit_code == 0
|
|
for cmd in (
|
|
"ui",
|
|
"preflight",
|
|
"auth",
|
|
"jobs",
|
|
"export",
|
|
"adopt",
|
|
"cancel",
|
|
"store",
|
|
"load",
|
|
"vendor",
|
|
):
|
|
assert cmd in result.output
|
|
|
|
|
|
def test_unknown_preset() -> None:
|
|
result = runner.invoke(app, ["bcda", "preflight", "--preset", "Nope"])
|
|
assert result.exit_code == 1 and "Unknown preset" in result.output
|
|
|
|
|
|
def test_env_credentials_win(monkeypatch) -> None:
|
|
monkeypatch.setenv("BCDA_CLIENT_ID", "env")
|
|
monkeypatch.setenv("BCDA_CLIENT_SECRET", "secret")
|
|
seen = {}
|
|
|
|
def remember(conn, output_dir=None): # noqa: ANN001, ANN202
|
|
seen["conn"] = conn
|
|
return _FakeClient()
|
|
|
|
monkeypatch.setattr(svc, "make_client", remember)
|
|
monkeypatch.setattr(svc, "preflight", lambda c: {"ok": True})
|
|
assert runner.invoke(app, ["bcda", "preflight", "--preset", "Nope"]).exit_code == 0
|
|
assert seen["conn"].client_id == "env"
|
|
|
|
|
|
def test_preflight_and_auth(monkeypatch) -> None:
|
|
monkeypatch.setattr(
|
|
svc, "preflight", lambda c: {"ok": True, "fhir_version": "4.0.1"}
|
|
)
|
|
result = runner.invoke(app, ["bcda", "preflight"])
|
|
assert result.exit_code == 0 and '"fhir_version": "4.0.1"' in result.output
|
|
monkeypatch.setattr(svc, "preflight", lambda c: {"ok": False, "error": "down"})
|
|
assert runner.invoke(app, ["bcda", "preflight"]).exit_code == 1
|
|
monkeypatch.setattr(svc, "auth_check", lambda c: {"token_chars": 40})
|
|
result = runner.invoke(app, ["bcda", "auth", "--preset", "Small Model Entity"])
|
|
assert result.exit_code == 0 and '"token_chars": 40' in result.output
|
|
|
|
|
|
def test_jobs(monkeypatch) -> None:
|
|
rows = [
|
|
{
|
|
"job_id": "1",
|
|
"status": "completed",
|
|
"endpoint": "Group/all",
|
|
"types": "Patient",
|
|
"started": "s",
|
|
"ended": "e",
|
|
"reusable": True,
|
|
"job_url": "u",
|
|
}
|
|
]
|
|
monkeypatch.setattr(svc, "server_jobs", lambda c, limit: rows[:limit])
|
|
result = runner.invoke(app, ["bcda", "jobs", "--limit", "1"])
|
|
assert (
|
|
result.exit_code == 0
|
|
and "reusable" in result.output
|
|
and "1 job(s)" in result.output
|
|
)
|
|
|
|
|
|
class _FakeStore:
|
|
def get_files(self, key): # noqa: ANN001, ANN202
|
|
return [{"storage_path": f"{key}/a"}]
|
|
|
|
|
|
def test_export_per_type_and_one_job(monkeypatch) -> None:
|
|
monkeypatch.setattr(svc, "make_store", lambda path=None: _FakeStore())
|
|
calls = []
|
|
monkeypatch.setattr(
|
|
svc,
|
|
"export_per_type",
|
|
lambda store, client, **kw: calls.append(("per", kw)) or {"Patient": "K1"},
|
|
)
|
|
monkeypatch.setattr(
|
|
svc,
|
|
"start_export",
|
|
lambda store, client, **kw: calls.append(("one", kw)) or "K2",
|
|
)
|
|
result = runner.invoke(
|
|
app,
|
|
[
|
|
"bcda",
|
|
"export",
|
|
"-t",
|
|
"Patient",
|
|
"--system-type",
|
|
"DDPS",
|
|
"--outcome",
|
|
"complete",
|
|
"--since",
|
|
"2026-01-01T00:00:00+00:00",
|
|
],
|
|
)
|
|
assert result.exit_code == 0, result.output
|
|
assert (
|
|
"_typeFilter=ExplanationOfBenefit?_tag=" in result.output
|
|
and "Patient: job K1, 1 file(s)" in result.output
|
|
)
|
|
assert (
|
|
calls[0][0] == "per"
|
|
and calls[0][1]["types"] == ["Patient"]
|
|
and calls[0][1]["since"].startswith("2026")
|
|
)
|
|
result = runner.invoke(
|
|
app, ["bcda", "export", "--one-job", "--endpoint", "Group/runout"]
|
|
)
|
|
assert (
|
|
result.exit_code == 0
|
|
and "Patient,Coverage,ExplanationOfBenefit: job K2" in result.output
|
|
)
|
|
assert (
|
|
calls[-1][0] == "one"
|
|
and calls[-1][1]["endpoint"] == "Group/runout"
|
|
and calls[-1][1]["type_filter"] is None
|
|
)
|
|
|
|
|
|
def test_adopt_and_cancel(monkeypatch) -> None:
|
|
monkeypatch.setattr(svc, "make_store", lambda path=None: _FakeStore())
|
|
seen = {}
|
|
monkeypatch.setattr(
|
|
svc,
|
|
"adopt_job",
|
|
lambda store, client, url, types=None: (
|
|
seen.update(url=url, types=types) or "KA"
|
|
),
|
|
)
|
|
result = runner.invoke(app, ["bcda", "adopt", "77", "-t", "Patient"])
|
|
assert result.exit_code == 0 and "adopted job 77 as KA" in result.output
|
|
assert seen["url"].endswith("/api/v3/jobs/77") and seen["types"] == ["Patient"]
|
|
monkeypatch.setattr(svc, "cancel_job", lambda client, url: seen.update(cancel=url))
|
|
result = runner.invoke(app, ["bcda", "cancel", "78"])
|
|
assert result.exit_code == 0 and seen["cancel"].endswith("/jobs/78")
|
|
|
|
class _ResumeStore(_FakeStore):
|
|
def get_job(self, key): # noqa: ANN001, ANN202
|
|
return {"status": "complete", "transaction_time": "2026-10-25T00:00:00Z"}
|
|
|
|
monkeypatch.setattr(svc, "make_store", lambda path=None: _ResumeStore())
|
|
monkeypatch.setattr(
|
|
svc, "resume_job", lambda store, client, key: seen.update(resume=key) or key
|
|
)
|
|
result = runner.invoke(app, ["bcda", "resume", "K9"])
|
|
assert result.exit_code == 0, result.output
|
|
assert (
|
|
seen["resume"] == "K9"
|
|
and "transactionTime=2026-10-25T00:00:00Z" in result.output
|
|
)
|
|
|
|
|
|
def test_store_load_vendor(monkeypatch, tmp_path) -> None:
|
|
monkeypatch.setattr(svc, "make_store", lambda path=None: object())
|
|
monkeypatch.setattr(svc, "store_path", lambda: tmp_path)
|
|
monkeypatch.setattr(
|
|
svc,
|
|
"store_jobs",
|
|
lambda s: [
|
|
{
|
|
"key": "K1",
|
|
"status": "complete",
|
|
"types": "Patient",
|
|
"files": 2,
|
|
"bytes": 1234,
|
|
"ts": "t",
|
|
}
|
|
],
|
|
)
|
|
result = runner.invoke(app, ["bcda", "store"])
|
|
assert (
|
|
result.exit_code == 0 and "K1" in result.output and "1 job(s)" in result.output
|
|
)
|
|
monkeypatch.setattr(
|
|
svc, "load_and_publish", lambda **kw: {"bcda.patient": 5, **{"kw": len(kw)}}
|
|
)
|
|
result = runner.invoke(app, ["bcda", "load", "--no-pipeline", "--skip-flatten"])
|
|
assert (
|
|
result.exit_code == 0
|
|
and "bcda.patient: 5 rows" in result.output
|
|
and "Replica published" in result.output
|
|
)
|
|
|
|
monkeypatch.setattr(svc, "vendor_status", lambda: {"available": False})
|
|
assert runner.invoke(app, ["bcda", "vendor"]).exit_code == 1
|
|
status = {
|
|
"available": True,
|
|
"sources": [
|
|
{
|
|
"name": "bcda-app",
|
|
"kind": "github",
|
|
"origin": "CMSgov/bcda-app",
|
|
"commit": "abcdef123456789",
|
|
"files": 3,
|
|
}
|
|
],
|
|
"files": [],
|
|
"problems": [],
|
|
}
|
|
monkeypatch.setattr(svc, "vendor_status", lambda: status)
|
|
monkeypatch.setattr(svc, "vendor_check", lambda: [])
|
|
result = runner.invoke(app, ["bcda", "vendor", "--check"])
|
|
assert (
|
|
result.exit_code == 0
|
|
and "matches the manifest" in result.output
|
|
and "no upstream drift" in result.output
|
|
)
|
|
monkeypatch.setattr(svc, "vendor_check", lambda: ["bcda-app: swagger.json changed"])
|
|
result = runner.invoke(app, ["bcda", "vendor", "--check"])
|
|
assert result.exit_code == 1 and "1 upstream change(s)" in result.output
|
|
status["problems"] = ["missing x"]
|
|
result = runner.invoke(app, ["bcda", "vendor"])
|
|
assert result.exit_code == 1 and "! missing x" in result.output
|
|
|
|
|
|
def test_ui_launches_streamlit(monkeypatch) -> None:
|
|
with patch("subprocess.call", return_value=0) as call:
|
|
result = runner.invoke(app, ["bcda", "ui", "--port", "9999", "--no-headless"])
|
|
assert result.exit_code == 0 and "http://localhost:9999" in result.output
|
|
cmd = call.call_args.args[0]
|
|
assert cmd[1:3] == ["-m", "bcda.ui"]
|
|
assert cmd[cmd.index("--server.headless") + 1] == "false"
|
|
|
|
|
|
# ── sync ─────────────────────────────────────────────────────
|
|
|
|
|
|
def test_help_lists_sync() -> None:
|
|
result = runner.invoke(app, ["bcda", "sync", "--help"])
|
|
assert result.exit_code == 0
|
|
for cmd in ("status", "check", "apply", "pull"):
|
|
assert cmd in result.output
|
|
|
|
|
|
def test_export_since_auto_refused(monkeypatch) -> None:
|
|
monkeypatch.setattr(svc, "make_store", lambda path=None: _FakeStore())
|
|
monkeypatch.setattr(
|
|
svc, "resolve_since", lambda store, since, **kw: {"Patient": None}
|
|
)
|
|
|
|
def refuse(store, client, **kw): # noqa: ANN001, ANN003, ANN202
|
|
raise svc.SinceBlocked(
|
|
[
|
|
{
|
|
"level": "block",
|
|
"code": "no_baseline",
|
|
"message": "Patient: none",
|
|
"resource_type": "Patient",
|
|
}
|
|
]
|
|
)
|
|
|
|
monkeypatch.setattr(svc, "export_per_type", refuse)
|
|
result = runner.invoke(app, ["bcda", "export", "-t", "Patient", "--since", "auto"])
|
|
assert result.exit_code == 2
|
|
assert "_since Patient: — no baseline —" in result.output
|
|
assert "✗ no_baseline" in result.output and "--force" in result.output
|
|
|
|
|
|
def test_sync_status_and_check(monkeypatch) -> None:
|
|
monkeypatch.setattr(svc, "make_store", lambda path=None: _FakeStore())
|
|
status = {
|
|
"watermarks": [
|
|
{
|
|
"population": "current",
|
|
"type_filter": "",
|
|
"resource_type": "Patient",
|
|
"job_key": "B1",
|
|
"transaction_time": "2026-10-01T00:00:00.000+00:00",
|
|
"kind": "baseline",
|
|
"applied": False,
|
|
}
|
|
],
|
|
"jobs": [
|
|
{
|
|
"key": "B1",
|
|
"kind": "baseline",
|
|
"endpoint": "Group/all",
|
|
"type_filter": "",
|
|
"types": "Patient",
|
|
"since": "",
|
|
"transaction_time": "2026-10-01T00:00:00.000+00:00",
|
|
"transaction_source": "job_row",
|
|
"applied": False,
|
|
}
|
|
],
|
|
"pending": ["B1"],
|
|
"ledger": [
|
|
{
|
|
"applied_seq": 1,
|
|
"job_key": "B0",
|
|
"kind": "baseline",
|
|
"ok": True,
|
|
"counts": {"patient": {"new": 3, "replaced": 0, "retired": 0}},
|
|
"notes": [],
|
|
}
|
|
],
|
|
"current": {"patient": 3},
|
|
}
|
|
monkeypatch.setattr(svc, "sync_status", lambda store=None: status)
|
|
result = runner.invoke(app, ["bcda", "sync", "status"])
|
|
assert result.exit_code == 0, result.output
|
|
assert "(not yet applied)" in result.output and "PENDING" in result.output
|
|
assert "patient=3" in result.output and "patient:3+/0~/0-" in result.output
|
|
result = runner.invoke(app, ["bcda", "sync", "status", "--json"])
|
|
assert result.exit_code == 0 and '"pending"' in result.output
|
|
|
|
plan = {"Patient": "2026-10-01T00:00:00.000+00:00", "Coverage": None}
|
|
monkeypatch.setattr(
|
|
svc,
|
|
"resolve_since",
|
|
lambda store, since, **kw: {
|
|
t: v for t, v in plan.items() if not kw.get("types") or t in kw["types"]
|
|
},
|
|
)
|
|
monkeypatch.setattr(
|
|
svc,
|
|
"since_checks",
|
|
lambda store, since, **kw: [
|
|
{
|
|
"level": "ok",
|
|
"code": "watermark",
|
|
"message": "fine",
|
|
"resource_type": "Patient",
|
|
}
|
|
],
|
|
)
|
|
result = runner.invoke(app, ["bcda", "sync", "check", "--offline"])
|
|
assert result.exit_code == 2 # Coverage has no baseline
|
|
assert (
|
|
"✓ watermark" in result.output
|
|
and "Coverage: _since — no baseline —" in result.output
|
|
)
|
|
result = runner.invoke(app, ["bcda", "sync", "check", "auto", "-t", "Patient"])
|
|
assert result.exit_code == 0, result.output
|
|
|
|
|
|
def test_sync_apply_and_pull(monkeypatch) -> None:
|
|
monkeypatch.setattr(svc, "make_store", lambda path=None: _FakeStore())
|
|
counts = {k: 0 for k in svc.sync.COUNT_FIELDS} | {
|
|
"rows": 10,
|
|
"new": 3,
|
|
"current": 13,
|
|
}
|
|
report = {
|
|
"ok": True,
|
|
"applied": [
|
|
{
|
|
"job_key": "D1",
|
|
"kind": "delta",
|
|
"since": "2026-10-01T00:00:00.000+00:00",
|
|
"transaction_time": "2026-10-08T00:00:00.000+00:00",
|
|
"ok": True,
|
|
"counts": {"explanation_of_benefit": counts},
|
|
"notes": ["transactionTime from job_row"],
|
|
}
|
|
],
|
|
"skipped": ["B1"],
|
|
}
|
|
seen: dict = {}
|
|
monkeypatch.setattr(
|
|
svc,
|
|
"apply_sync",
|
|
lambda store=None, keys=None, force=False: (
|
|
seen.update(keys=keys, force=force) or report
|
|
),
|
|
)
|
|
monkeypatch.setattr(svc, "load_and_publish", lambda **kw: {"bcda.patient": 5})
|
|
result = runner.invoke(app, ["bcda", "sync", "apply", "--force", "-k", "D1"])
|
|
assert result.exit_code == 0, result.output
|
|
assert "D1 delta" in result.output and "new= 3" in result.output
|
|
assert (
|
|
"already applied: B1" in result.output and "Replica published" in result.output
|
|
)
|
|
assert seen == {"keys": ["D1"], "force": True}
|
|
|
|
monkeypatch.setattr(
|
|
svc,
|
|
"resolve_since",
|
|
lambda store, since, **kw: {"Patient": "2026-10-01T00:00:00.000+00:00"},
|
|
)
|
|
calls: list = []
|
|
monkeypatch.setattr(
|
|
svc,
|
|
"export_per_type",
|
|
lambda store, client, **kw: calls.append(kw) or {"Patient": "D2"},
|
|
)
|
|
result = runner.invoke(app, ["bcda", "sync", "pull", "-t", "Patient", "--no-apply"])
|
|
assert result.exit_code == 0, result.output
|
|
assert calls[0]["since"] == "auto" and "Patient: job D2" in result.output
|
|
result = runner.invoke(app, ["bcda", "sync", "pull", "-t", "Patient"])
|
|
assert result.exit_code == 0 and "Replica published" in result.output
|
|
|
|
def refuse(store, client, **kw): # noqa: ANN001, ANN003, ANN202
|
|
raise svc.SinceBlocked(
|
|
[
|
|
{
|
|
"level": "block",
|
|
"code": "gap",
|
|
"message": "gap",
|
|
"resource_type": "Patient",
|
|
}
|
|
]
|
|
)
|
|
|
|
monkeypatch.setattr(svc, "export_per_type", refuse)
|
|
result = runner.invoke(app, ["bcda", "sync", "pull", "-t", "Patient"])
|
|
assert result.exit_code == 2 and "✗ gap" in result.output
|