fix(bcda): legacy cut-off fallback is the job's start, not its completion
Some checks failed
CI / test (pull_request) Failing after 2m21s
Infra CI / zotero (pull_request) Successful in 17s
Infra CI / docs (pull_request) Successful in 14s
Infra CI / notebooks (pull_request) Successful in 59s
Infra CI / api (pull_request) Successful in 20s
Infra CI / mc (pull_request) Failing after 35s
Infra CI / bcda (pull_request) Successful in 16s
Infra CI / llm (pull_request) Successful in 15s
CI / lint (push) Successful in 39s
CI / test (push) Failing after 2m52s
Infra CI / notebooks (push) Successful in 1m3s
Infra CI / bcda (push) Successful in 1m46s
Infra CI / mc (push) Failing after 46s
CI / notebooks-smoke (push) Successful in 1m45s
Infra CI / zotero (push) Successful in 16s
Infra CI / api (push) Successful in 1m9s
Infra CI / docs (push) Successful in 1m44s
Infra CI / llm (push) Successful in 52s
CI / lint (pull_request) Successful in 35s
CI / notebooks-smoke (pull_request) Successful in 1m36s
Some checks failed
CI / test (pull_request) Failing after 2m21s
Infra CI / zotero (pull_request) Successful in 17s
Infra CI / docs (pull_request) Successful in 14s
Infra CI / notebooks (pull_request) Successful in 59s
Infra CI / api (pull_request) Successful in 20s
Infra CI / mc (pull_request) Failing after 35s
Infra CI / bcda (pull_request) Successful in 16s
Infra CI / llm (pull_request) Successful in 15s
CI / lint (push) Successful in 39s
CI / test (push) Failing after 2m52s
Infra CI / notebooks (push) Successful in 1m3s
Infra CI / bcda (push) Successful in 1m46s
Infra CI / mc (push) Failing after 46s
CI / notebooks-smoke (push) Successful in 1m45s
Infra CI / zotero (push) Successful in 16s
Infra CI / api (push) Successful in 1m9s
Infra CI / docs (push) Successful in 1m44s
Infra CI / llm (push) Successful in 52s
CI / lint (pull_request) Successful in 35s
CI / notebooks-smoke (pull_request) Successful in 1m36s
A job recorded before manifests existed has no transactionTime. The first version fell back to the instant the `complete` row was written and called it "safe, slightly overlapping" — I had that backwards: a _since later than the server's true cut-off is a gap, not an overlap. The real store showed it: the sandbox stamps transactionTime as midnight UTC of the run day, so a baseline that finished at 16:53 was being credited with a 16:53 cut-off and every same-day delta (00:00) was read as predating it. The fallback is now the job's first ledger row (the request instant), which can only overlap; Store.job_started_at() reads it. `stack bcda resume <key>` still replaces the fallback with the server's value — done for the four pre-manifest jobs in the real store, whose watermarks now read 2026-10-25T00:00:00Z. refs #1139
This commit is contained in:
@@ -192,8 +192,12 @@ Details settled while building:
|
||||
- The store now writes `exports/<key>/manifest.json` (the completed-job
|
||||
body) and puts `transaction_time` and `request` on the job row; jobs
|
||||
recorded before that fall back to the manifest, then to the instant the
|
||||
`complete` row was written (later than the true cut-off, so a safe,
|
||||
overlapping `_since`; the ledger records which source was used). An
|
||||
job was *started* (its first ledger row): the server's snapshot cannot
|
||||
be newer than when it began processing, so that `_since` overlaps and
|
||||
never gaps, whereas the completion instant would be later than the true
|
||||
cut-off and could skip updates made while the job ran. The ledger
|
||||
records which source was used, and `stack bcda resume <key>` replaces
|
||||
the fallback with the server's value while the job is still listed. An
|
||||
adopted job's `_since` is read from its request line.
|
||||
- Current tables carry `_job_key`, `_transaction_time`, `_since`, `_kind`,
|
||||
`_applied_seq`; the connector's `_ensure()` drops `_`-prefixed columns
|
||||
|
||||
@@ -240,6 +240,18 @@ class Store:
|
||||
return row
|
||||
raise KeyError(f"Job not found: {key}")
|
||||
|
||||
def job_started_at(self, key: str) -> datetime | None:
|
||||
"""Instant of the first transition recorded for *key* (the request), or ``None``.
|
||||
|
||||
``bcda.sync`` uses it as the conservative cut-off for a job whose
|
||||
manifest and ``transaction_time`` are both missing: the server's
|
||||
data cannot be newer than the moment it started processing the
|
||||
request, so a ``_since`` at the start only overlaps, never gaps.
|
||||
"""
|
||||
table = self._read_parquet(self._jobs_path, JOBS_SCHEMA)
|
||||
rows = [r for r in table.to_pylist() if r["key"] == key and r.get("ts")]
|
||||
return min(r["ts"] for r in rows) if rows else None
|
||||
|
||||
def list_jobs(self, *, status: str | None = None) -> list[dict]:
|
||||
"""List all jobs, optionally filtered by status."""
|
||||
table = self._read_parquet(self._jobs_path, JOBS_SCHEMA)
|
||||
|
||||
@@ -162,7 +162,7 @@ class JobInfo:
|
||||
type_filter: str
|
||||
since: str
|
||||
transaction_time: datetime
|
||||
transaction_source: str # manifest | job_row | completed_at
|
||||
transaction_source: str # job_row | manifest | started_at
|
||||
ts: datetime
|
||||
|
||||
@property
|
||||
@@ -188,8 +188,13 @@ def job_info(store: Store, job: dict) -> JobInfo:
|
||||
|
||||
``transactionTime`` comes from the job row (new ledgers), else the
|
||||
saved manifest, else — for jobs recorded before either existed — the
|
||||
instant the ``complete`` row was written (later than the true
|
||||
cut-off, so a safe, slightly overlapping ``_since``).
|
||||
instant the job was *started* (its first ledger row). The server's
|
||||
snapshot cannot be newer than the moment it began processing the
|
||||
request, so a ``_since`` at the start overlaps and never gaps; the
|
||||
completion instant would be *later* than the true cut-off and could
|
||||
skip updates made while the job ran. ``Store.resume`` replaces the
|
||||
fallback with the server's value whenever the job is still on the
|
||||
server.
|
||||
"""
|
||||
types, filt = split_types(job.get("types") or "")
|
||||
tt = _maybe_instant(job.get("transaction_time"))
|
||||
@@ -198,8 +203,8 @@ def job_info(store: Store, job: dict) -> JobInfo:
|
||||
tt = _maybe_instant(store.manifest(job["key"]).get("transactionTime"))
|
||||
source = "manifest"
|
||||
if tt is None:
|
||||
tt = _maybe_instant(job.get("ts"))
|
||||
source = "completed_at"
|
||||
tt = store.job_started_at(job["key"]) or _maybe_instant(job.get("ts"))
|
||||
source = "started_at"
|
||||
return JobInfo(
|
||||
key=job["key"],
|
||||
endpoint=job.get("endpoint") or "Group/all",
|
||||
|
||||
@@ -323,11 +323,25 @@ class TestChecks:
|
||||
|
||||
def test_job_info_falls_back_for_legacy_rows(self, tmp_path: Path) -> None:
|
||||
store = Store(path=str(tmp_path))
|
||||
t_start = datetime(2026, 10, 1, 9, 0, tzinfo=timezone.utc)
|
||||
store._append_job(
|
||||
key="OLD", endpoint="Group/all", types="Patient", status="complete"
|
||||
key="OLD",
|
||||
endpoint="Group/all",
|
||||
types="Patient",
|
||||
status="polling",
|
||||
ts=t_start,
|
||||
)
|
||||
store._append_job(
|
||||
key="OLD",
|
||||
endpoint="Group/all",
|
||||
types="Patient",
|
||||
status="complete",
|
||||
ts=t_start + timedelta(hours=1),
|
||||
)
|
||||
info = sync.job_info(store, store.get_job("OLD"))
|
||||
assert info.transaction_source == "completed_at" and info.kind == "baseline"
|
||||
# the conservative fallback is the *start* of the job, not its completion
|
||||
assert info.transaction_source == "started_at" and info.kind == "baseline"
|
||||
assert info.transaction_time == t_start
|
||||
store._write_manifest("OLD", {"transactionTime": "2026-08-01T00:00:00.000Z"})
|
||||
info = sync.job_info(store, store.get_job("OLD"))
|
||||
assert info.transaction_source == "manifest"
|
||||
|
||||
Reference in New Issue
Block a user