71861d9d32
All checks were successful
CI / lint (push) Successful in 29s
CI / notebooks-smoke (push) Successful in 1m25s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Successful in 56s
Infra CI / zotero (push) Successful in 13s
Infra CI / docs (push) Successful in 1m14s
Infra CI / api (push) Successful in 50s
Infra CI / mc (push) Successful in 19s
Deploy / report (push) Successful in 13s
CI / test (push) Successful in 14m27s
Harden / build-scan-report (push) Successful in 26m15s
Renovate / renovate (push) Successful in 15s
Notebooks Integration / notebooks-integration (push) Successful in 7m16s
Zotero Sync / zotero-sync (push) Successful in 53s
Package Supply Chain / pkg-supply-chain (push) Successful in 58s
The M5 close-out missed half the issue's scope: #514 says 'OPPS/PFS reference data' and I cut over only OPPS, leaving PFS — the largest reference domain, 23.5M rows across 8 tables — entirely on the monolith, including pfs.* queries in the very notebook whose OPPS query was migrated. This completes PFS the same way: - publish_opps_to_lake.py → publish_reference_to_lake.py with a schema registry (opps: 3 tables, pfs: 8); host-side docker-exec wrapper extracted to dev/scripts/_lake.py, shared by the ingests. - PFS published to the lake and read-back verified: carrier_locality 21,863,770 rows in 10.1s, plus rvu/gpci/clinical_labor/medical_ equipment/medical_supply/physician_work_time/zip_carrier_locality. - New dev/scripts/ingest_pfs.py wraps pfs.pipe.load_all (previously ad-hoc, no entrypoint) with the standard plumbing: duckdb_batch preflight, replica refresh, lake publish. - 5 notebooks migrated: pfs_calcs, pfs_reconciliation, skin_sub_budget_neutrality read the lake as their primary connection; skin_sub_pricing and skin_sub_cost_sharing switch their pure-pfs cells to the lake. The one cross-source join (pfs × skin_subs) stays on the monolith mirror, annotated. All 5 headless-verified in prod: zero cell errors. - pfs_calcs leaves the pre-commit host-run safe list (the lake catalog is compose-internal); the nightly integration covers it in-container.
63 lines
1.8 KiB
Python
63 lines
1.8 KiB
Python
"""Host-side lake publish helper shared by the ingest scripts.
|
|
|
|
The DuckLake catalog (postgres) and RustFS are compose-internal, so
|
|
publishing runs docker-exec'd in the notebooks container with
|
|
``POSTGRES_PASSWORD`` from ``.env``. Import from a sibling dev script
|
|
(the script's own directory is on ``sys.path``)::
|
|
|
|
import _lake
|
|
_lake.publish_lake(("opps",))
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import subprocess
|
|
from pathlib import Path
|
|
|
|
ROOT = Path(__file__).resolve().parents[2]
|
|
|
|
|
|
def publish_lake(schemas: tuple[str, ...]) -> None:
|
|
"""Publish reference schemas to the DuckLake lakehouse (M5, #514).
|
|
|
|
The lake is the authoritative store for reference data; the
|
|
monolith's copies stay as a deprecated mirror for the analytics
|
|
pipe and the read replica.
|
|
"""
|
|
pw = ""
|
|
env_file = ROOT / ".env"
|
|
if env_file.exists():
|
|
for line in env_file.read_text().splitlines():
|
|
if line.startswith("POSTGRES_PASSWORD="):
|
|
pw = line.split("=", 1)[1].strip()
|
|
break
|
|
if not pw:
|
|
raise SystemExit(
|
|
"lake publish: POSTGRES_PASSWORD not in .env (use --no-lake to skip)"
|
|
)
|
|
script = ROOT / "dev" / "scripts" / "publish_reference_to_lake.py"
|
|
subprocess.run(
|
|
["docker", "cp", str(script), "notebooks:/tmp/publish_reference_to_lake.py"],
|
|
check=True,
|
|
)
|
|
subprocess.run(
|
|
[
|
|
"docker",
|
|
"exec",
|
|
"-e",
|
|
f"POSTGRES_PASSWORD={pw}",
|
|
"-e",
|
|
"PYTHONPATH=/home/kert/src",
|
|
"notebooks",
|
|
"uv",
|
|
"run",
|
|
"--project",
|
|
"/home/kert/workspace",
|
|
"python",
|
|
"/tmp/publish_reference_to_lake.py",
|
|
"--schemas",
|
|
*schemas,
|
|
],
|
|
check=True,
|
|
)
|