71861d9d32
All checks were successful
CI / lint (push) Successful in 29s
CI / notebooks-smoke (push) Successful in 1m25s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Successful in 56s
Infra CI / zotero (push) Successful in 13s
Infra CI / docs (push) Successful in 1m14s
Infra CI / api (push) Successful in 50s
Infra CI / mc (push) Successful in 19s
Deploy / report (push) Successful in 13s
CI / test (push) Successful in 14m27s
Harden / build-scan-report (push) Successful in 26m15s
Renovate / renovate (push) Successful in 15s
Notebooks Integration / notebooks-integration (push) Successful in 7m16s
Zotero Sync / zotero-sync (push) Successful in 53s
Package Supply Chain / pkg-supply-chain (push) Successful in 58s
The M5 close-out missed half the issue's scope: #514 says 'OPPS/PFS reference data' and I cut over only OPPS, leaving PFS — the largest reference domain, 23.5M rows across 8 tables — entirely on the monolith, including pfs.* queries in the very notebook whose OPPS query was migrated. This completes PFS the same way: - publish_opps_to_lake.py → publish_reference_to_lake.py with a schema registry (opps: 3 tables, pfs: 8); host-side docker-exec wrapper extracted to dev/scripts/_lake.py, shared by the ingests. - PFS published to the lake and read-back verified: carrier_locality 21,863,770 rows in 10.1s, plus rvu/gpci/clinical_labor/medical_ equipment/medical_supply/physician_work_time/zip_carrier_locality. - New dev/scripts/ingest_pfs.py wraps pfs.pipe.load_all (previously ad-hoc, no entrypoint) with the standard plumbing: duckdb_batch preflight, replica refresh, lake publish. - 5 notebooks migrated: pfs_calcs, pfs_reconciliation, skin_sub_budget_neutrality read the lake as their primary connection; skin_sub_pricing and skin_sub_cost_sharing switch their pure-pfs cells to the lake. The one cross-source join (pfs × skin_subs) stays on the monolith mirror, annotated. All 5 headless-verified in prod: zero cell errors. - pfs_calcs leaves the pre-commit host-run safe list (the lake catalog is compose-internal); the nightly integration covers it in-container.
115 lines
2.7 KiB
Python
115 lines
2.7 KiB
Python
import marimo
|
|
|
|
__generated_with = "0.21.1"
|
|
app = marimo.App(width="medium")
|
|
|
|
|
|
@app.cell(hide_code=True)
|
|
def _():
|
|
import marimo as mo
|
|
|
|
return (mo,)
|
|
|
|
|
|
@app.cell(hide_code=True)
|
|
def _(mo):
|
|
mo.md("""
|
|
# PFS Reconciliation
|
|
|
|
Runs `rec.pricers.pfs.PfsPricer` against `pfs.carrier_locality`
|
|
for a chosen year and reports the per-row delta. Goal is perfect
|
|
1:1 concordance. Tracks **homelab/stack#340**.
|
|
""")
|
|
return
|
|
|
|
|
|
@app.cell(hide_code=True)
|
|
def _():
|
|
from conf import connect
|
|
from rec.engine import reconcile
|
|
from rec.pricers.pfs import PfsPricer
|
|
|
|
# PFS reference data lives in the DuckLake lakehouse (M5, #514);
|
|
# queries are unchanged — the lake is the default database.
|
|
con = connect.ducklake()
|
|
pricer = PfsPricer()
|
|
return con, pricer, reconcile
|
|
|
|
|
|
@app.cell(hide_code=True)
|
|
def _(con, mo, pricer):
|
|
_years = pricer.years_available(con)
|
|
if not _years:
|
|
year_picker = mo.ui.dropdown(
|
|
options={"(no data loaded)": 0}, value="(no data loaded)", label="Year"
|
|
)
|
|
else:
|
|
year_picker = mo.ui.dropdown(
|
|
options={str(y): y for y in _years},
|
|
value=str(_years[-1]),
|
|
label="Year",
|
|
)
|
|
|
|
tolerance = mo.ui.slider(
|
|
start=0, stop=10, step=1, value=0, label="Tolerance (cents)"
|
|
)
|
|
|
|
mo.hstack([year_picker, tolerance], justify="start", gap=1)
|
|
return tolerance, year_picker
|
|
|
|
|
|
@app.cell(hide_code=True)
|
|
def _(con, mo, pricer, reconcile, tolerance, year_picker):
|
|
_year = int(year_picker.value) if year_picker.value else 0
|
|
if _year == 0:
|
|
result = None
|
|
mo.md("*Load `pfs.rvu`, `pfs.gpci`, and `pfs.carrier_locality` first.*")
|
|
else:
|
|
result = reconcile(pricer, con, _year, tolerance_cents=int(tolerance.value))
|
|
mo.md(result.summary_md(top_n=25))
|
|
return (result,)
|
|
|
|
|
|
@app.cell(hide_code=True)
|
|
def _(mo):
|
|
mo.md("""
|
|
## Delta table
|
|
|
|
Every row in the outer join, sorted by the largest absolute delta.
|
|
``is_exact`` uses the tolerance above; ``is_near`` is always a 1¢
|
|
window. Null ``fee_gt`` means the row is calculated-only; null
|
|
``fee_calc`` means ground-truth-only.
|
|
""")
|
|
return
|
|
|
|
|
|
@app.cell(hide_code=True)
|
|
def _(result):
|
|
_deltas = result.deltas if result is not None else None
|
|
_deltas
|
|
return
|
|
|
|
|
|
@app.cell(hide_code=True)
|
|
def _(mo):
|
|
mo.md("""
|
|
## Warnings
|
|
|
|
Non-fatal issues surfaced by the engine — duplicate join keys,
|
|
missing columns, rule lookup failures, etc.
|
|
""")
|
|
return
|
|
|
|
|
|
@app.cell(hide_code=True)
|
|
def _(mo, result):
|
|
if result is None or not result.warnings:
|
|
mo.md("*(none)*")
|
|
else:
|
|
mo.md("\n".join(f"- {w}" for w in result.warnings))
|
|
return
|
|
|
|
|
|
if __name__ == "__main__":
|
|
app.run()
|