homelab/stack
1
0
Fork 0
Code Issues 20 Pull Requests Actions Packages Projects 1 Releases Wiki Activity

feat: full session — mail servers, comment pipeline, PRISMA fetch, email ingest
Some checks failed
CI / skinny-install (aco) (push) Successful in 1m12s
Details
CI / skinny-install (api) (push) Successful in 30s
Details
CI / skinny-install (bcda) (push) Successful in 36s
Details
CI / skinny-install (bib) (push) Successful in 35s
Details
CI / skinny-install (bls) (push) Successful in 27s
Details
CI / skinny-install (ccw) (push) Successful in 32s
Details
CI / skinny-install (cli) (push) Successful in 41s
Details
CI / skinny-install (cms) (push) Successful in 37s
Details
CI / skinny-install (conf) (push) Successful in 38s
Details
CI / skinny-install (opps) (push) Successful in 33s
Details
CI / skinny-install (perf) (push) Successful in 38s
Details
CI / skinny-install (pfs) (push) Successful in 38s
Details
CI / skinny-install (rex) (push) Successful in 34s
Details
Deploy / build-scan-report (push) Failing after 46s
Details
Infra CI / notebooks (push) Failing after 25s
Details
Infra CI / zotero (push) Successful in 12s
Details
Infra CI / docs (push) Failing after 16s
Details
CI / lint-test (push) Failing after 11m2s
Details
Infra CI / mc (push) Successful in 21s
Details
Infra CI / api (push) Successful in 29s
Details
Package Supply Chain / pkg-supply-chain (push) Failing after 41s
Details

Browse Source 
Mail: Maddy on DO (corwins.media+Resend, fhirworx.io+Postmark),
touchless/stateless/idempotent. Gitea SMTP via env_file. CMS inbox
at cmsupdates@mail.fhirworx.io with IMAP→bib poller.

Bib: regulations.gov v4 client, Federal Register discovery, 164K
comment backfill (running), IMAP email ingest, Zotero sync routing.

PRISMA: altcha PoW solver, CrossRef DOI resolution, 83/129 PDFs.
Zotero: schema parity, ops module, CLI, fail-fast guard.
CI: docs.Dockerfile COPY glob fix (tracks #341).
Infra: Gitea+marimo fhirworx themes, IOM/OIG modules.
This commit is contained in:
kert
2026-04-16 09:04:38 -04:00
parent 2a2df70e21
commit 16f3b43974
134 changed files with 23603 additions and 3944 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
.dockerignore
View File

@@ -13,6 +13,12 @@ tuva/
dev/
tests/
infra/
# Carve out the pieces that feed image builds so they ride along in the
# build context. `infra/` is otherwise heavy (certs, state, configs).
!infra/marimo/theme/
!infra/marimo/theme/**
!infra/gitea/
!infra/gitea/**
assets/
cloud/
mirrors/

14
.env.example
View File

@@ -51,3 +51,17 @@ GF_ADMIN_PASSWORD=admin
# ── Woodpecker ──────────────────────────────────────
WOODPECKER_ADMIN=kert
# ── PRISMA / LLM ───────────────────────────────────
# Model-agnostic by design. Flip PRISMA_LLM_PROVIDER to "openai-compat"
# (with PRISMA_LLM_BASE_URL pointing at a vLLM/Ollama endpoint) to
# swap away from Anthropic — no code changes.
PRISMA_LLM_PROVIDER=anthropic
PRISMA_LLM_MODEL=claude-opus-4-6
PRISMA_LLM_BASE_URL=
ANTHROPIC_API_KEY=
OPENAI_API_KEY=
DIGITAL_OCEAN_PAT= # for `stack prisma vpn up` — droplet for fallback fetches
PRISMA_VPN_REGION=nyc3 # DO region slug (reachable from your ISP)
PRISMA_FETCH_PROXY= # populated by `stack prisma vpn up`; e.g. socks5://127.0.0.1:1080
UNPAYWALL_EMAIL=dev@fhirworx.io # required by Unpaywall API ToS

22
.gitea/workflows/deploy.yml
View File

@@ -18,7 +18,7 @@ jobs:
run: curl -sL https://github.com/google/go-containerregistry/releases/latest/download/go-containerregistry_Linux_x86_64.tar.gz | tar xz -C /usr/local/bin crane
- name: Log in to registry
run: crane auth login gitea:3000 -u "${{ secrets.REGISTRY_USER }}" -p "${{ secrets.REGISTRY_TOKEN }}"
run: crane auth login git:3000 -u "${{ secrets.REGISTRY_USER }}" -p "${{ secrets.REGISTRY_TOKEN }}"
env:
CRANE_INSECURE: "true"
@@ -39,8 +39,8 @@ jobs:
- name: Push notebooks
run: |
docker save local/notebooks:build -o /tmp/notebooks.tar
crane push /tmp/notebooks.tar gitea:3000/homelab/stack/notebooks:${{ env.SHORT_SHA }} --insecure
crane push /tmp/notebooks.tar gitea:3000/homelab/stack/notebooks:latest --insecure
crane push /tmp/notebooks.tar git:3000/homelab/stack/notebooks:${{ env.SHORT_SHA }} --insecure
crane push /tmp/notebooks.tar git:3000/homelab/stack/notebooks:latest --insecure
- name: Build zotero
run: docker build -f infra/images/zotero.Dockerfile -t local/zotero:build data/zotero/
@@ -48,8 +48,8 @@ jobs:
- name: Push zotero
run: |
docker save local/zotero:build -o /tmp/zotero.tar
crane push /tmp/zotero.tar gitea:3000/homelab/stack/zotero:${{ env.SHORT_SHA }} --insecure
crane push /tmp/zotero.tar gitea:3000/homelab/stack/zotero:latest --insecure
crane push /tmp/zotero.tar git:3000/homelab/stack/zotero:${{ env.SHORT_SHA }} --insecure
crane push /tmp/zotero.tar git:3000/homelab/stack/zotero:latest --insecure
- name: Build docs
run: docker build -f infra/images/docs.Dockerfile -t local/docs:build .
@@ -57,8 +57,8 @@ jobs:
- name: Push docs
run: |
docker save local/docs:build -o /tmp/docs.tar
crane push /tmp/docs.tar gitea:3000/homelab/stack/docs:${{ env.SHORT_SHA }} --insecure
crane push /tmp/docs.tar gitea:3000/homelab/stack/docs:latest --insecure
crane push /tmp/docs.tar git:3000/homelab/stack/docs:${{ env.SHORT_SHA }} --insecure
crane push /tmp/docs.tar git:3000/homelab/stack/docs:latest --insecure
- name: Build api
run: docker build -f infra/images/api.Dockerfile -t local/api:build .
@@ -66,8 +66,8 @@ jobs:
- name: Push api
run: |
docker save local/api:build -o /tmp/api.tar
crane push /tmp/api.tar gitea:3000/homelab/stack/api:${{ env.SHORT_SHA }} --insecure
crane push /tmp/api.tar gitea:3000/homelab/stack/api:latest --insecure
crane push /tmp/api.tar git:3000/homelab/stack/api:${{ env.SHORT_SHA }} --insecure
crane push /tmp/api.tar git:3000/homelab/stack/api:latest --insecure
- name: Build mc
run: docker build -f infra/images/mc.Dockerfile -t local/mc:build infra/rustfs/
@@ -75,8 +75,8 @@ jobs:
- name: Push mc
run: |
docker save local/mc:build -o /tmp/mc.tar
crane push /tmp/mc.tar gitea:3000/homelab/stack/mc:${{ env.SHORT_SHA }} --insecure
crane push /tmp/mc.tar gitea:3000/homelab/stack/mc:latest --insecure
crane push /tmp/mc.tar git:3000/homelab/stack/mc:${{ env.SHORT_SHA }} --insecure
crane push /tmp/mc.tar git:3000/homelab/stack/mc:latest --insecure
- name: Scan notebooks
run: trivy image --severity HIGH,CRITICAL --exit-code 0 --format json -o notebooks-scan.json local/notebooks:build

22
.gitea/workflows/harden.yml
View File

@@ -19,7 +19,7 @@ jobs:
run: curl -sL https://github.com/google/go-containerregistry/releases/latest/download/go-containerregistry_Linux_x86_64.tar.gz | tar xz -C /usr/local/bin crane
- name: Log in to registry
run: crane auth login gitea:3000 -u "${{ secrets.REGISTRY_USER }}" -p "${{ secrets.REGISTRY_TOKEN }}"
run: crane auth login git:3000 -u "${{ secrets.REGISTRY_USER }}" -p "${{ secrets.REGISTRY_TOKEN }}"
env:
CRANE_INSECURE: "true"
@@ -37,8 +37,8 @@ jobs:
- name: Push notebooks
run: |
docker save local/notebooks:build -o /tmp/notebooks.tar
crane push /tmp/notebooks.tar gitea:3000/homelab/stack/notebooks:hardened --insecure
crane push /tmp/notebooks.tar gitea:3000/homelab/stack/notebooks:latest --insecure
crane push /tmp/notebooks.tar git:3000/homelab/stack/notebooks:hardened --insecure
crane push /tmp/notebooks.tar git:3000/homelab/stack/notebooks:latest --insecure
- name: Build zotero
run: docker build --no-cache -f infra/images/zotero.Dockerfile -t local/zotero:build data/zotero/
@@ -46,8 +46,8 @@ jobs:
- name: Push zotero
run: |
docker save local/zotero:build -o /tmp/zotero.tar
crane push /tmp/zotero.tar gitea:3000/homelab/stack/zotero:hardened --insecure
crane push /tmp/zotero.tar gitea:3000/homelab/stack/zotero:latest --insecure
crane push /tmp/zotero.tar git:3000/homelab/stack/zotero:hardened --insecure
crane push /tmp/zotero.tar git:3000/homelab/stack/zotero:latest --insecure
- name: Build docs
run: docker build --no-cache -f infra/images/docs.Dockerfile -t local/docs:build .
@@ -55,8 +55,8 @@ jobs:
- name: Push docs
run: |
docker save local/docs:build -o /tmp/docs.tar
crane push /tmp/docs.tar gitea:3000/homelab/stack/docs:hardened --insecure
crane push /tmp/docs.tar gitea:3000/homelab/stack/docs:latest --insecure
crane push /tmp/docs.tar git:3000/homelab/stack/docs:hardened --insecure
crane push /tmp/docs.tar git:3000/homelab/stack/docs:latest --insecure
- name: Build api
run: docker build --no-cache -f infra/images/api.Dockerfile -t local/api:build .
@@ -64,8 +64,8 @@ jobs:
- name: Push api
run: |
docker save local/api:build -o /tmp/api.tar
crane push /tmp/api.tar gitea:3000/homelab/stack/api:hardened --insecure
crane push /tmp/api.tar gitea:3000/homelab/stack/api:latest --insecure
crane push /tmp/api.tar git:3000/homelab/stack/api:hardened --insecure
crane push /tmp/api.tar git:3000/homelab/stack/api:latest --insecure
- name: Build mc
run: docker build --no-cache -f infra/images/mc.Dockerfile -t local/mc:build infra/rustfs/
@@ -73,8 +73,8 @@ jobs:
- name: Push mc
run: |
docker save local/mc:build -o /tmp/mc.tar
crane push /tmp/mc.tar gitea:3000/homelab/stack/mc:hardened --insecure
crane push /tmp/mc.tar gitea:3000/homelab/stack/mc:latest --insecure
crane push /tmp/mc.tar git:3000/homelab/stack/mc:hardened --insecure
crane push /tmp/mc.tar git:3000/homelab/stack/mc:latest --insecure
- name: Scan notebooks
run: trivy image --severity HIGH,CRITICAL --exit-code 0 --format json -o notebooks-scan.json local/notebooks:build

22
.gitea/workflows/rebuild-all.yml
View File

@@ -17,7 +17,7 @@ jobs:
run: curl -sL https://github.com/google/go-containerregistry/releases/latest/download/go-containerregistry_Linux_x86_64.tar.gz | tar xz -C /usr/local/bin crane
- name: Log in to registry
run: crane auth login gitea:3000 -u "${{ secrets.REGISTRY_USER }}" -p "${{ secrets.REGISTRY_TOKEN }}"
run: crane auth login git:3000 -u "${{ secrets.REGISTRY_USER }}" -p "${{ secrets.REGISTRY_TOKEN }}"
env:
CRANE_INSECURE: "true"
@@ -38,8 +38,8 @@ jobs:
- name: Push notebooks
run: |
docker save local/notebooks:build -o /tmp/notebooks.tar
crane push /tmp/notebooks.tar gitea:3000/homelab/stack/notebooks:${{ env.SHORT_SHA }} --insecure
crane push /tmp/notebooks.tar gitea:3000/homelab/stack/notebooks:latest --insecure
crane push /tmp/notebooks.tar git:3000/homelab/stack/notebooks:${{ env.SHORT_SHA }} --insecure
crane push /tmp/notebooks.tar git:3000/homelab/stack/notebooks:latest --insecure
- name: Build zotero
run: docker build -f infra/images/zotero.Dockerfile -t local/zotero:build data/zotero/
@@ -47,8 +47,8 @@ jobs:
- name: Push zotero
run: |
docker save local/zotero:build -o /tmp/zotero.tar
crane push /tmp/zotero.tar gitea:3000/homelab/stack/zotero:${{ env.SHORT_SHA }} --insecure
crane push /tmp/zotero.tar gitea:3000/homelab/stack/zotero:latest --insecure
crane push /tmp/zotero.tar git:3000/homelab/stack/zotero:${{ env.SHORT_SHA }} --insecure
crane push /tmp/zotero.tar git:3000/homelab/stack/zotero:latest --insecure
- name: Build docs
run: docker build -f infra/images/docs.Dockerfile -t local/docs:build .
@@ -56,8 +56,8 @@ jobs:
- name: Push docs
run: |
docker save local/docs:build -o /tmp/docs.tar
crane push /tmp/docs.tar gitea:3000/homelab/stack/docs:${{ env.SHORT_SHA }} --insecure
crane push /tmp/docs.tar gitea:3000/homelab/stack/docs:latest --insecure
crane push /tmp/docs.tar git:3000/homelab/stack/docs:${{ env.SHORT_SHA }} --insecure
crane push /tmp/docs.tar git:3000/homelab/stack/docs:latest --insecure
- name: Build api
run: docker build -f infra/images/api.Dockerfile -t local/api:build .
@@ -65,8 +65,8 @@ jobs:
- name: Push api
run: |
docker save local/api:build -o /tmp/api.tar
crane push /tmp/api.tar gitea:3000/homelab/stack/api:${{ env.SHORT_SHA }} --insecure
crane push /tmp/api.tar gitea:3000/homelab/stack/api:latest --insecure
crane push /tmp/api.tar git:3000/homelab/stack/api:${{ env.SHORT_SHA }} --insecure
crane push /tmp/api.tar git:3000/homelab/stack/api:latest --insecure
- name: Build mc
run: docker build -f infra/images/mc.Dockerfile -t local/mc:build infra/rustfs/
@@ -74,8 +74,8 @@ jobs:
- name: Push mc
run: |
docker save local/mc:build -o /tmp/mc.tar
crane push /tmp/mc.tar gitea:3000/homelab/stack/mc:${{ env.SHORT_SHA }} --insecure
crane push /tmp/mc.tar gitea:3000/homelab/stack/mc:latest --insecure
crane push /tmp/mc.tar git:3000/homelab/stack/mc:${{ env.SHORT_SHA }} --insecure
crane push /tmp/mc.tar git:3000/homelab/stack/mc:latest --insecure
- name: Scan notebooks
run: trivy image --severity HIGH,CRITICAL --exit-code 0 --format json -o notebooks-scan.json local/notebooks:build

1
.gitignore vendored
View File

@@ -50,3 +50,4 @@ docs/node_modules/
docs/build/
docs/.docusaurus/
hw/node_modules/
infra/marimo/src/

380
assets/css/gitea.css
View File

@@ -1,380 +0,0 @@
/* HTI-5 — fhirworx design system for Gitea */
@import url('https://fonts.googleapis.com/css2?family=Playfair+Display:wght@400;600;700&family=Source+Serif+4:ital,wght@0,300;0,400;0,600;1,400&family=JetBrains+Mono:wght@400;500;600&display=swap');
/* Theme metadata */
gitea-theme-meta-info {
--theme-display-name: "Fhirworx";
}
:root {
--is-dark-theme: false;
/* HTI-5 Color Palette */
--background: #F7F5F0;
--foreground: #1A1A18;
--card: #FAFAF7;
--primary: #1C2B3A;
--primary-foreground: #F7F5F0;
--secondary: #EDEBE6;
--muted-foreground: #6B6B68;
--border: #D4D0C8;
--sidebar: #1C2B3A;
--sidebar-foreground: #B8C5D0;
--sidebar-accent: #253748;
--destructive: #C0392B;
--chart-4: #2E8B6E;
--chart-5: #C8702A;
--font-display: "Playfair Display", Georgia, serif;
--font-body: "Source Serif 4", Georgia, serif;
--font-mono: "JetBrains Mono", "Fira Code", monospace;
/* Gitea color mappings — derived from hti5 tokens */
--color-primary: var(--primary);
--color-primary-light: #2E3D8F;
--color-primary-dark: #111c26;
--color-primary-alpha-10: color-mix(in srgb, var(--primary) 10%, transparent);
--color-primary-alpha-20: color-mix(in srgb, var(--primary) 20%, transparent);
--color-primary-alpha-40: color-mix(in srgb, var(--primary) 40%, transparent);
--color-secondary: var(--muted-foreground);
--color-secondary-light: #9B9B98;
--color-secondary-dark: var(--foreground);
/* Background colors */
--color-body: var(--background);
--color-box-body: var(--card);
--color-box-header: var(--secondary);
/* Text colors */
--color-text: var(--foreground);
--color-text-light: var(--muted-foreground);
--color-text-dark: var(--foreground);
/* Input colors */
--color-input-text: var(--foreground);
--color-input-background: var(--card);
--color-input-border: var(--border);
/* Status colors */
--color-success: var(--chart-4);
--color-success-text: #fff;
--color-warning: var(--chart-5);
--color-warning-text: #fff;
--color-error: var(--destructive);
--color-error-text: #fff;
--color-info: var(--primary);
--color-info-text: var(--primary-foreground);
/* Border */
--color-border: var(--border);
/* Link colors */
--color-link: var(--primary);
--color-link-hover: var(--foreground);
/* Code */
--color-code-bg: var(--secondary);
--color-code-border: var(--border);
/* Shadows */
--color-shadow: rgba(0, 0, 0, 0.1);
}
/* Global styles */
body {
font-family: var(--font-body) !important;
background: var(--background) !important;
color: var(--foreground) !important;
}
/* Headings */
h1, h2, h3, h4, h5, h6,
.header-wrapper .header {
font-family: var(--font-display) !important;
font-weight: 600 !important;
line-height: 1.3 !important;
}
h1 { font-size: 22px !important; color: var(--foreground) !important; }
h2 { font-size: 17px !important; color: var(--primary) !important; }
h3 { font-size: 14px !important; color: var(--foreground) !important; }
/* Navbar */
.ui.secondary.menu.navbar,
.navbar {
background: var(--sidebar) !important;
border-bottom: 3px solid var(--foreground) !important;
}
.navbar .item {
font-family: var(--font-body) !important;
color: var(--sidebar-foreground) !important;
}
.navbar .item:hover {
background: var(--sidebar-accent) !important;
color: var(--sidebar-accent-foreground, #EEE9E0) !important;
}
.navbar .item.active {
background: var(--sidebar-accent) !important;
color: #fff !important;
}
/* Buttons */
.ui.button,
.button {
font-family: var(--font-body) !important;
font-weight: 500 !important;
border-radius: var(--radius, 0.2rem) !important;
transition: all 0.15s ease !important;
}
.ui.primary.button,
.ui.blue.button {
background: var(--primary) !important;
color: var(--primary-foreground) !important;
border: 1px solid var(--primary) !important;
}
.ui.primary.button:hover,
.ui.blue.button:hover {
background: var(--sidebar-accent) !important;
color: #fff !important;
}
.ui.green.button {
background: var(--chart-4) !important;
color: #fff !important;
border: 1px solid var(--chart-4) !important;
}
.ui.green.button:hover {
opacity: 0.85;
}
.ui.red.button {
background: var(--destructive) !important;
color: #fff !important;
}
/* Segments and boxes */
.ui.segment,
.ui.segments,
.repository,
.box {
background: var(--card) !important;
border: 1px solid var(--border) !important;
border-radius: var(--radius, 0.2rem) !important;
}
.ui.attached.header,
.box-header {
background: var(--secondary) !important;
border-bottom: 1px solid var(--border) !important;
font-family: var(--font-display) !important;
font-size: 13px !important;
color: var(--foreground) !important;
}
/* Tables */
.ui.table {
background: var(--card) !important;
border: 1px solid var(--border) !important;
}
.ui.table th {
background: var(--primary) !important;
color: var(--primary-foreground) !important;
font-family: var(--font-mono) !important;
font-size: 10px !important;
text-transform: uppercase !important;
border: 1px solid var(--sidebar-accent) !important;
}
.ui.table td {
background: var(--card) !important;
color: var(--foreground) !important;
border: 1px solid var(--border) !important;
}
.ui.table tr:hover td {
background: var(--secondary) !important;
}
/* Forms */
.ui.form input,
.ui.form textarea,
.ui.form select,
.ui.input input {
font-family: var(--font-body) !important;
background: var(--card) !important;
color: var(--foreground) !important;
border: 1px solid var(--border) !important;
border-radius: var(--radius, 0.2rem) !important;
}
.ui.form input:focus,
.ui.form textarea:focus,
.ui.input input:focus {
border-color: var(--primary) !important;
background: var(--card) !important;
}
/* Labels — mono for metadata labels (issue numbers, ref badges) */
.ui.label {
font-family: var(--font-mono) !important;
font-size: 11px !important;
border-radius: var(--radius, 0.2rem) !important;
}
/* Repo/issue titles — Playfair */
.repository .header,
.issue-title,
.repository h1,
.repository h2,
.repository h3 {
font-family: var(--font-display) !important;
font-weight: 700 !important;
}
/* PR and issue descriptions — serif body */
.comment-content,
.issue-content,
.render-content {
font-family: var(--font-body) !important;
font-size: 15px !important;
line-height: 1.65 !important;
}
/* Commit hashes, branch names, file paths — mono */
.commit-sha,
.sha,
[class*="sha"],
.branch-name,
.tag-name,
.file-name,
code.ref {
font-family: var(--font-mono) !important;
font-size: 12px !important;
}
/* Dropdowns */
.ui.dropdown .menu {
background: var(--card) !important;
border: 1px solid var(--border) !important;
}
.ui.dropdown .menu .item {
color: var(--foreground) !important;
}
.ui.dropdown .menu .item:hover {
background: var(--secondary) !important;
}
/* Code and diffs */
.code-view,
.file-view,
.CodeMirror,
.highlight {
font-family: var(--font-mono) !important;
background: var(--card) !important;
}
.diff-file-box .diff-file-header {
background: var(--secondary) !important;
}
/* Commit graph */
.repository .commit-list .commit {
border-bottom: 1px solid var(--border) !important;
}
/* Issues and PRs */
.issue-list .item,
.pull-list .item {
border-bottom: 1px solid var(--border) !important;
}
/* Footer */
.footer {
background: var(--secondary) !important;
border-top: 3px solid var(--foreground) !important;
}
/* Cards */
.ui.card,
.ui.cards .card {
background: var(--card) !important;
border: 1px solid var(--border) !important;
border-radius: var(--radius, 0.2rem) !important;
}
/* Messages */
.ui.message {
font-family: var(--font-body) !important;
border-radius: var(--radius, 0.2rem) !important;
}
.ui.positive.message,
.ui.success.message {
background: color-mix(in srgb, var(--chart-4) 12%, var(--background)) !important;
border: 1px solid var(--chart-4) !important;
color: var(--chart-4) !important;
}
.ui.negative.message,
.ui.error.message {
background: color-mix(in srgb, var(--destructive) 10%, var(--background)) !important;
border: 1px solid var(--destructive) !important;
color: var(--destructive) !important;
}
.ui.warning.message {
background: color-mix(in srgb, var(--chart-5) 10%, var(--background)) !important;
border: 1px solid var(--chart-5) !important;
color: var(--chart-5) !important;
}
.ui.info.message {
background: color-mix(in srgb, var(--primary) 10%, var(--background)) !important;
border: 1px solid var(--primary) !important;
color: var(--primary) !important;
}
/* Scrollbars */
::-webkit-scrollbar {
width: 10px;
height: 10px;
}
::-webkit-scrollbar-track {
background: var(--secondary);
}
::-webkit-scrollbar-thumb {
background: var(--border);
border-radius: 4px;
}
::-webkit-scrollbar-thumb:hover {
background: var(--muted-foreground);
}
/* Selection */
::selection {
background: var(--primary) !important;
color: var(--primary-foreground) !important;
}
/* Links */
a {
color: var(--primary) !important;
}
a:hover {
color: var(--foreground) !important;
}

77
compose.yml
View File

@@ -129,13 +129,30 @@ services:
- no-new-privileges:true
restart: unless-stopped
gitea:
image: gitea/gitea:1.25.4-rootless
container_name: gitea
git:
build:
context: ./infra/gitea
dockerfile: Dockerfile
args:
GITEA_VERSION: v1.25.4
# BUILD_TAG bumps `?v=...` on every asset URL → busts browser/CDN
# cache. Use BUILD_TAG=$(date +%s) when invoking docker compose build,
# or the Dockerfile defaults to current epoch.
BUILD_TAG: "${BUILD_TAG:-}"
image: fhirworx/git:v1.25.4
container_name: git
networks:
- gateway
- storage
- ci
env_file:
# Written by `stack mail wire-git` (or `stack mail provision`) from
# the mail droplet's credentials cache. Marked optional so cold-start
# works before the mail droplet exists; once it does, re-running
# wire-git populates this file and `docker compose up -d git`
# picks it up.
- path: .state/git/mailer.env
required: false
environment:
- GITEA__database__DB_TYPE=postgres
- GITEA__database__HOST=postgres:5432
@@ -165,13 +182,12 @@ services:
- GITEA__server__ROOT_URL=https://git.${DOMAIN:-fhirworx.io}/
- GITEA__server__SSH_DOMAIN=git.${DOMAIN:-fhirworx.io}
- GITEA__webhook__ALLOWED_HOST_LIST=woodpecker-server,ci.${DOMAIN:-fhirworx.io},${HOST_IP:-192.168.1.192},172.19.0.0/16
- GITEA__ui__THEMES=gitea-auto,gitea-light,gitea-dark,fhirworx
- GITEA__ui__THEMES=fhirworx,fhirworx-dark
- GITEA__ui__DEFAULT_THEME=fhirworx
volumes:
- gitea_data:/var/lib/gitea
- gitea_config:/etc/gitea
- ./infra/gitea/custom:/var/lib/gitea/custom
- ./assets/css/gitea.css:/var/lib/gitea/custom/public/assets/css/theme-fhirworx.css:ro
ports:
- "2222:2222"
- "3000:3000"
@@ -208,7 +224,7 @@ services:
- woodpecker_data:/var/lib/woodpecker
- ./assets/css/woodpecker.css:/etc/woodpecker/custom.css:ro
depends_on:
- gitea
- git
- postgres
restart: unless-stopped
@@ -240,7 +256,7 @@ services:
- ci
- storage
environment:
- GITEA_INSTANCE_URL=http://gitea:3000
- GITEA_INSTANCE_URL=http://git:3000
- GITEA_RUNNER_REGISTRATION_TOKEN=${ACT_RUNNER_TOKEN}
- GITEA_RUNNER_NAME=homelab-runner
- GITEA_RUNNER_LABELS=ubuntu-latest:docker://catthehacker/ubuntu:act-latest
@@ -250,7 +266,7 @@ services:
- act_runner_data:/data
- ./infra/act-runner/config.yaml:/config.yaml:ro
depends_on:
- gitea
- git
security_opt:
- no-new-privileges:true
restart: unless-stopped
@@ -276,8 +292,10 @@ services:
- POLARIS_ROOT_SECRET=${POLARIS_ROOT_SECRET}
volumes:
- ./notebooks:/home/kert/notebooks
- ./infra/marimo:/home/kert/.config/marimo
- ./infra/marimo/home-page-patched.js:/home/kert/workspace/.venv/lib/python3.13/site-packages/marimo/_static/assets/home-page-itW0tRmv.js:ro
# Mount only user-editable config files, not the whole infra/marimo
# directory — the theme and source lives inside the baked image now.
- ./infra/marimo/marimo.toml:/home/kert/.config/marimo/marimo.toml
- ./infra/marimo/snippets:/home/kert/.config/marimo/snippets:ro
- ./data:/home/kert/data
- ./data/zotero/data:/home/kert/zotero:ro
- ./src:/home/kert/src:ro
@@ -512,6 +530,39 @@ services:
- no-new-privileges:true
restart: unless-stopped
# IMAP → bib poller. Pulls UNSEEN mail from `cmsupdates@mail.fhirworx.io`
# every MAIL_POLL_INTERVAL seconds and upserts each as a Source item.
# Idempotent (server-side `\Seen` flag), so frequency is purely a
# latency knob — 600s = ~10min average from arrival → bib.
mail-poller:
image: ${IMAGE_PREFIX:-fhirworx}/api:${COMMIT_SHA:-latest}
pull_policy: if_not_present
container_name: mail-poller
# Needs `gateway` for outbound DNS + IMAPS to mail.fhirworx.io;
# `data` to share bib.sqlite with the api/lake services.
networks:
- gateway
- data
environment:
- MAIL_POLL_INTERVAL=${MAIL_POLL_INTERVAL:-600}
volumes:
- ./data:/app/data
- ./.state:/app/.state
# Mount source live so adding mailboxes / tweaking the poller
# doesn't require an image rebuild — the api image's baked venv
# provides interpreter + deps; --no-sync keeps `uv run` from
# going back to pypi.
- ./src:/app/src:ro
- ./pyproject.toml:/app/pyproject.toml:ro
command: >
sh -c 'while true; do
uv run --no-sync stack bib ingest-mail || true;
sleep $${MAIL_POLL_INTERVAL};
done'
security_opt:
- no-new-privileges:true
restart: unless-stopped
# Observability Stack
jaeger:
image: jaegertracing/all-in-one:latest
@@ -709,8 +760,8 @@ services:
- OAUTH2_PROXY_OIDC_ISSUER_URL=https://git.${DOMAIN:-fhirworx.io}/
- OAUTH2_PROXY_SKIP_OIDC_DISCOVERY=true
- OAUTH2_PROXY_LOGIN_URL=https://git.${DOMAIN:-fhirworx.io}/login/oauth/authorize
- OAUTH2_PROXY_REDEEM_URL=http://gitea:3000/login/oauth/access_token
- OAUTH2_PROXY_OIDC_JWKS_URL=http://gitea:3000/login/oauth/keys
- OAUTH2_PROXY_REDEEM_URL=http://git:3000/login/oauth/access_token
- OAUTH2_PROXY_OIDC_JWKS_URL=http://git:3000/login/oauth/keys
- OAUTH2_PROXY_INSECURE_OIDC_SKIP_ISSUER_VERIFICATION=true
- OAUTH2_PROXY_REDIRECT_URL=https://auth.${DOMAIN:-fhirworx.io}/oauth2/callback
- OAUTH2_PROXY_COOKIE_DOMAINS=.${DOMAIN:-fhirworx.io}
@@ -724,7 +775,7 @@ services:
- OAUTH2_PROXY_SKIP_PROVIDER_BUTTON=true
- OAUTH2_PROXY_CUSTOM_SIGN_IN_LOGO=-
depends_on:
- gitea
- git
healthcheck:
test: ["CMD", "oauth2-proxy", "--version"]
interval: 30s

20
dev/scripts/add_carrier_to_zotero.py
View File

@@ -33,20 +33,24 @@ CARRIER_FILES = {
},
}
def add_carrier_year(db: Db, year: int, info: dict) -> None:
"""Add one carrier year to Zotero: parent item + file attachments."""
now = now_iso()
# Create parent webpage item
parent_id = db.create_item(TYPE_MAP["webpage"], now=now)
db.set_fields(parent_id, {
"title": info["title"],
"date": f"{year}-01-01",
"url": info["url"],
"accessDate": now,
"websiteType": "Government Data Portal",
"websiteTitle": "Centers for Medicare & Medicaid Services",
})
db.set_fields(
parent_id,
{
"title": info["title"],
"date": f"{year}-01-01",
"url": info["url"],
"accessDate": now,
"websiteType": "Government Data Portal",
"websiteTitle": "Centers for Medicare & Medicaid Services",
},
)
db.sync_tags(parent_id, ["module:pfs", f"year:{year}"])
print(f"Created parent item for {info['title']}")

140
dev/scripts/add_cf_history_to_zotero.py Normal file
View File

@@ -0,0 +1,140 @@
"""Add the AMA Medicare Physician CF history PDF to Zotero.
The document is the authoritative public source for the historical
record of CMS PFS conversion factors from CY1992 through present,
including the four-CF split (QP APM / non-APM / Anesthesia × 2)
that took effect CY2026. It explains that the Final Rule's
conversion factor for a given year is derived from the prior year's
CF via update factor × budget-neutrality adjustor × performance
adjustment, and then **baked into** the published value — which is
why ``pfs.rules.RULES[year].conversion_factor`` is already BN-
adjusted and should never be multiplied by a separate BN term at
payment-calculation time.
Source:
https://www.ama-assn.org/system/files/cf-history.pdf
Tags:
- module:pfs
- source:ama
- file:cf-history
- year:1992 … year:2026 (range covered)
Usage:
uv run python dev/scripts/add_cf_history_to_zotero.py [--pdf PATH]
The PDF is downloaded from AMA if no local path is supplied.
"""
from __future__ import annotations
import argparse
import subprocess
import urllib.request
from pathlib import Path
from conf import path as _conf_path
from zot.db import TYPE_MAP, Db, generate_key, now_iso
ZOTERO_DB = str(_conf_path("db.zotero"))
ZOTERO_STORAGE = str(_conf_path("storage.zotero"))
CF_HISTORY_URL = "https://www.ama-assn.org/system/files/cf-history.pdf"
CF_HISTORY_TITLE = "History of Medicare Physician Payment Schedule Conversion Factors"
CF_HISTORY_YEARS = range(1992, 2027) # 1992 → 2026
def _download(dest: Path) -> None:
"""Fetch the PDF from AMA if missing."""
if dest.exists() and dest.stat().st_size > 0:
print(f"Using existing {dest}")
return
dest.parent.mkdir(parents=True, exist_ok=True)
print(f"Downloading {CF_HISTORY_URL} → {dest}")
urllib.request.urlretrieve(CF_HISTORY_URL, dest) # noqa: S310
print(f" {dest.stat().st_size:,} bytes")
def _add_parent_item(db: Db) -> int:
"""Create the parent report item + tags. Returns item id."""
now = now_iso()
item_id = db.create_item(TYPE_MAP["report"], now=now)
db.set_fields(
item_id,
{
"title": CF_HISTORY_TITLE,
"date": "2026-01-01",
"url": CF_HISTORY_URL,
"accessDate": now,
"institution": "American Medical Association",
"reportType": "Data Provenance",
"abstractNote": (
"Authoritative historical record of the Medicare Physician "
"Payment Schedule conversion factor from CY1992 through the "
"present. For CY2026, CMS finalised FOUR conversion factors "
"(APM / non-APM × standard / anesthesia) under MACRA 2015. "
"The published CF for each year already reflects the "
"statutory update factor, budget-neutrality adjustor, and "
"any performance adjustment — callers should NOT apply BN "
"as a separate multiplier at payment-calculation time."
),
},
)
tags = ["module:pfs", "source:ama", "file:cf-history"] + [
f"year:{y}" for y in CF_HISTORY_YEARS
]
db.sync_tags(item_id, tags)
print(f"Created parent item {item_id} with {len(tags)} tags")
return item_id
def _attach_pdf(db: Db, parent_id: int, pdf_path: Path) -> None:
"""Copy the PDF into Zotero storage and create the attachment."""
att_key = generate_key()
storage_dir = Path(ZOTERO_STORAGE) / att_key
subprocess.run(["sudo", "mkdir", "-p", str(storage_dir)], check=True)
dest = storage_dir / pdf_path.name
subprocess.run(["sudo", "cp", str(pdf_path), str(dest)], check=True)
subprocess.run(
["sudo", "chown", "-R", "100999:100999", str(storage_dir)],
check=True,
)
att_id = db.add_attachment(
parent_id,
key=att_key,
content_type="application/pdf",
path=f"storage:{pdf_path.name}",
)
db.set_field(att_id, "title", pdf_path.name)
print(f"Attached {pdf_path.name} (attachment id {att_id}, key {att_key})")
def main() -> None:
ap = argparse.ArgumentParser(description=__doc__ or "")
ap.add_argument(
"--pdf",
type=Path,
default=Path("/tmp/cf_history/cf-history.pdf"),
help="Local path to cf-history.pdf (downloads from AMA if missing).",
)
args = ap.parse_args()
_download(args.pdf)
with Db(ZOTERO_DB) as db:
parent_id = _add_parent_item(db)
_attach_pdf(db, parent_id, args.pdf)
parent_key = db.con.execute(
"SELECT key FROM items WHERE itemID = ?", (parent_id,)
).fetchone()[0]
db.commit()
print()
print(f"Done. Parent item key: {parent_key}")
print("Cite in docstrings with:")
print(f" :pincite:`{parent_key}`")
if __name__ == "__main__":
main()

164
dev/scripts/add_pprrvu_historical_to_zotero.py Normal file
View File

@@ -0,0 +1,164 @@
"""Backfill ALL quarterly PPRRVU releases for CY2015–CY2024 into Zotero.
CMS publishes a fresh PPRRVU file each calendar quarter. Sometimes
the CF and/or RVUs carry the same values across all four quarters;
sometimes a mid-year CAA retroactive update changes them. Example:
2015-2023: CF constant across Q1/Q2/Q3/Q4 for a given year.
2024: Q1 CF = 32.7442 (original Final Rule),
Q2/Q3/Q4 CF = 33.2875 (after CAA 2024 adjustment).
We archive ALL four quarters per year to preserve provenance and let
downstream code pick the release that matches a given carrier file's
publication date. See :pincite:`VVBEVYLC` — the AMA CF history table
— for the authoritative summary across years.
Each year's four releases are added to Zotero as a SINGLE parent
webpage item with ``year:YYYY`` + ``file:rvu`` + ``source:cms-website``
+ ``release:q1/q2/q3/q4`` tags; every release's xlsx/csv/txt trio
becomes an attachment on that parent.
Source zip URLs live in ``/tmp/rvu_urls_final.txt`` (tab-separated
``year\\tquarter\\turl``) produced by the CMS PFS RVU index scraper.
Usage:
uv run python dev/scripts/add_pprrvu_historical_to_zotero.py
Assumes the zips have already been downloaded + extracted to
``/tmp/pprrvu_dl/all/extracted/{year}{q}/``.
"""
from __future__ import annotations
import subprocess
from pathlib import Path
from conf import path as _conf_path
from zot.db import TYPE_MAP, Db, generate_key, now_iso
ZOTERO_DB = str(_conf_path("db.zotero"))
ZOTERO_STORAGE = str(_conf_path("storage.zotero"))
URL_TSV = Path("/tmp/rvu_urls_final.txt")
EXTRACTED_ROOT = Path("/tmp/pprrvu_dl/all/extracted")
QUARTER_LABEL = {"a": "Q1", "b": "Q2", "c": "Q3", "d": "Q4"}
def _load_url_map() -> dict[int, dict[str, str]]:
"""Parse the tab-separated URL list into {year: {quarter: url}}."""
urls: dict[int, dict[str, str]] = {}
for line in URL_TSV.read_text().splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
parts = line.split("\t")
if len(parts) != 3:
continue
year, q, url = parts
urls.setdefault(int(year), {})[q] = url
return urls
def add_quarter(db: Db, year: int, q: str, url: str) -> int:
"""Create one parent item per (year, quarter).
Unambiguous ``release:q1..q4`` tag on the parent is what the
pipe loader uses to dedupe to the latest quarter per year.
"""
now = now_iso()
parent_id = db.create_item(TYPE_MAP["webpage"], now=now)
db.set_fields(
parent_id,
{
"title": (
f"CY {year} PFS Q{'abcd'.index(q) + 1} — PPRRVU "
f"({QUARTER_LABEL[q]} release)"
),
"date": f"{year}-01-01",
"url": url,
"accessDate": now,
"websiteType": "Government Data Portal",
"websiteTitle": "Centers for Medicare & Medicaid Services",
},
)
db.sync_tags(
parent_id,
[
"module:pfs",
"file:rvu",
"source:cms-website",
f"year:{year}",
f"release:{QUARTER_LABEL[q].lower()}",
],
)
extracted_dir = EXTRACTED_ROOT / f"{year}{q}"
if not extracted_dir.exists():
print(f" {year} {QUARTER_LABEL[q]}: no extracted dir")
return parent_id
attached = 0
# Walk recursively; older zips nest files under a subfolder.
for filepath in sorted(extracted_dir.rglob("*")):
if not filepath.is_file():
continue
fn = filepath.name
if "PPRRVU" not in fn.upper():
continue
if filepath.suffix.lower() not in (".xlsx", ".csv", ".txt"):
continue
att_key = generate_key()
storage_dir = Path(ZOTERO_STORAGE) / att_key
subprocess.run(["sudo", "mkdir", "-p", str(storage_dir)], check=True)
dest = storage_dir / fn
subprocess.run(["sudo", "cp", str(filepath), str(dest)], check=True)
subprocess.run(
["sudo", "chown", "-R", "100999:100999", str(storage_dir)],
check=True,
)
ext = filepath.suffix.lower()
content_type = {
".xlsx": (
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
),
".csv": "text/csv",
".txt": "text/plain",
}[ext]
att_id = db.add_attachment(
parent_id,
key=att_key,
content_type=content_type,
path=f"storage:{fn}",
)
db.set_field(att_id, "title", fn)
attached += 1
print(f" {year} {QUARTER_LABEL[q]}: parent {parent_id}, {attached} files")
return parent_id
def main() -> None:
urls = _load_url_map()
if not urls:
raise SystemExit(f"No URLs parsed from {URL_TSV}")
if not EXTRACTED_ROOT.exists():
raise SystemExit(
f"Extracted dir {EXTRACTED_ROOT} not found — download+unzip the zips first."
)
with Db(ZOTERO_DB) as db:
for year in sorted(urls):
for q in sorted(urls[year]):
add_quarter(db, year, q, urls[year][q])
db.commit()
print("\nDone. Restart Zotero + re-run pfs ingestion.")
if __name__ == "__main__":
main()

23
dev/scripts/add_zipcode_to_zotero.py
View File

@@ -7,13 +7,12 @@ Usage:
uv run python dev/scripts/add_zipcode_to_zotero.py
"""
import sqlite3
import subprocess
import zipfile
from pathlib import Path
from conf import path as _conf_path
from zot.db import FIELD_MAP, TYPE_MAP, Db, generate_key, now_iso
from zot.db import TYPE_MAP, Db, generate_key, now_iso
ZOTERO_DB = str(_conf_path("db.zotero"))
ZOTERO_STORAGE = str(_conf_path("storage.zotero"))
@@ -76,6 +75,7 @@ ZIPCODE_FILES = {
},
}
def add_zipcode_year(db: Db, year: int, info: dict) -> None:
now = now_iso()
@@ -87,14 +87,17 @@ def add_zipcode_year(db: Db, year: int, info: dict) -> None:
# Create parent webpage item
parent_id = db.create_item(TYPE_MAP["webpage"], now=now)
db.set_fields(parent_id, {
"title": info["title"],
"date": f"{year}-01-01",
"url": info["url"],
"accessDate": now,
"websiteType": "Government Data Portal",
"websiteTitle": "Centers for Medicare & Medicaid Services",
})
db.set_fields(
parent_id,
{
"title": info["title"],
"date": f"{year}-01-01",
"url": info["url"],
"accessDate": now,
"websiteType": "Government Data Portal",
"websiteTitle": "Centers for Medicare & Medicaid Services",
},
)
# Tags: module:pfs + year:YYYY
db.sync_tags(parent_id, ["module:pfs", f"year:{year}"])

4
dev/scripts/backends/gitea.py
View File

@@ -135,7 +135,9 @@ def _docker_login_step(registry: str) -> str:
# ── Workflow generators ──────────────────────────────────────────
def _gen_ci(runner: str, uv_version: str, coverage_threshold: int = 99, **_kw: object) -> tuple[str, str]:
def _gen_ci(
runner: str, uv_version: str, coverage_threshold: int = 99, **_kw: object
) -> tuple[str, str]:
content = f"""\
{_HEADER}
name: CI

4
dev/scripts/backends/github.py
View File

@@ -129,7 +129,9 @@ def _failure_step(workflow_name: str, job_name: str) -> str:
# ── Workflow generators ───────────────────────────────────────────
def _gen_ci(runner: str, uv_version: str, coverage_threshold: int = 99, **_kw: object) -> tuple[str, str]:
def _gen_ci(
runner: str, uv_version: str, coverage_threshold: int = 99, **_kw: object
) -> tuple[str, str]:
content = f"""\
{_HEADER}
name: CI

118
dev/scripts/bootstrap_sso.py
View File

@@ -38,6 +38,7 @@ def _headers(token: str) -> dict:
# State persistence
# ---------------------------------------------------------------------------
def _load_state() -> dict:
if OAUTH_STATE.exists():
return json.loads(OAUTH_STATE.read_text())
@@ -65,6 +66,7 @@ def _write_env_file(path: Path, content: str) -> bool:
# Admin user
# ---------------------------------------------------------------------------
def _ensure_admin(client: httpx.Client) -> str | None:
r = client.get(f"{GITEA_API}/user", auth=(ADMIN_USER, ADMIN_PASS))
if r.status_code == 200:
@@ -72,22 +74,45 @@ def _ensure_admin(client: httpx.Client) -> str | None:
elif r.status_code == 401:
# Try to create; if user exists, change the password instead
result = subprocess.run(
["docker", "exec", "gitea", "gitea", "admin", "user", "create",
"--username", ADMIN_USER,
"--password", ADMIN_PASS,
"--email", f"{ADMIN_USER}@{DOMAIN}",
"--admin"],
capture_output=True, text=True,
[
"docker",
"exec",
"gitea",
"gitea",
"admin",
"user",
"create",
"--username",
ADMIN_USER,
"--password",
ADMIN_PASS,
"--email",
f"{ADMIN_USER}@{DOMAIN}",
"--admin",
],
capture_output=True,
text=True,
)
if result.returncode == 0:
print(f" ok: admin '{ADMIN_USER}' created")
elif "already exists" in result.stderr:
subprocess.run(
["docker", "exec", "gitea", "gitea", "admin", "user",
"change-password", "--username", ADMIN_USER,
"--password", ADMIN_PASS,
"--must-change-password=false"],
capture_output=True, text=True,
[
"docker",
"exec",
"gitea",
"gitea",
"admin",
"user",
"change-password",
"--username",
ADMIN_USER,
"--password",
ADMIN_PASS,
"--must-change-password=false",
],
capture_output=True,
text=True,
)
print(f" ok: admin '{ADMIN_USER}' password synced")
else:
@@ -139,8 +164,12 @@ def _ensure_token(client: httpx.Client) -> str | None:
# OAuth2 application
# ---------------------------------------------------------------------------
def _ensure_oauth_app(
client: httpx.Client, token: str, name: str, redirect_uri: str,
client: httpx.Client,
token: str,
name: str,
redirect_uri: str,
) -> tuple[str, str] | None:
headers = _headers(token)
state = _load_state()
@@ -191,6 +220,7 @@ def _ensure_oauth_app(
# Downstream wiring
# ---------------------------------------------------------------------------
def _write_oauth2_proxy_env(client_id: str, client_secret: str) -> bool:
state = _load_state()
cookie_secret = state.get("oauth2_proxy_cookie_secret")
@@ -198,11 +228,14 @@ def _write_oauth2_proxy_env(client_id: str, client_secret: str) -> bool:
cookie_secret = secrets.token_hex(16)
state["oauth2_proxy_cookie_secret"] = cookie_secret
_save_state(state)
return _write_env_file(OAUTH2_PROXY_ENV, (
f"OAUTH2_PROXY_CLIENT_ID={client_id}\n"
f"OAUTH2_PROXY_CLIENT_SECRET={client_secret}\n"
f"OAUTH2_PROXY_COOKIE_SECRET={cookie_secret}\n"
))
return _write_env_file(
OAUTH2_PROXY_ENV,
(
f"OAUTH2_PROXY_CLIENT_ID={client_id}\n"
f"OAUTH2_PROXY_CLIENT_SECRET={client_secret}\n"
f"OAUTH2_PROXY_COOKIE_SECRET={cookie_secret}\n"
),
)
# ---------------------------------------------------------------------------
@@ -210,9 +243,26 @@ def _write_oauth2_proxy_env(client_id: str, client_secret: str) -> bool:
# ---------------------------------------------------------------------------
SUBDOMAINS = [
"", "dashboard", "docs", "git", "ci", "notebooks", "zotero",
"webdav", "api", "nessie", "trino", "polaris", "grafana",
"prometheus", "jaeger", "loki", "s3", "s3console", "traefik", "auth",
"",
"dashboard",
"docs",
"git",
"ci",
"notebooks",
"zotero",
"webdav",
"api",
"nessie",
"trino",
"polaris",
"grafana",
"prometheus",
"jaeger",
"loki",
"s3",
"s3console",
"traefik",
"auth",
]
@@ -261,16 +311,21 @@ def _sync_tunnel_dns(client: httpx.Client) -> None:
ingress = []
for sub in SUBDOMAINS:
hostname = f"{sub}.{DOMAIN}" if sub else DOMAIN
ingress.append({
"hostname": hostname,
"service": "http://traefik:80",
"originRequest": {},
})
ingress.append(
{
"hostname": hostname,
"service": "http://traefik:80",
"originRequest": {},
}
)
ingress.append({"service": "http_status:404", "originRequest": {}})
r = client.put(
api,
headers={"Authorization": f"Bearer {token}", "Content-Type": "application/json"},
headers={
"Authorization": f"Bearer {token}",
"Content-Type": "application/json",
},
json={"config": {"ingress": ingress}},
)
if r.status_code == 200 and r.json().get("success"):
@@ -283,6 +338,7 @@ def _sync_tunnel_dns(client: httpx.Client) -> None:
# Entry point
# ---------------------------------------------------------------------------
def main() -> None:
if not ADMIN_PASS:
print("ERROR: GITEA_ADMIN_PASSWORD not set")
@@ -291,6 +347,7 @@ def main() -> None:
print("==> Waiting for Gitea...")
with httpx.Client(timeout=10) as client:
import time
for _ in range(60):
try:
r = client.get(f"{GITEA_API}/settings/api")
@@ -309,7 +366,9 @@ def main() -> None:
sys.exit(1)
creds = _ensure_oauth_app(
client, token, "platform-sso",
client,
token,
"platform-sso",
f"https://auth.{DOMAIN}/oauth2/callback",
)
if creds:
@@ -345,7 +404,10 @@ def _ensure_dns(client: httpx.Client) -> None:
zone = "f8553bde1ddb415b8c3e5dbec4b28330"
tunnel = "1389035e-d3ba-4a4f-969d-a369c07ee057"
tunnel_cname = f"{tunnel}.cfargotunnel.com"
headers = {"Authorization": f"Bearer {cf_token}", "Content-Type": "application/json"}
headers = {
"Authorization": f"Bearer {cf_token}",
"Content-Type": "application/json",
}
# Fetch existing records
r = client.get(

20
dev/scripts/build_product_susceptibility.py
View File

@@ -27,7 +27,7 @@ def main() -> None:
con = duckdb.connect(str(DUCKDB_PATH))
con.execute("DROP TABLE IF EXISTS skin_subs.product_susceptibility")
con.execute(f"""
con.execute("""
CREATE TABLE skin_subs.product_susceptibility AS
WITH product_base AS (
SELECT
@@ -198,7 +198,9 @@ def main() -> None:
FROM skin_subs.product_susceptibility
GROUP BY susceptibility_tier ORDER BY avg_score DESC
""").fetchall():
print(f" {r[0]:10s} n={r[1]:3d} avg_score={r[2]:6.3f} avg_asp=${r[3] or 0:>8}")
print(
f" {r[0]:10s} n={r[1]:3d} avg_score={r[2]:6.3f} avg_asp=${r[3] or 0:>8}"
)
# Top 20 most susceptible
print("\n Top 20 most susceptible products:")
@@ -210,9 +212,11 @@ def main() -> None:
WHERE susceptibility_score IS NOT NULL
ORDER BY susceptibility_score DESC LIMIT 20
""").fetchall():
print(f" {r[0]} {(r[1] or ''):25s} {(r[2] or ''):20s} "
f"score={r[4]:5.3f} ({r[5]}) asp=${r[6] or 0:>8} "
f"rcts={r[7]:2d} provs={r[8]:3d} office={r[9] or 0}%")
print(
f" {r[0]} {(r[1] or ''):25s} {(r[2] or ''):20s} "
f"score={r[4]:5.3f} ({r[5]}) asp=${r[6] or 0:>8} "
f"rcts={r[7]:2d} provs={r[8]:3d} office={r[9] or 0}%"
)
# Cross-reference: do susceptible products appear in enforcement?
print("\n Susceptibility by category:")
@@ -227,8 +231,10 @@ def main() -> None:
HAVING count(*) >= 2
ORDER BY avg_score DESC
""").fetchall():
print(f" {(r[0] or ''):40s} n={r[1]:3d} score={r[2]:6.3f} "
f"asp=${r[3] or 0:>8} rcts={r[4]:3d}")
print(
f" {(r[0] or ''):40s} n={r[1]:3d} score={r[2]:6.3f} "
f"asp=${r[3] or 0:>8} rcts={r[4]:3d}"
)
# Final table count
print("\n All skin_subs tables:")

60
dev/scripts/build_skin_subs_analytics.py
View File

@@ -93,23 +93,27 @@ def build_asp_trajectories(con: duckdb.DuckDBPyConnection) -> None:
ORDER BY hcpcs_code, quarter
""")
count = con.execute(
"SELECT count(*) FROM skin_subs.asp_trajectories"
).fetchone()[0]
count = con.execute("SELECT count(*) FROM skin_subs.asp_trajectories").fetchone()[0]
print(f" Rows: {count}")
# Summary stats
for label, q in [
("Products with anomalous spikes",
"SELECT count(DISTINCT hcpcs_code) FROM skin_subs.asp_trajectories WHERE anomalous_spike"),
("Products that lose under flat rate (latest quarter)",
"""SELECT count(DISTINCT hcpcs_code) FROM skin_subs.asp_trajectories
(
"Products with anomalous spikes",
"SELECT count(DISTINCT hcpcs_code) FROM skin_subs.asp_trajectories WHERE anomalous_spike",
),
(
"Products that lose under flat rate (latest quarter)",
"""SELECT count(DISTINCT hcpcs_code) FROM skin_subs.asp_trajectories
WHERE flat_rate_impact = 'loses'
AND quarter = (SELECT max(quarter) FROM skin_subs.asp_trajectories)"""),
("Products that gain under flat rate (latest quarter)",
"""SELECT count(DISTINCT hcpcs_code) FROM skin_subs.asp_trajectories
AND quarter = (SELECT max(quarter) FROM skin_subs.asp_trajectories)""",
),
(
"Products that gain under flat rate (latest quarter)",
"""SELECT count(DISTINCT hcpcs_code) FROM skin_subs.asp_trajectories
WHERE flat_rate_impact = 'gains'
AND quarter = (SELECT max(quarter) FROM skin_subs.asp_trajectories)"""),
AND quarter = (SELECT max(quarter) FROM skin_subs.asp_trajectories)""",
),
]:
val = con.execute(q).fetchone()[0]
print(f" {label}: {val}")
@@ -123,7 +127,9 @@ def build_asp_trajectories(con: duckdb.DuckDBPyConnection) -> None:
WHERE quarter = (SELECT max(quarter) FROM skin_subs.asp_trajectories)
ORDER BY flat_rate_delta DESC LIMIT 10
""").fetchall():
print(f" {r[0]} {(r[1] or ''):30s} {(r[2] or ''):20s} pay=${r[3]:>8} delta=${r[4]:>+8}")
print(
f" {r[0]} {(r[1] or ''):30s} {(r[2] or ''):20s} pay=${r[3]:>8} delta=${r[4]:>+8}"
)
def build_market_segmentation(con: duckdb.DuckDBPyConnection) -> None:
@@ -202,9 +208,9 @@ def build_market_segmentation(con: duckdb.DuckDBPyConnection) -> None:
ORDER BY cs.total_paid DESC NULLS LAST
""")
count = con.execute(
"SELECT count(*) FROM skin_subs.products_enriched"
).fetchone()[0]
count = con.execute("SELECT count(*) FROM skin_subs.products_enriched").fetchone()[
0
]
print(f" Products: {count}")
# Manufacturer summary
@@ -236,7 +242,9 @@ def build_market_segmentation(con: duckdb.DuckDBPyConnection) -> None:
ORDER BY total_revenue DESC LIMIT 10
""").fetchall():
rev = f"${r[3]:,.0f}" if r[3] else "n/a"
print(f" {(r[0] or ''):30s} prods={r[1]:3d} active={r[2]:3d} rev={rev:>12} avg_asp=${r[4] or 0:>8}")
print(
f" {(r[0] or ''):30s} prods={r[1]:3d} active={r[2]:3d} rev={rev:>12} avg_asp=${r[4] or 0:>8}"
)
print("\n Products by category:")
for r in con.execute("""
@@ -247,7 +255,9 @@ def build_market_segmentation(con: duckdb.DuckDBPyConnection) -> None:
WHERE category IS NOT NULL
GROUP BY category ORDER BY n DESC
""").fetchall():
print(f" {(r[0] or ''):40s} n={r[1]:3d} claims={r[2]:3d} avg_asp=${r[3] or 0:>8}")
print(
f" {(r[0] or ''):40s} n={r[1]:3d} claims={r[2]:3d} avg_asp=${r[3] or 0:>8}"
)
def build_utilization_metrics(con: duckdb.DuckDBPyConnection) -> None:
@@ -327,8 +337,12 @@ def build_utilization_metrics(con: duckdb.DuckDBPyConnection) -> None:
# Print summaries
prov_count = con.execute("SELECT count(*) FROM skin_subs.providers").fetchone()[0]
bene_count = con.execute("SELECT count(*) FROM skin_subs.beneficiaries").fetchone()[0]
util_count = con.execute("SELECT count(*) FROM skin_subs.utilization_summary").fetchone()[0]
bene_count = con.execute("SELECT count(*) FROM skin_subs.beneficiaries").fetchone()[
0
]
util_count = con.execute(
"SELECT count(*) FROM skin_subs.utilization_summary"
).fetchone()[0]
print(f" Providers: {prov_count}")
print(f" Beneficiaries: {bene_count}")
print(f" Utilization summary rows: {util_count}")
@@ -339,7 +353,9 @@ def build_utilization_metrics(con: duckdb.DuckDBPyConnection) -> None:
unique_patients, total_paid, paid_per_patient
FROM skin_subs.providers ORDER BY total_paid DESC LIMIT 10
""").fetchall():
print(f" NPI {r[0]} {r[1]:15s} {r[2]} pts={r[3]:3d} paid=${r[4]:>10,.2f} per_pt=${r[5]:>8,.2f}")
print(
f" NPI {r[0]} {r[1]:15s} {r[2]} pts={r[3]:3d} paid=${r[4]:>10,.2f} per_pt=${r[5]:>8,.2f}"
)
print("\n Utilization by setting:")
for r in con.execute("""
@@ -348,7 +364,9 @@ def build_utilization_metrics(con: duckdb.DuckDBPyConnection) -> None:
FROM skin_subs.utilization_summary
GROUP BY setting ORDER BY paid DESC
""").fetchall():
print(f" {r[0]:15s} lines={r[1]:5d} paid=${r[2]:>12,.2f} per_bene=${r[3]:>8,.2f}")
print(
f" {r[0]:15s} lines={r[1]:5d} paid=${r[2]:>12,.2f} per_bene=${r[3]:>8,.2f}"
)
print("\n Utilization by specialty:")
for r in con.execute("""

41
dev/scripts/build_skin_subs_anomaly.py
View File

@@ -9,7 +9,6 @@ Usage:
from __future__ import annotations
import math
from pathlib import Path
import duckdb
@@ -41,10 +40,10 @@ KNOWN_FRAUD_PATTERNS = {
# Setting risk weights (higher = more fraud-prone based on OIG findings)
SETTING_RISK = {
"office": 1.5, # Lowest oversight, wound care mills
"snf": 1.3, # Kickback-prone, captive patients
"asc": 0.8, # Moderate oversight
"hopd": 0.5, # Institutional controls, auditable
"office": 1.5, # Lowest oversight, wound care mills
"snf": 1.3, # Kickback-prone, captive patients
"asc": 0.8, # Moderate oversight
"hopd": 0.5, # Institutional controls, auditable
}
@@ -120,10 +119,10 @@ def build_anomaly_scores(con: duckdb.DuckDBPyConnection) -> None:
ELSE 0 END AS units_zscore,
-- Setting risk weight
CASE ps.primary_setting
WHEN 'office' THEN {SETTING_RISK['office']}
WHEN 'snf' THEN {SETTING_RISK['snf']}
WHEN 'asc' THEN {SETTING_RISK['asc']}
WHEN 'hopd' THEN {SETTING_RISK['hopd']}
WHEN 'office' THEN {SETTING_RISK["office"]}
WHEN 'snf' THEN {SETTING_RISK["snf"]}
WHEN 'asc' THEN {SETTING_RISK["asc"]}
WHEN 'hopd' THEN {SETTING_RISK["hopd"]}
ELSE 1.0 END AS setting_risk,
-- Geographic risk (state per-bene spend z-score)
(SELECT round((g.paid_per_bene - agg.m) / nullif(agg.s, 0), 2)
@@ -178,7 +177,9 @@ def build_anomaly_scores(con: duckdb.DuckDBPyConnection) -> None:
FROM skin_subs.anomaly_scores
GROUP BY risk_tier ORDER BY avg_score DESC
""").fetchall():
print(f" {r[0]:10s} n={r[1]:3d} avg_paid=${r[2]:>10,.2f} avg_score={r[3]}")
print(
f" {r[0]:10s} n={r[1]:3d} avg_paid=${r[2]:>10,.2f} avg_score={r[3]}"
)
# Top risk providers
print("\n Top 10 highest-risk providers:")
@@ -189,9 +190,11 @@ def build_anomaly_scores(con: duckdb.DuckDBPyConnection) -> None:
FROM skin_subs.anomaly_scores
ORDER BY composite_risk DESC LIMIT 10
""").fetchall():
print(f" NPI {r[0]} {r[1]:15s} {r[2]} {r[3]:8s} "
f"paid=${r[4]:>10,.2f} per_pt=${r[5]:>8,.2f} "
f"risk={r[6]:5.2f} ({r[7]}) vol_z={r[8]:+5.2f} int_z={r[9]:+5.2f}")
print(
f" NPI {r[0]} {r[1]:15s} {r[2]} {r[3]:8s} "
f"paid=${r[4]:>10,.2f} per_pt=${r[5]:>8,.2f} "
f"risk={r[6]:5.2f} ({r[7]}) vol_z={r[8]:+5.2f} int_z={r[9]:+5.2f}"
)
def build_enforcement_correlation(con: duckdb.DuckDBPyConnection) -> None:
@@ -258,7 +261,9 @@ def build_enforcement_correlation(con: duckdb.DuckDBPyConnection) -> None:
WHERE fraud_pattern_match IS NOT NULL
GROUP BY fraud_pattern_match ORDER BY avg_risk DESC
""").fetchall():
print(f" {r[0]:15s} n={r[1]:3d} avg_risk={r[2]:5.2f} avg_paid=${r[3]:>10,.2f}")
print(
f" {r[0]:15s} n={r[1]:3d} avg_risk={r[2]:5.2f} avg_paid=${r[3]:>10,.2f}"
)
# Do fraud-pattern providers score higher than peers?
print("\n Risk scores: fraud-pattern vs non-match:")
@@ -273,10 +278,14 @@ def build_enforcement_correlation(con: duckdb.DuckDBPyConnection) -> None:
FROM skin_subs.provider_risk_profile
GROUP BY group_name
""").fetchall():
print(f" {r[0]:10s} n={r[1]:3d} risk={r[2]:6.3f} vol_z={r[3]:+6.3f} int_z={r[4]:+6.3f}")
print(
f" {r[0]:10s} n={r[1]:3d} risk={r[2]:6.3f} vol_z={r[3]:+6.3f} int_z={r[4]:+6.3f}"
)
# Benford analysis summary
print("\n Benford's law chi² (lower = more conformant, >15.5 = suspicious at p<0.05):")
print(
"\n Benford's law chi² (lower = more conformant, >15.5 = suspicious at p<0.05):"
)
for r in con.execute("""
SELECT risk_tier, count(*) as n,
round(avg(benford_chi2), 4) as avg_chi2,

126
dev/scripts/build_skin_subs_evidence_base.py
View File

@@ -56,26 +56,73 @@ SKIN_SUBS_COLLECTIONS = {
# ---------------------------------------------------------------------------
MANUFACTURERS = [
"organogenesis", "mimedx", "smith nephew", "smith & nephew",
"integra", "solsys", "amnioexcel", "derma sciences",
"healthpoint", "shire", "acelity", "kci", "3m",
"molnlycke", "medline", "hollister", "coloplast",
"stryker", "zimmer biomet", "wright medical",
"solventum", "apria", "anika", "musculoskeletal transplant",
"surmodics", "tissue regenix", "nuo therapeutics",
"sanara medtech", "kerecis", "human bioprocessing",
"alphatec", "biosig technologies",
"organogenesis",
"mimedx",
"smith nephew",
"smith & nephew",
"integra",
"solsys",
"amnioexcel",
"derma sciences",
"healthpoint",
"shire",
"acelity",
"kci",
"3m",
"molnlycke",
"medline",
"hollister",
"coloplast",
"stryker",
"zimmer biomet",
"wright medical",
"solventum",
"apria",
"anika",
"musculoskeletal transplant",
"surmodics",
"tissue regenix",
"nuo therapeutics",
"sanara medtech",
"kerecis",
"human bioprocessing",
"alphatec",
"biosig technologies",
]
# Brand names that indicate manufacturer-linked studies
BRAND_NAMES = [
"apligraf", "dermagraft", "epifix", "grafix", "amnioexcel",
"dermacell", "oasis", "primatrix", "integra", "graftjacket",
"dermapure", "affinity", "biovance", "cytal", "endoform",
"kerecis omega3", "novafix", "puraply", "restorigin",
"surgicraft", "theraskin", "amnioburn", "clarix",
"epicord", "genesis", "grafix core", "grafix prime",
"innovamatrix", "nushield", "stravix", "woundex",
"apligraf",
"dermagraft",
"epifix",
"grafix",
"amnioexcel",
"dermacell",
"oasis",
"primatrix",
"integra",
"graftjacket",
"dermapure",
"affinity",
"biovance",
"cytal",
"endoform",
"kerecis omega3",
"novafix",
"puraply",
"restorigin",
"surgicraft",
"theraskin",
"amnioburn",
"clarix",
"epicord",
"genesis",
"grafix core",
"grafix prime",
"innovamatrix",
"nushield",
"stravix",
"woundex",
]
# Patterns suggesting industry funding
@@ -206,9 +253,7 @@ def detect_coi(extra: str, abstract: str, title: str) -> list[str]:
def main() -> None:
parser = argparse.ArgumentParser(
description="Build skin-subs evidence base"
)
parser = argparse.ArgumentParser(description="Build skin-subs evidence base")
parser.add_argument("--dry-run", action="store_true")
args = parser.parse_args()
@@ -261,19 +306,13 @@ def main() -> None:
for row in rows:
tags = item_tags.get(row["id"], [])
col_name = assign_collection(
tags, row["title"] or "", row["abstract"] or ""
)
col_name = assign_collection(tags, row["title"] or "", row["abstract"] or "")
if col_name in col_map:
assignments.append((row["id"], col_map[col_name]))
collection_counts[col_name] = (
collection_counts.get(col_name, 0) + 1
)
collection_counts[col_name] = collection_counts.get(col_name, 0) + 1
print(" Assignment distribution:")
for name, count in sorted(
collection_counts.items(), key=lambda x: -x[1]
):
for name, count in sorted(collection_counts.items(), key=lambda x: -x[1]):
print(f" {name:30s}: {count:>5}")
# --- Step 4: COI / funding enrichment ---
@@ -326,8 +365,7 @@ def main() -> None:
for tag in new_tags:
tag_id = store._ensure_tag(tag)
con.execute(
"INSERT OR IGNORE INTO item_tags (item_id, tag_id) "
"VALUES (?, ?)",
"INSERT OR IGNORE INTO item_tags (item_id, tag_id) VALUES (?, ?)",
(item_id, tag_id),
)
total_tags_added += 1
@@ -340,13 +378,27 @@ def main() -> None:
# Tag schema coverage
expected_tags = [
"module:skin-subs",
"source:pubmed", "source:oig", "source:cms", "source:court",
"source:doj", "source:gao", "source:medpac",
"source:mac-lcd", "source:industry",
"type:clinical", "type:economic", "type:fraud",
"type:rct", "type:review", "type:meta-analysis",
"type:report", "type:rule", "type:filing",
"type:lcd", "type:position", "type:press-release",
"source:pubmed",
"source:oig",
"source:cms",
"source:court",
"source:doj",
"source:gao",
"source:medpac",
"source:mac-lcd",
"source:industry",
"type:clinical",
"type:economic",
"type:fraud",
"type:rct",
"type:review",
"type:meta-analysis",
"type:report",
"type:rule",
"type:filing",
"type:lcd",
"type:position",
"type:press-release",
]
for tag in expected_tags:
count = con.execute(

49
dev/scripts/build_skin_subs_geo_setting.py
View File

@@ -23,7 +23,21 @@ MAC_JURISDICTIONS = {
"CGS (J15)": ["KY", "OH"],
"WPS (J5/J8)": ["IA", "IN", "KS", "MI", "MO", "NE"],
"NGS (J6/JK)": ["CT", "IL", "MA", "ME", "MN", "NH", "NY", "RI", "VT", "WI"],
"Noridian (JE/JF)": ["AK", "AZ", "CA", "HI", "ID", "MT", "ND", "NV", "OR", "SD", "UT", "WA", "WY"],
"Noridian (JE/JF)": [
"AK",
"AZ",
"CA",
"HI",
"ID",
"MT",
"ND",
"NV",
"OR",
"SD",
"UT",
"WA",
"WY",
],
}
# Invert: state → MAC
@@ -72,15 +86,14 @@ def build_geographic_summary(con: duckdb.DuckDBPyConnection) -> None:
ORDER BY total_paid DESC
""")
count = con.execute(
"SELECT count(*) FROM skin_subs.geographic_summary"
).fetchone()[0]
count = con.execute("SELECT count(*) FROM skin_subs.geographic_summary").fetchone()[
0
]
print(f" States: {count}")
# Add MAC jurisdiction column
mac_cases = " ".join(
f"WHEN state = '{st}' THEN '{mac}'"
for st, mac in STATE_TO_MAC.items()
f"WHEN state = '{st}' THEN '{mac}'" for st, mac in STATE_TO_MAC.items()
)
con.execute(f"""
ALTER TABLE skin_subs.geographic_summary
@@ -110,9 +123,11 @@ def build_geographic_summary(con: duckdb.DuckDBPyConnection) -> None:
print("\n By MAC jurisdiction:")
for r in con.execute("SELECT * FROM skin_subs.mac_summary").fetchall():
print(f" {r[0]:25s} states={r[1]:2d} lines={r[2]:5d} "
f"paid=${r[5]:>10,.2f} per_bene=${r[6]:>8,.2f} "
f"office={r[7]}% podiatry={r[8]}%")
print(
f" {r[0]:25s} states={r[1]:2d} lines={r[2]:5d} "
f"paid=${r[5]:>10,.2f} per_bene=${r[6]:>8,.2f} "
f"office={r[7]}% podiatry={r[8]}%"
)
print("\n Top 5 states by spend per beneficiary:")
for r in con.execute("""
@@ -121,8 +136,10 @@ def build_geographic_summary(con: duckdb.DuckDBPyConnection) -> None:
FROM skin_subs.geographic_summary
ORDER BY paid_per_bene DESC LIMIT 5
""").fetchall():
print(f" {r[0]} {r[1]:25s} benes={r[2]:4d} "
f"per_bene=${r[4]:>8,.2f} office={r[5]}% podiatry={r[6]}%")
print(
f" {r[0]} {r[1]:25s} benes={r[2]:4d} "
f"per_bene=${r[4]:>8,.2f} office={r[5]}% podiatry={r[6]}%"
)
def build_setting_analysis(con: duckdb.DuckDBPyConnection) -> None:
@@ -175,9 +192,7 @@ def build_setting_analysis(con: duckdb.DuckDBPyConnection) -> None:
ORDER BY total_paid DESC
""")
count = con.execute(
"SELECT count(*) FROM skin_subs.setting_analysis"
).fetchone()[0]
count = con.execute("SELECT count(*) FROM skin_subs.setting_analysis").fetchone()[0]
print(f" Setting×specialty×product combinations: {count}")
# Setting summary
@@ -191,8 +206,10 @@ def build_setting_analysis(con: duckdb.DuckDBPyConnection) -> None:
FROM skin_subs.setting_analysis
GROUP BY setting ORDER BY paid DESC
""").fetchall():
print(f" {r[0]:15s} lines={r[1]:5d} provs={r[2]:4d} "
f"paid=${r[3]:>12,.2f} units={r[4]:4.1f} per_bene=${r[5]:>8,.2f}")
print(
f" {r[0]:15s} lines={r[1]:5d} provs={r[2]:4d} "
f"paid=${r[3]:>12,.2f} units={r[4]:4.1f} per_bene=${r[5]:>8,.2f}"
)
# Specialty×setting cross-tab
print("\n Specialty × setting (total paid):")

9
dev/scripts/check_pdf_coverage.py
View File

@@ -14,7 +14,6 @@ import sqlite3
from pathlib import Path
from conf import path as conf_path
from zot.db import FIELD_MAP
ZOTERO_DB = str(conf_path("db.zotero"))
@@ -54,7 +53,9 @@ def main() -> None:
print(f"Total skin-subs items: {total:>6}")
print(f"Items with DOI: {with_doi:>6} ({with_doi * 100 // total}%)")
print(f"Items with PDF attached: {with_pdf:>6} ({with_pdf * 100 // max(total, 1)}%)")
print(
f"Items with PDF attached: {with_pdf:>6} ({with_pdf * 100 // max(total, 1)}%)"
)
print(f"DOIs without PDF: {with_doi - with_pdf:>6}")
print(f"Coverage (of DOI items): {with_pdf * 100 // max(with_doi, 1)}%")
@@ -75,7 +76,9 @@ def main() -> None:
ORDER BY total DESC
""").fetchall():
pct = r["with_pdf"] * 100 // max(r["total"], 1)
print(f" {r['typeName']:20s} total={r['total']:>5} pdf={r['with_pdf']:>5} ({pct}%)")
print(
f" {r['typeName']:20s} total={r['total']:>5} pdf={r['with_pdf']:>5} ({pct}%)"
)
# Storage size
storage = Path(conf_path("storage.zotero"))

27
dev/scripts/collect_grey_lit_skin_subs.py
View File

@@ -20,7 +20,7 @@ import argparse
from dataclasses import dataclass, field
from datetime import datetime
from bib.item import Rule, Source
from bib.item import Source
from bib.store import Store
# ---------------------------------------------------------------------------
@@ -158,8 +158,7 @@ CMS_RULES = [
),
GreyLitEntry(
title=(
"CY 2024 PFS Final Rule (CMS-1784-F) — Skin Substitute "
"Payment Under Part B"
"CY 2024 PFS Final Rule (CMS-1784-F) — Skin Substitute Payment Under Part B"
),
url="https://www.federalregister.gov/documents/2023/11/16/2023-24184/medicare-and-medicaid-programs-cy-2024-payment-policies-under-the-physician-fee-schedule",
source_tag="cms",
@@ -296,8 +295,7 @@ DOJ_ENFORCEMENT = [
),
GreyLitEntry(
title=(
"USA v. Gehrke & King (D. Ariz.) — $1.2B Mobile Wound Care "
"Fraud Scheme"
"USA v. Gehrke & King (D. Ariz.) — $1.2B Mobile Wound Care Fraud Scheme"
),
url="https://www.justice.gov/usao-az/pr/two-individuals-charged-12-billion-health-care-fraud-scheme-involving-mobile-wound-care",
source_tag="court",
@@ -314,10 +312,7 @@ DOJ_ENFORCEMENT = [
extra_tags=["case:gehrke-king", "entity:daz"],
),
GreyLitEntry(
title=(
"USA v. Azar Nasser (E.D. Mich.) — $60M Skin Substitute "
"Fraud Ring"
),
title=("USA v. Azar Nasser (E.D. Mich.) — $60M Skin Substitute Fraud Ring"),
url="https://www.justice.gov/usao-edmi/pr/metro-detroit-physician-charged-60-million-health-care-fraud-scheme",
source_tag="court",
type_tag="filing",
@@ -403,8 +398,7 @@ GAO_MEDPAC = [
),
GreyLitEntry(
title=(
"MedPAC March 2025 Report to Congress — Payment for "
"Wound Care Products"
"MedPAC March 2025 Report to Congress — Payment for Wound Care Products"
),
url="https://www.medpac.gov/document/march-2025-report-to-the-congress/",
source_tag="medpac",
@@ -559,7 +553,9 @@ INDUSTRY = [
]
ALL_ENTRIES = OIG_REPORTS + CMS_RULES + DOJ_ENFORCEMENT + GAO_MEDPAC + MAC_LCDS + INDUSTRY
ALL_ENTRIES = (
OIG_REPORTS + CMS_RULES + DOJ_ENFORCEMENT + GAO_MEDPAC + MAC_LCDS + INDUSTRY
)
# ---------------------------------------------------------------------------
@@ -598,8 +594,11 @@ def entry_to_item(entry: GreyLitEntry) -> Source:
def main() -> None:
parser = argparse.ArgumentParser(description="Grey literature collection")
parser.add_argument("--dry-run", action="store_true",
help="Print catalogue only, don't write to bib.sqlite")
parser.add_argument(
"--dry-run",
action="store_true",
help="Print catalogue only, don't write to bib.sqlite",
)
args = parser.parse_args()
print("=" * 70)

15
dev/scripts/download_opps_files.py
View File

@@ -60,8 +60,9 @@ def download_file(url: str, dest: Path) -> bool:
return True
try:
print(f" GET {dest.name} ...", end="", flush=True)
resp = httpx.get(url, headers={"User-Agent": USER_AGENT},
timeout=60, follow_redirects=True)
resp = httpx.get(
url, headers={"User-Agent": USER_AGENT}, timeout=60, follow_redirects=True
)
if resp.status_code == 200:
dest.write_bytes(resp.content)
size_mb = len(resp.content) / 1024 / 1024
@@ -112,7 +113,9 @@ def main() -> None:
print(f"Downloaded: {downloaded}/{total} files")
print(f"Location: {OPPS_DIR}")
if set(OPPS_ADDENDA_URLS) != set(str(y) for y in range(2014, 2027)):
missing = sorted(set(str(y) for y in range(2014, 2027)) - set(OPPS_ADDENDA_URLS))
missing = sorted(
set(str(y) for y in range(2014, 2027)) - set(OPPS_ADDENDA_URLS)
)
print(f"\nMissing years (purged from CMS): {', '.join(missing)}")
print(" These may be recoverable from the Wayback Machine.")
@@ -122,8 +125,10 @@ def main() -> None:
files = list(year_dir.glob("*.zip"))
if files:
total_size = sum(f.stat().st_size for f in files)
print(f" {year_dir.name}: {len(files)} files "
f"({total_size / 1024 / 1024:.1f} MB)")
print(
f" {year_dir.name}: {len(files)} files "
f"({total_size / 1024 / 1024:.1f} MB)"
)
if __name__ == "__main__":

29
dev/scripts/dump_zot_schema.py Executable file
View File

@@ -0,0 +1,29 @@
"""Thin shim — logic moved to :mod:`zot.ops`.
Prefer: ``uv run stack zot dump-schema``.
"""
from __future__ import annotations
import sys
from pathlib import Path
def main() -> int:
from zot.ops import dump_schema
if len(sys.argv) != 2:
print(f"usage: {sys.argv[0]} <path/to/zotero.sqlite>", file=sys.stderr)
return 2
maps = dump_schema(Path(sys.argv[1]))
for name, m in maps.items():
print(f"{name}: dict[str, int] = {{")
for k, v in m.items():
print(f' "{k}": {v},')
print("}")
print()
return 0
if __name__ == "__main__":
sys.exit(main())

104
dev/scripts/enrich_entity_tags_and_load_duckdb.py
View File

@@ -134,9 +134,7 @@ def main() -> None:
# --- Step 2: Assign snowball articles to collections ---
print("\n--- Assigning snowball articles to collections ---")
# Find collection keys
col_rows = con.execute(
"SELECT key, name FROM collections"
).fetchall()
col_rows = con.execute("SELECT key, name FROM collections").fetchall()
col_name_to_key = {r["name"]: r["key"] for r in col_rows}
snowball_items = con.execute(
@@ -163,7 +161,9 @@ def main() -> None:
(col_id_row["id"], item_row["id"]),
)
con.commit()
print(f" Assigned {len(snowball_items)} snowball articles to Clinical Evidence")
print(
f" Assigned {len(snowball_items)} snowball articles to Clinical Evidence"
)
else:
print(" No unassigned snowball articles or collection not found")
@@ -183,67 +183,69 @@ def main() -> None:
).fetchall()
tag_list = [t["name"] for t in fresh_tags]
evidence_rows.append({
"bib_key": row["key"],
"title": (row["title"] or "")[:300],
"url": row["url"] or "",
"date_published": row["date_published"] or "",
"institution": row["institution"] or "",
"item_type": row["item_type"] or "",
"tags": "; ".join(sorted(tag_list)),
"source_tag": next(
(t for t in tag_list if t.startswith("source:")), ""
),
"type_tags": "; ".join(
t for t in tag_list if t.startswith("type:")
),
"entity_tags": "; ".join(
t for t in tag_list if t.startswith("entity:")
),
"is_snowball": "source:snowball" in tag_list,
"has_abstract": bool(row["abstract"]),
})
evidence_rows.append(
{
"bib_key": row["key"],
"title": (row["title"] or "")[:300],
"url": row["url"] or "",
"date_published": row["date_published"] or "",
"institution": row["institution"] or "",
"item_type": row["item_type"] or "",
"tags": "; ".join(sorted(tag_list)),
"source_tag": next(
(t for t in tag_list if t.startswith("source:")), ""
),
"type_tags": "; ".join(t for t in tag_list if t.startswith("type:")),
"entity_tags": "; ".join(
t for t in tag_list if t.startswith("entity:")
),
"is_snowball": "source:snowball" in tag_list,
"has_abstract": bool(row["abstract"]),
}
)
import pyarrow as pa
schema = pa.schema([
("bib_key", pa.string()),
("title", pa.string()),
("url", pa.string()),
("date_published", pa.string()),
("institution", pa.string()),
("item_type", pa.string()),
("tags", pa.string()),
("source_tag", pa.string()),
("type_tags", pa.string()),
("entity_tags", pa.string()),
("is_snowball", pa.bool_()),
("has_abstract", pa.bool_()),
])
schema = pa.schema(
[
("bib_key", pa.string()),
("title", pa.string()),
("url", pa.string()),
("date_published", pa.string()),
("institution", pa.string()),
("item_type", pa.string()),
("tags", pa.string()),
("source_tag", pa.string()),
("type_tags", pa.string()),
("entity_tags", pa.string()),
("is_snowball", pa.bool_()),
("has_abstract", pa.bool_()),
]
)
arrays = [pa.array([r[f.name] for r in evidence_rows]) for f in schema]
arrow_tbl = pa.table(
dict(zip([f.name for f in schema], arrays)), schema=schema
)
arrow_tbl = pa.table(dict(zip([f.name for f in schema], arrays)), schema=schema)
ddb = duckdb.connect(str(DUCKDB_PATH))
ddb.execute("CREATE SCHEMA IF NOT EXISTS skin_subs")
ddb.execute("DROP TABLE IF EXISTS skin_subs.evidence_base")
ddb.register("arrow_tbl", arrow_tbl)
ddb.execute(
"CREATE TABLE skin_subs.evidence_base AS SELECT * FROM arrow_tbl"
)
count = ddb.execute(
"SELECT count(*) FROM skin_subs.evidence_base"
).fetchone()[0]
ddb.execute("CREATE TABLE skin_subs.evidence_base AS SELECT * FROM arrow_tbl")
count = ddb.execute("SELECT count(*) FROM skin_subs.evidence_base").fetchone()[0]
print(f" Loaded {count} rows into skin_subs.evidence_base")
# Validation
print("\n Validation:")
for q, label in [
("SELECT source_tag, count(*) c FROM skin_subs.evidence_base GROUP BY 1 ORDER BY c DESC LIMIT 5", "by_source"),
(
"SELECT source_tag, count(*) c FROM skin_subs.evidence_base GROUP BY 1 ORDER BY c DESC LIMIT 5",
"by_source",
),
("SELECT count(*) FROM skin_subs.evidence_base WHERE is_snowball", "snowball"),
("SELECT count(*) FROM skin_subs.evidence_base WHERE entity_tags != ''", "with_entities"),
(
"SELECT count(*) FROM skin_subs.evidence_base WHERE entity_tags != ''",
"with_entities",
),
]:
print(f" {label}: {ddb.execute(q).fetchall()}")
@@ -254,9 +256,7 @@ def main() -> None:
"WHERE table_schema = 'skin_subs' ORDER BY table_name"
).fetchall()
for t in tbls:
cnt = ddb.execute(
f"SELECT count(*) FROM skin_subs.{t[0]}"
).fetchone()[0]
cnt = ddb.execute(f"SELECT count(*) FROM skin_subs.{t[0]}").fetchone()[0]
print(f" skin_subs.{t[0]:30s}: {cnt:>6} rows")
ddb.close()

94
dev/scripts/extract_study_characteristics.py
View File

@@ -232,23 +232,25 @@ def main() -> None:
if "type:fraud" in tag_set:
domains.append("fraud")
chars.append({
"pmid": pmid,
"bib_key": row["key"],
"first_author": first_author,
"year": year,
"pub_type": pub_type,
"journal": journal,
"title": title[:200],
"products": "; ".join(products) if products else "",
"product_count": len(products),
"sample_size": sample_size,
"is_industry_linked": is_industry_linked,
"is_single_product": is_single_product,
"stance": stance,
"domains": "; ".join(domains),
"url": row["url"] or "",
})
chars.append(
{
"pmid": pmid,
"bib_key": row["key"],
"first_author": first_author,
"year": year,
"pub_type": pub_type,
"journal": journal,
"title": title[:200],
"products": "; ".join(products) if products else "",
"product_count": len(products),
"sample_size": sample_size,
"is_industry_linked": is_industry_linked,
"is_single_product": is_single_product,
"stance": stance,
"domains": "; ".join(domains),
"url": row["url"] or "",
}
)
store.close()
@@ -269,7 +271,7 @@ def main() -> None:
# Top products
product_counts: dict[str, int] = {}
for c in chars:
for p in (c["products"].split("; ") if c["products"] else []):
for p in c["products"].split("; ") if c["products"] else []:
product_counts[p] = product_counts.get(p, 0) + 1
print("\n Top 15 products mentioned:")
for p, ct in sorted(product_counts.items(), key=lambda x: -x[1])[:15]:
@@ -285,23 +287,25 @@ def main() -> None:
# Register Python list as table
import pyarrow as pa
schema = pa.schema([
("pmid", pa.string()),
("bib_key", pa.string()),
("first_author", pa.string()),
("year", pa.string()),
("pub_type", pa.string()),
("journal", pa.string()),
("title", pa.string()),
("products", pa.string()),
("product_count", pa.int32()),
("sample_size", pa.int32()),
("is_industry_linked", pa.bool_()),
("is_single_product", pa.bool_()),
("stance", pa.string()),
("domains", pa.string()),
("url", pa.string()),
])
schema = pa.schema(
[
("pmid", pa.string()),
("bib_key", pa.string()),
("first_author", pa.string()),
("year", pa.string()),
("pub_type", pa.string()),
("journal", pa.string()),
("title", pa.string()),
("products", pa.string()),
("product_count", pa.int32()),
("sample_size", pa.int32()),
("is_industry_linked", pa.bool_()),
("is_single_product", pa.bool_()),
("stance", pa.string()),
("domains", pa.string()),
("url", pa.string()),
]
)
arrays = [
pa.array([c["pmid"] for c in chars]),
@@ -334,10 +338,22 @@ def main() -> None:
# Quick validation queries
print("\n Validation:")
for q, label in [
("SELECT pub_type, count(*) c FROM skin_subs.study_characteristics GROUP BY 1 ORDER BY c DESC LIMIT 5", "pub_type"),
("SELECT count(*) FROM skin_subs.study_characteristics WHERE product_count > 0", "with_products"),
("SELECT count(*) FROM skin_subs.study_characteristics WHERE sample_size IS NOT NULL", "with_sample_size"),
("SELECT count(*) FROM skin_subs.study_characteristics WHERE is_industry_linked", "industry_linked"),
(
"SELECT pub_type, count(*) c FROM skin_subs.study_characteristics GROUP BY 1 ORDER BY c DESC LIMIT 5",
"pub_type",
),
(
"SELECT count(*) FROM skin_subs.study_characteristics WHERE product_count > 0",
"with_products",
),
(
"SELECT count(*) FROM skin_subs.study_characteristics WHERE sample_size IS NOT NULL",
"with_sample_size",
),
(
"SELECT count(*) FROM skin_subs.study_characteristics WHERE is_industry_linked",
"industry_linked",
),
]:
result = ddb.execute(q).fetchall()
print(f" {label}: {result}")

76
dev/scripts/fetch_pdfs.py
View File

@@ -19,7 +19,6 @@ Usage:
from __future__ import annotations
import argparse
import hashlib
import os
import re
import sqlite3
@@ -47,8 +46,12 @@ def fetch_unpaywall(doi: str) -> str | None:
"""Get PDF URL from Unpaywall for a DOI. Returns URL or None."""
url = f"https://api.unpaywall.org/v2/{doi}"
try:
resp = httpx.get(url, params={"email": UNPAYWALL_EMAIL},
headers={"User-Agent": USER_AGENT}, timeout=15)
resp = httpx.get(
url,
params={"email": UNPAYWALL_EMAIL},
headers={"User-Agent": USER_AGENT},
timeout=15,
)
if resp.status_code != 200:
return None
data = resp.json()
@@ -68,7 +71,7 @@ def doi_to_pmcid(dois: list[str]) -> dict[str, str]:
result = {}
# API accepts up to 200 IDs per request
for i in range(0, len(dois), 200):
batch = dois[i:i + 200]
batch = dois[i : i + 200]
try:
resp = httpx.get(
"https://www.ncbi.nlm.nih.gov/pmc/utils/idconv/v1.0/",
@@ -190,7 +193,9 @@ def fetch_scihub(doi: str) -> str | None:
for mirror in SCIHUB_MIRRORS:
try:
with httpx.Client(timeout=15, follow_redirects=True, **client_kwargs) as client:
with httpx.Client(
timeout=15, follow_redirects=True, **client_kwargs
) as client:
resp = client.get(
f"{mirror}/{doi}",
headers={"User-Agent": "Mozilla/5.0"},
@@ -273,7 +278,12 @@ def attach_pdf_to_item(
"""INSERT INTO itemAttachments
(itemID, parentItemID, linkMode, contentType, path, storageModTime)
VALUES (?, ?, 0, 'application/pdf', ?, ?)""",
(att_item_id, item_id, f"storage:{key}/{pdf_path.name}", int(time.time() * 1000)),
(
att_item_id,
item_id,
f"storage:{key}/{pdf_path.name}",
int(time.time() * 1000),
),
)
# Move PDF to Zotero storage
@@ -282,6 +292,7 @@ def attach_pdf_to_item(
dest = storage_dir / pdf_path.name
if pdf_path != dest:
import shutil
shutil.copy2(pdf_path, dest)
con.commit()
@@ -295,7 +306,8 @@ def attach_pdf_to_item(
def get_items_needing_pdfs(con: sqlite3.Connection) -> list[dict]:
"""Get skin-subs items with DOI but no PDF attachment."""
rows = con.execute("""
rows = con.execute(
"""
SELECT DISTINCT i.itemID,
(SELECT idv.value FROM itemData id
JOIN itemDataValues idv ON id.valueID = idv.valueID
@@ -310,17 +322,26 @@ def get_items_needing_pdfs(con: sqlite3.Connection) -> list[dict]:
WHERE parentItemID IS NOT NULL
AND contentType = 'application/pdf'
)
""", (FIELD_MAP["DOI"], TYPE_MAP["journalArticle"])).fetchall()
return [{"item_id": r[0], "doi": r[1]}
for r in rows if r[1]]
""",
(FIELD_MAP["DOI"], TYPE_MAP["journalArticle"]),
).fetchall()
return [{"item_id": r[0], "doi": r[1]} for r in rows if r[1]]
def main() -> None:
parser = argparse.ArgumentParser(description="Headless PDF retrieval")
parser.add_argument("--limit", type=int, default=0, help="Max items to process (0=all)")
parser.add_argument("--source", choices=["all", "unpaywall", "pmc", "s2", "worker", "scihub"],
default="all", help="Which source to use")
parser.add_argument("--proxy", help="SOCKS/HTTP proxy for SciHub (e.g. socks5://localhost:1080)")
parser.add_argument(
"--limit", type=int, default=0, help="Max items to process (0=all)"
)
parser.add_argument(
"--source",
choices=["all", "unpaywall", "pmc", "s2", "worker", "scihub"],
default="all",
help="Which source to use",
)
parser.add_argument(
"--proxy", help="SOCKS/HTTP proxy for SciHub (e.g. socks5://localhost:1080)"
)
args = parser.parse_args()
global SCIHUB_PROXY
@@ -338,7 +359,7 @@ def main() -> None:
items = get_items_needing_pdfs(con)
if args.limit:
items = items[:args.limit]
items = items[: args.limit]
print(f"\nItems needing PDFs: {len(items)}")
tmp_dir = Path("/tmp/pdf_downloads")
@@ -427,8 +448,10 @@ def main() -> None:
else:
consecutive_failures += 1
if (i + 1) % 25 == 0:
print(f" {i + 1}/{len(remaining)} worker={stats.get('worker', 0)} "
f"fails={consecutive_failures}")
print(
f" {i + 1}/{len(remaining)} worker={stats.get('worker', 0)} "
f"fails={consecutive_failures}"
)
time.sleep(1.5) # Be polite to Worker + SciHub
print(f" CF Worker: {stats.get('worker', 0)} PDFs")
@@ -442,7 +465,9 @@ def main() -> None:
consecutive_failures = 0
for i, item in enumerate(remaining):
if consecutive_failures >= 10:
print(f" Stopping: {consecutive_failures} consecutive failures (CAPTCHA?)")
print(
f" Stopping: {consecutive_failures} consecutive failures (CAPTCHA?)"
)
break
pdf_url = fetch_scihub(item["doi"])
@@ -459,15 +484,22 @@ def main() -> None:
consecutive_failures += 1
if (i + 1) % 25 == 0:
print(f" {i + 1}/{len(remaining)} scihub={stats['scihub']} "
f"fails={consecutive_failures}")
print(
f" {i + 1}/{len(remaining)} scihub={stats['scihub']} "
f"fails={consecutive_failures}"
)
time.sleep(1) # Be polite to SciHub
print(f" SciHub: {stats['scihub']} PDFs")
stats["failed"] = len([it for it in items if not it.get("_done")])
total_found = (stats["unpaywall"] + stats["pmc"] + stats.get("s2", 0)
+ stats.get("worker", 0) + stats.get("scihub", 0))
total_found = (
stats["unpaywall"]
+ stats["pmc"]
+ stats.get("s2", 0)
+ stats.get("worker", 0)
+ stats.get("scihub", 0)
)
print(f"\n{'=' * 60}")
print(f"Results: {total_found} PDFs downloaded")

75
dev/scripts/fix_zotero_dates.py
View File

@@ -1,79 +1,22 @@
"""Fix and prevent non-ISO 8601 dates in Zotero's SQLite database.
"""Thin shim — logic moved to :mod:`zot.ops`.
Zotero internally writes timestamps as 'YYYY-MM-DD HH:MM:SS' (SQL format)
but its sync engine requires 'YYYY-MM-DDTHH:MM:SSZ' (ISO 8601).
This script:
1. Converts all existing space-separated timestamps to ISO 8601
2. Installs SQLite triggers that auto-convert on every INSERT/UPDATE
Safe to run multiple times (triggers use CREATE IF NOT EXISTS).
Usage:
uv run python dev/scripts/fix_zotero_dates.py
Prefer: ``uv run stack zot fix-dates``.
"""
from __future__ import annotations
import sqlite3
import sys
DB_PATH = "zotero/data/zotero.sqlite"
COLUMNS = ("dateAdded", "dateModified", "clientDateModified")
from pathlib import Path
def main() -> int:
db = sqlite3.connect(DB_PATH, timeout=10)
db.execute("PRAGMA wal_checkpoint(TRUNCATE)")
from zot.ops import fix_dates
# Fix existing bad dates
for col in COLUMNS:
fixed = db.execute(
f"UPDATE items SET {col} = REPLACE({col}, ' ', 'T') || 'Z'"
f" WHERE {col} LIKE '____-__-__ __:__:__'"
f" AND {col} NOT LIKE '%T%'",
).rowcount
if fixed:
print(f"Fixed {col}: {fixed}")
# Install auto-fix triggers
for col in COLUMNS:
for op in ("INSERT", "UPDATE"):
trigger = f"fix_{col}_{op.lower()}"
db.execute(f"""
CREATE TRIGGER IF NOT EXISTS {trigger}
AFTER {op} ON items
FOR EACH ROW
WHEN NEW.{col} LIKE '____-__-__ __:__:__'
AND NEW.{col} NOT LIKE '%T%'
BEGIN
UPDATE items SET {col} = REPLACE(NEW.{col}, ' ', 'T') || 'Z'
WHERE itemID = NEW.itemID;
END
""")
db.commit()
# Verify
triggers = [
r[0]
for r in db.execute(
"SELECT name FROM sqlite_master WHERE type='trigger' AND name LIKE 'fix_%'"
)
]
print(f"Triggers installed: {len(triggers)}")
for col in COLUMNS:
bad = db.execute(
f"SELECT COUNT(*) FROM items WHERE {col} NOT LIKE '%T%'"
).fetchone()[0]
if bad:
print(f"WARNING: {col} still has {bad} bad values")
db.close()
return 1
db.close()
print("All dates valid.")
return 0
db = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("zotero/data/zotero.sqlite")
out = fix_dates(db)
for k, v in out.items():
print(f"{k}: {v}")
return 0 if out.get("remaining", 0) == 0 else 1
if __name__ == "__main__":

132
dev/scripts/fix_zotero_keys.py
View File

@@ -1,136 +1,22 @@
"""Fix invalid Zotero object keys.
"""Thin shim — logic moved to :mod:`zot.ops`.
Zotero 8 requires keys to be exactly 8 characters from
[23456789ABCDEFGHIJKLMNPQRSTUVWXYZ]. Bulk-imported items may have
keys containing 0, 1, or O which cause "key is not valid" errors
when Zotero tries to create annotations or other child items.
This script:
1. Finds all items and collections with invalid keys
2. Generates new valid keys (no collisions)
3. Updates the items/collections tables
4. Renames storage folders to match new keys
Usage:
# Stop Zotero first, then:
uv run python dev/scripts/fix_zotero_keys.py
Prefer: ``uv run stack zot fix-keys``.
"""
from __future__ import annotations
import os
import random
import re
import shutil
import sqlite3
import sys
ALLOWED = "23456789ABCDEFGHIJKLMNPQRSTUVWXYZ"
KEY_RE = re.compile(rf"^[{ALLOWED}]{{8}}$")
DB_PATH = "zotero/data/zotero.sqlite"
STORAGE_DIR = "zotero/data/storage"
def generate_key(existing: set[str]) -> str:
"""Generate a valid 8-char key not in *existing*."""
while True:
key = "".join(random.choices(ALLOWED, k=8))
if key not in existing:
existing.add(key)
return key
from pathlib import Path
def main() -> int:
if not os.path.exists(DB_PATH):
print(f"ERROR: {DB_PATH} not found", file=sys.stderr)
return 1
from zot.ops import fix_keys
# Backup
backup = DB_PATH + ".pre-keyfix.bak"
if not os.path.exists(backup):
shutil.copy2(DB_PATH, backup)
print(f"Backed up to {backup}")
db = sqlite3.connect(DB_PATH, timeout=10)
# Checkpoint WAL first
db.execute("PRAGMA wal_checkpoint(TRUNCATE)")
# Collect all existing keys
existing_keys: set[str] = set()
for (k,) in db.execute("SELECT key FROM items"):
existing_keys.add(k)
for (k,) in db.execute("SELECT key FROM collections"):
existing_keys.add(k)
# Find invalid item keys
invalid_items: list[tuple[int, str]] = []
for row in db.execute("SELECT itemID, key FROM items"):
if not KEY_RE.match(row[1]):
invalid_items.append(row)
# Find invalid collection keys
invalid_colls: list[tuple[int, str]] = []
for row in db.execute("SELECT collectionID, key FROM collections"):
if not KEY_RE.match(row[1]):
invalid_colls.append(row)
print(f"Invalid item keys: {len(invalid_items)}")
print(f"Invalid collection keys: {len(invalid_colls)}")
if not invalid_items and not invalid_colls:
print("Nothing to fix!")
db.close()
return 0
# Fix items
renames: list[tuple[str, str]] = [] # (old_key, new_key) for storage
for item_id, old_key in invalid_items:
new_key = generate_key(existing_keys)
db.execute("UPDATE items SET key = ? WHERE itemID = ?", (new_key, item_id))
old_dir = os.path.join(STORAGE_DIR, old_key)
if os.path.isdir(old_dir):
renames.append((old_key, new_key))
# Fix collections
for coll_id, old_key in invalid_colls:
new_key = generate_key(existing_keys)
db.execute(
"UPDATE collections SET key = ? WHERE collectionID = ?",
(new_key, coll_id),
)
db.commit()
db.close()
print(f"Updated {len(invalid_items)} items + {len(invalid_colls)} collections")
# Rename storage folders
renamed = 0
for old_key, new_key in renames:
old_dir = os.path.join(STORAGE_DIR, old_key)
new_dir = os.path.join(STORAGE_DIR, new_key)
if os.path.isdir(old_dir) and not os.path.exists(new_dir):
os.rename(old_dir, new_dir)
renamed += 1
print(f"Renamed {renamed} storage folders")
# Verify
db = sqlite3.connect(DB_PATH, timeout=10)
remaining = 0
for (k,) in db.execute("SELECT key FROM items"):
if not KEY_RE.match(k):
remaining += 1
for (k,) in db.execute("SELECT key FROM collections"):
if not KEY_RE.match(k):
remaining += 1
db.close()
if remaining:
print(f"WARNING: {remaining} invalid keys remaining!")
return 1
print("All keys valid. Restart Zotero to apply.")
return 0
db = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("zotero/data/zotero.sqlite")
out = fix_keys(db)
for k, v in out.items():
print(f"{k}: {v}")
return 0 if out.get("remaining", 0) == 0 else 1
if __name__ == "__main__":

14
dev/scripts/ingest_asp.py
View File

@@ -291,7 +291,9 @@ def main() -> None:
& ~all_df["hcpcs_code"].isin(skin_codes)
]["hcpcs_code"].unique()
if len(q4_but_not_skin) > 0:
print(f" Excluded Q4 codes NOT in skin sub universe: {sorted(q4_but_not_skin)}")
print(
f" Excluded Q4 codes NOT in skin sub universe: {sorted(q4_but_not_skin)}"
)
if not skin_df.empty:
skin_df["payment_limit"] = clean_numeric(skin_df["payment_limit"])
@@ -338,13 +340,15 @@ def main() -> None:
""",
[str(OUTPUT_CSV)],
)
count = con.execute(
"SELECT count(*) FROM skin_subs.asp_quarterly"
).fetchone()[0]
count = con.execute("SELECT count(*) FROM skin_subs.asp_quarterly").fetchone()[
0
]
print(f" Loaded {count} rows into skin_subs.asp_quarterly")
else:
print(" WARNING: No skin substitute data found in ASP files")
print(" This may be expected — skin subs may not appear in main ASP pricing files")
print(
" This may be expected — skin subs may not appear in main ASP pricing files"
)
print(" They may be in separate NOC or tissue coding files")
con.close()

60
dev/scripts/ingest_opps.py
View File

@@ -19,7 +19,6 @@ from __future__ import annotations
import argparse
import io
import re
import zipfile
from pathlib import Path
@@ -133,14 +132,17 @@ def _find_header_row(df: pd.DataFrame, col_map: dict[str, str]) -> int:
vals = [str(v).strip().lower() for v in row if pd.notna(v) and str(v).strip()]
if len(vals) < 3:
continue
matches = sum(1 for v in vals for k in target_cols if v == k or (len(k) > 3 and k in v))
matches = sum(
1 for v in vals for k in target_cols if v == k or (len(k) > 3 and k in v)
)
if matches >= 3:
return i
return 0
def read_from_zip(zpath: Path, col_map: dict[str, str],
pattern: str = "") -> pd.DataFrame:
def read_from_zip(
zpath: Path, col_map: dict[str, str], pattern: str = ""
) -> pd.DataFrame:
"""Read and normalise a data file from inside a ZIP.
Handles CMS preamble rows in both CSV and Excel formats.
@@ -168,7 +170,9 @@ def read_from_zip(zpath: Path, col_map: dict[str, str],
df = pd.read_csv(io.StringIO(buf), dtype=str, on_bad_lines="skip")
else:
# Excel: read without header to find preamble extent
df_raw = pd.read_excel(io.BytesIO(data), dtype=str, header=None, nrows=20)
df_raw = pd.read_excel(
io.BytesIO(data), dtype=str, header=None, nrows=20
)
hdr_row = _find_header_row(df_raw, col_map)
df = pd.read_excel(io.BytesIO(data), dtype=str, header=hdr_row)
@@ -224,7 +228,11 @@ def ingest_all(con: duckdb.DuckDBPyConnection, year_filter: str = "") -> None:
df = read_from_zip(addenda_zip, ADDENDUM_A_COL_MAP, pattern="addendum a")
if not df.empty:
df["year"] = int(year)
for col in ["relative_weight", "payment_rate", "minimum_unadjusted_copayment"]:
for col in [
"relative_weight",
"payment_rate",
"minimum_unadjusted_copayment",
]:
if col in df.columns:
df[col] = clean_numeric(df[col])
apc_frames.append(df)
@@ -234,7 +242,11 @@ def ingest_all(con: duckdb.DuckDBPyConnection, year_filter: str = "") -> None:
df = read_from_zip(addenda_zip, ADDENDUM_B_COL_MAP, pattern="addendum b")
if not df.empty:
df["year"] = int(year)
for col in ["relative_weight", "payment_rate", "minimum_unadjusted_copayment"]:
for col in [
"relative_weight",
"payment_rate",
"minimum_unadjusted_copayment",
]:
if col in df.columns:
df[col] = clean_numeric(df[col])
addb_frames.append(df)
@@ -247,22 +259,28 @@ def ingest_all(con: duckdb.DuckDBPyConnection, year_filter: str = "") -> None:
all_apc = pd.concat(apc_frames, ignore_index=True)
con.execute("DROP TABLE IF EXISTS opps.apc_weight")
con.execute("CREATE TABLE opps.apc_weight AS SELECT * FROM all_apc")
print(f" opps.apc_weight: {len(all_apc)} rows "
f"({all_apc['year'].nunique()} years)")
print(
f" opps.apc_weight: {len(all_apc)} rows "
f"({all_apc['year'].nunique()} years)"
)
if addb_frames:
all_addb = pd.concat(addb_frames, ignore_index=True)
con.execute("DROP TABLE IF EXISTS opps.addendum_b")
con.execute("CREATE TABLE opps.addendum_b AS SELECT * FROM all_addb")
print(f" opps.addendum_b: {len(all_addb)} rows "
f"({all_addb['year'].nunique()} years)")
print(
f" opps.addendum_b: {len(all_addb)} rows "
f"({all_addb['year'].nunique()} years)"
)
if wage_frames:
all_wage = pd.concat(wage_frames, ignore_index=True)
con.execute("DROP TABLE IF EXISTS opps.wage_index")
con.execute("CREATE TABLE opps.wage_index AS SELECT * FROM all_wage")
print(f" opps.wage_index: {len(all_wage)} rows "
f"({all_wage['year'].nunique()} years)")
print(
f" opps.wage_index: {len(all_wage)} rows "
f"({all_wage['year'].nunique()} years)"
)
# Skin sub specific: extract skin sub codes from Addendum B history
if addb_frames:
@@ -270,14 +288,20 @@ def ingest_all(con: duckdb.DuckDBPyConnection, year_filter: str = "") -> None:
all_addb = pd.concat(addb_frames, ignore_index=True)
if "hcpcs" in all_addb.columns:
skin_codes = all_addb[
all_addb["hcpcs"].astype(str).str.match(r"^Q4\d{2,3}$|^C527[1-8]$", na=False)
all_addb["hcpcs"]
.astype(str)
.str.match(r"^Q4\d{2,3}$|^C527[1-8]$", na=False)
]
if not skin_codes.empty:
con.execute("DROP TABLE IF EXISTS opps.skin_sub_addendum_b")
con.execute("CREATE TABLE opps.skin_sub_addendum_b AS SELECT * FROM skin_codes")
print(f" opps.skin_sub_addendum_b: {len(skin_codes)} rows "
f"({skin_codes['year'].nunique()} years, "
f"{skin_codes['hcpcs'].nunique()} unique codes)")
con.execute(
"CREATE TABLE opps.skin_sub_addendum_b AS SELECT * FROM skin_codes"
)
print(
f" opps.skin_sub_addendum_b: {len(skin_codes)} rows "
f"({skin_codes['year'].nunique()} years, "
f"{skin_codes['hcpcs'].nunique()} unique codes)"
)
# SI distribution for skin subs
if "status_indicator" in skin_codes.columns:

34
dev/scripts/register_seeds.py
View File

@@ -14,10 +14,7 @@ Usage::
from __future__ import annotations
import argparse
import shutil
import sqlite3
import subprocess
from dataclasses import dataclass, field
from pathlib import Path
@@ -294,10 +291,6 @@ SEED_REGISTRY: tuple[SeedSpec, ...] = (
# ── Helpers ───────────────────────────────────────────────────────
def _content_type(path: Path) -> str:
ext = path.suffix.lower()
types = {
@@ -317,9 +310,7 @@ def _content_type(path: Path) -> str:
# ── Phase A: tag existing Zotero attachments ──────────────────────
def tag_existing_zotero(
db: Db, spec: SeedSpec, *, dry_run: bool = False
) -> bool:
def tag_existing_zotero(db: Db, spec: SeedSpec, *, dry_run: bool = False) -> bool:
"""Find a Zotero attachment by filename and tag its parent."""
filename = Path(spec.path).name
row = db.con.execute(
@@ -344,9 +335,7 @@ def tag_existing_zotero(
# ── Phase B: create missing Zotero items ──────────────────────────
def register_in_zotero(
db: Db, spec: SeedSpec, *, dry_run: bool = False
) -> None:
def register_in_zotero(db: Db, spec: SeedSpec, *, dry_run: bool = False) -> None:
"""Create a Zotero webpage item + file attachment for a seed."""
if not spec.url:
if dry_run:
@@ -361,14 +350,17 @@ def register_in_zotero(
# Create parent webpage item
parent_id = db.create_item(TYPE_MAP["webpage"], now=now)
db.set_fields(parent_id, {
"title": spec.title,
"url": spec.url,
"date": now[:10],
"accessDate": now,
"websiteType": "Government Data Portal",
"websiteTitle": "Centers for Medicare & Medicaid Services",
})
db.set_fields(
parent_id,
{
"title": spec.title,
"url": spec.url,
"date": now[:10],
"accessDate": now,
"websiteType": "Government Data Portal",
"websiteTitle": "Centers for Medicare & Medicaid Services",
},
)
db.sync_tags(parent_id, [SEED_TAG] + spec.tags)
# Attach file (skip directories)

25
dev/scripts/search_pubmed_skin_subs.py
View File

@@ -185,15 +185,21 @@ def efetch_articles(
except (httpx.RemoteProtocolError, httpx.ReadTimeout) as exc:
if attempt < max_retries - 1:
wait = 2 ** (attempt + 1)
print(f" RETRY batch {i // batch_size + 1} "
f"(attempt {attempt + 2}/{max_retries}, "
f"wait {wait}s): {exc}")
print(
f" RETRY batch {i // batch_size + 1} "
f"(attempt {attempt + 2}/{max_retries}, "
f"wait {wait}s): {exc}"
)
time.sleep(wait)
else:
print(f" SKIP batch {i // batch_size + 1} after "
f"{max_retries} attempts: {exc}")
print(f" efetch: batch {i // batch_size + 1}/{len(pmids) // batch_size + 1}, "
f"got {len(articles)} articles so far")
print(
f" SKIP batch {i // batch_size + 1} after "
f"{max_retries} attempts: {exc}"
)
print(
f" efetch: batch {i // batch_size + 1}/{len(pmids) // batch_size + 1}, "
f"got {len(articles)} articles so far"
)
return articles
@@ -379,8 +385,9 @@ def article_to_source(article: Article, domain_tags: list[str]) -> Source:
def main() -> None:
parser = argparse.ArgumentParser(description="PubMed skin substitutes search")
parser.add_argument("--dry-run", action="store_true",
help="Search only, don't write to bib.sqlite")
parser.add_argument(
"--dry-run", action="store_true", help="Search only, don't write to bib.sqlite"
)
args = parser.parse_args()
print("=" * 70)

60
dev/scripts/snowball_citations.py
View File

@@ -63,9 +63,7 @@ def elink_cited_by(pmids: list[str], batch_size: int = 50) -> dict[str, list[str
)
time.sleep(RATE_LIMIT)
try:
resp = httpx.get(
f"{EUTILS_BASE}/elink.fcgi", params=params, timeout=60
)
resp = httpx.get(f"{EUTILS_BASE}/elink.fcgi", params=params, timeout=60)
resp.raise_for_status()
root = ET.fromstring(resp.text) # noqa: S314
for linkset in root.findall(".//LinkSet"):
@@ -111,13 +109,18 @@ def efetch_basic(pmids: list[str], batch_size: int = 100) -> list[dict]:
for i in range(0, len(pmids), batch_size):
batch = pmids[i : i + batch_size]
params = _params(
db="pubmed", id=",".join(batch), rettype="xml", retmode="xml",
db="pubmed",
id=",".join(batch),
rettype="xml",
retmode="xml",
)
for attempt in range(3):
time.sleep(RATE_LIMIT * (attempt + 1))
try:
resp = httpx.get(
f"{EUTILS_BASE}/efetch.fcgi", params=params, timeout=120,
f"{EUTILS_BASE}/efetch.fcgi",
params=params,
timeout=120,
)
resp.raise_for_status()
root = ET.fromstring(resp.text) # noqa: S314
@@ -155,7 +158,9 @@ def efetch_basic(pmids: list[str], batch_size: int = 100) -> list[dict]:
# Journal + year
journal_el = article_el.find("Journal")
journal = _text(journal_el, "Title") if journal_el is not None else ""
journal = (
_text(journal_el, "Title") if journal_el is not None else ""
)
year = ""
pub_date = article_el.find(".//PubDate")
if pub_date is not None:
@@ -178,16 +183,18 @@ def efetch_basic(pmids: list[str], batch_size: int = 100) -> list[dict]:
if pt.text:
pub_types.append(pt.text)
articles.append({
"pmid": pmid,
"title": title,
"abstract": "\n\n".join(abstract_parts),
"authors": authors,
"journal": journal,
"year": year,
"doi": doi,
"pub_types": pub_types,
})
articles.append(
{
"pmid": pmid,
"title": title,
"abstract": "\n\n".join(abstract_parts),
"authors": authors,
"journal": journal,
"year": year,
"doi": doi,
"pub_types": pub_types,
}
)
break
except (httpx.RemoteProtocolError, httpx.ReadTimeout) as exc:
if attempt < 2:
@@ -209,8 +216,12 @@ def efetch_basic(pmids: list[str], batch_size: int = 100) -> list[dict]:
def main() -> None:
parser = argparse.ArgumentParser(description="Snowball citation chasing")
parser.add_argument("--dry-run", action="store_true")
parser.add_argument("--seed-limit", type=int, default=200,
help="Max seed articles for forward snowball")
parser.add_argument(
"--seed-limit",
type=int,
default=200,
help="Max seed articles for forward snowball",
)
args = parser.parse_args()
print("=" * 70)
@@ -298,9 +309,16 @@ def main() -> None:
# --- Relevance filter: must mention skin/wound in title or abstract ---
skin_keywords = [
"skin substitute", "skin substitutes", "wound", "ulcer",
"biological dressing", "tissue product", "graft",
"dermal", "epidermal", "bioengineered",
"skin substitute",
"skin substitutes",
"wound",
"ulcer",
"biological dressing",
"tissue product",
"graft",
"dermal",
"epidermal",
"bioengineered",
]
relevant = []
for art in new_articles:

2
dev/scripts/sync_secrets.py
View File

@@ -41,7 +41,7 @@ def main() -> int:
mapping = {item["env"]: item["key"] for item in mapping_list}
if not mapping:
print(f"No secret mappings defined in [databricks.secrets]")
print("No secret mappings defined in [databricks.secrets]")
return 0
from aco.lake.unity import UnityClient

133
dev/scripts/zotero_archive_migrate.py
View File

@@ -12,9 +12,17 @@ from pathlib import Path
from zot.db import generate_key
HOST_DB = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/tmp/host-zotero.sqlite")
CONTAINER_DB = Path(sys.argv[2]) if len(sys.argv) > 2 else Path("/home/ubuntu/Zotero/zotero.sqlite")
HOST_STORAGE = Path(sys.argv[3]) if len(sys.argv) > 3 else Path("/home/ubuntu/data/Zotero/storage")
CONTAINER_STORAGE = Path(sys.argv[4]) if len(sys.argv) > 4 else Path("/home/ubuntu/Zotero/storage")
CONTAINER_DB = (
Path(sys.argv[2])
if len(sys.argv) > 2
else Path("/home/ubuntu/Zotero/zotero.sqlite")
)
HOST_STORAGE = (
Path(sys.argv[3]) if len(sys.argv) > 3 else Path("/home/ubuntu/data/Zotero/storage")
)
CONTAINER_STORAGE = (
Path(sys.argv[4]) if len(sys.argv) > 4 else Path("/home/ubuntu/Zotero/storage")
)
LIBRARY_ID = 1 # user library
@@ -41,16 +49,32 @@ def main():
dst.execute("PRAGMA foreign_keys=OFF") # we handle ordering ourselves
# Collect existing keys in destination
existing_keys = {r[0] for r in dst.execute("SELECT key FROM items WHERE libraryID=?", (LIBRARY_ID,))}
existing_keys |= {r[0] for r in dst.execute("SELECT key FROM collections WHERE libraryID=?", (LIBRARY_ID,))}
existing_keys = {
r[0]
for r in dst.execute("SELECT key FROM items WHERE libraryID=?", (LIBRARY_ID,))
}
existing_keys |= {
r[0]
for r in dst.execute(
"SELECT key FROM collections WHERE libraryID=?", (LIBRARY_ID,)
)
}
# --- Max IDs in destination ---
max_item_id = dst.execute("SELECT COALESCE(MAX(itemID),0) FROM items").fetchone()[0]
max_coll_id = dst.execute("SELECT COALESCE(MAX(collectionID),0) FROM collections").fetchone()[0]
max_value_id = dst.execute("SELECT COALESCE(MAX(valueID),0) FROM itemDataValues").fetchone()[0]
max_creator_id = dst.execute("SELECT COALESCE(MAX(creatorID),0) FROM creators").fetchone()[0]
max_coll_id = dst.execute(
"SELECT COALESCE(MAX(collectionID),0) FROM collections"
).fetchone()[0]
max_value_id = dst.execute(
"SELECT COALESCE(MAX(valueID),0) FROM itemDataValues"
).fetchone()[0]
max_creator_id = dst.execute(
"SELECT COALESCE(MAX(creatorID),0) FROM creators"
).fetchone()[0]
max_tag_id = dst.execute("SELECT COALESCE(MAX(tagID),0) FROM tags").fetchone()[0]
max_word_id = dst.execute("SELECT COALESCE(MAX(wordID),0) FROM fulltextWords").fetchone()[0]
max_word_id = dst.execute(
"SELECT COALESCE(MAX(wordID),0) FROM fulltextWords"
).fetchone()[0]
# ============================================================
# 1. Create "archive" collection + mirror host collection tree
@@ -103,7 +127,10 @@ def main():
src_value_map[r["valueID"]] = dst_values[val]
else:
max_value_id += 1
dst.execute("INSERT INTO itemDataValues (valueID, value) VALUES (?,?)", (max_value_id, r["value"]))
dst.execute(
"INSERT INTO itemDataValues (valueID, value) VALUES (?,?)",
(max_value_id, r["value"]),
)
dst_values[val] = max_value_id
src_value_map[r["valueID"]] = max_value_id
print(f"Mapped {len(src_value_map)} itemDataValues")
@@ -112,11 +139,15 @@ def main():
# 3. Migrate creators (deduplicated by lastName+firstName+fieldMode)
# ============================================================
dst_creators: dict[tuple, int] = {}
for r in dst.execute("SELECT creatorID, firstName, lastName, fieldMode FROM creators"):
for r in dst.execute(
"SELECT creatorID, firstName, lastName, fieldMode FROM creators"
):
dst_creators[(r["lastName"], r["firstName"], r["fieldMode"])] = r["creatorID"]
src_creator_map: dict[int, int] = {}
for r in src.execute("SELECT creatorID, firstName, lastName, fieldMode FROM creators"):
for r in src.execute(
"SELECT creatorID, firstName, lastName, fieldMode FROM creators"
):
ck = (r["lastName"], r["firstName"], r["fieldMode"])
if ck in dst_creators:
src_creator_map[r["creatorID"]] = dst_creators[ck]
@@ -143,7 +174,9 @@ def main():
src_tag_map[r["tagID"]] = dst_tags[r["name"]]
else:
max_tag_id += 1
dst.execute("INSERT INTO tags (tagID, name) VALUES (?,?)", (max_tag_id, r["name"]))
dst.execute(
"INSERT INTO tags (tagID, name) VALUES (?,?)", (max_tag_id, r["name"])
)
dst_tags[r["name"]] = max_tag_id
src_tag_map[r["tagID"]] = max_tag_id
print(f"Mapped {len(src_tag_map)} tags")
@@ -161,7 +194,10 @@ def main():
src_word_map[r["wordID"]] = dst_words[r["word"]]
else:
max_word_id += 1
dst.execute("INSERT INTO fulltextWords (wordID, word) VALUES (?,?)", (max_word_id, r["word"]))
dst.execute(
"INSERT INTO fulltextWords (wordID, word) VALUES (?,?)",
(max_word_id, r["word"]),
)
dst_words[r["word"]] = max_word_id
src_word_map[r["wordID"]] = max_word_id
print(f"Mapped {len(src_word_map)} fulltextWords")
@@ -187,7 +223,15 @@ def main():
dst.execute(
"INSERT INTO items (itemID, itemTypeID, dateAdded, dateModified, clientDateModified, libraryID, key, version, synced) "
"VALUES (?,?,?,?,?,?,?,0,0)",
(max_item_id, hi["itemTypeID"], hi["dateAdded"], hi["dateModified"], hi["clientDateModified"], LIBRARY_ID, new_key),
(
max_item_id,
hi["itemTypeID"],
hi["dateAdded"],
hi["dateModified"],
hi["clientDateModified"],
LIBRARY_ID,
new_key,
),
)
print(f"Migrated {len(item_id_map)} items")
@@ -209,12 +253,19 @@ def main():
# 8. Migrate itemCreators
# ============================================================
count = 0
for r in src.execute("SELECT itemID, creatorID, creatorTypeID, orderIndex FROM itemCreators"):
for r in src.execute(
"SELECT itemID, creatorID, creatorTypeID, orderIndex FROM itemCreators"
):
if r["itemID"] not in item_id_map:
continue
dst.execute(
"INSERT INTO itemCreators (itemID, creatorID, creatorTypeID, orderIndex) VALUES (?,?,?,?)",
(item_id_map[r["itemID"]], src_creator_map[r["creatorID"]], r["creatorTypeID"], r["orderIndex"]),
(
item_id_map[r["itemID"]],
src_creator_map[r["creatorID"]],
r["creatorTypeID"],
r["orderIndex"],
),
)
count += 1
print(f"Migrated {count} itemCreators rows")
@@ -223,14 +274,30 @@ def main():
# 9. Migrate itemAttachments
# ============================================================
count = 0
for r in src.execute("SELECT itemID, parentItemID, linkMode, contentType, charsetID, path, syncState, storageModTime, storageHash FROM itemAttachments"):
for r in src.execute(
"SELECT itemID, parentItemID, linkMode, contentType, charsetID, path, syncState, storageModTime, storageHash FROM itemAttachments"
):
if r["itemID"] not in item_id_map:
continue
parent = item_id_map.get(r["parentItemID"]) if r["parentItemID"] is not None else None
parent = (
item_id_map.get(r["parentItemID"])
if r["parentItemID"] is not None
else None
)
dst.execute(
"INSERT INTO itemAttachments (itemID, parentItemID, linkMode, contentType, charsetID, path, syncState, storageModTime, storageHash) "
"VALUES (?,?,?,?,?,?,?,?,?)",
(item_id_map[r["itemID"]], parent, r["linkMode"], r["contentType"], r["charsetID"], r["path"], r["syncState"], r["storageModTime"], r["storageHash"]),
(
item_id_map[r["itemID"]],
parent,
r["linkMode"],
r["contentType"],
r["charsetID"],
r["path"],
r["syncState"],
r["storageModTime"],
r["storageHash"],
),
)
count += 1
print(f"Migrated {count} itemAttachments rows")
@@ -242,7 +309,11 @@ def main():
for r in src.execute("SELECT itemID, parentItemID, note, title FROM itemNotes"):
if r["itemID"] not in item_id_map:
continue
parent = item_id_map.get(r["parentItemID"]) if r["parentItemID"] is not None else None
parent = (
item_id_map.get(r["parentItemID"])
if r["parentItemID"] is not None
else None
)
dst.execute(
"INSERT INTO itemNotes (itemID, parentItemID, note, title) VALUES (?,?,?,?)",
(item_id_map[r["itemID"]], parent, r["note"], r["title"]),
@@ -283,13 +354,21 @@ def main():
# 13. Migrate fulltextItems + fulltextItemWords
# ============================================================
count = 0
for r in src.execute("SELECT itemID, indexedPages, totalPages, indexedChars, totalChars FROM fulltextItems"):
for r in src.execute(
"SELECT itemID, indexedPages, totalPages, indexedChars, totalChars FROM fulltextItems"
):
if r["itemID"] not in item_id_map:
continue
dst.execute(
"INSERT INTO fulltextItems (itemID, indexedPages, totalPages, indexedChars, totalChars, version, synced) "
"VALUES (?,?,?,?,?,0,0)",
(item_id_map[r["itemID"]], r["indexedPages"], r["totalPages"], r["indexedChars"], r["totalChars"]),
(
item_id_map[r["itemID"]],
r["indexedPages"],
r["totalPages"],
r["indexedChars"],
r["totalChars"],
),
)
count += 1
print(f"Migrated {count} fulltextItems rows")
@@ -312,13 +391,17 @@ def main():
# ============================================================
# Zotero triggers prevent adding child attachments/notes to collections
child_item_ids = set()
for r in src.execute("SELECT itemID FROM itemAttachments WHERE parentItemID IS NOT NULL"):
for r in src.execute(
"SELECT itemID FROM itemAttachments WHERE parentItemID IS NOT NULL"
):
child_item_ids.add(r["itemID"])
for r in src.execute("SELECT itemID FROM itemNotes WHERE parentItemID IS NOT NULL"):
child_item_ids.add(r["itemID"])
count = 0
for r in src.execute("SELECT collectionID, itemID, orderIndex FROM collectionItems"):
for r in src.execute(
"SELECT collectionID, itemID, orderIndex FROM collectionItems"
):
if r["itemID"] not in item_id_map:
continue
if r["collectionID"] not in coll_id_map:

105
dev/scripts/zotero_fix_fields.py
View File

@@ -1,100 +1,27 @@
"""Fix Zotero items that have data stored under base field IDs instead of mapped field IDs.
"""Thin shim — logic moved to :mod:`zot.ops`.
Zotero uses type-specific fields (e.g. caseName for case items) that map to base fields
(e.g. title). When items have data under the base field ID but their item type requires the
mapped field ID, Zotero logs "is not a valid field" errors.
This script remaps base field IDs to the correct type-specific field IDs, and removes
truly invalid field data (fields with no mapping for that type).
Prefer: ``uv run stack zot fix-fields``.
"""
import sqlite3
from __future__ import annotations
import sys
from pathlib import Path
DB = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/home/ubuntu/Zotero/zotero.sqlite")
def main() -> int:
from zot.ops import fix_fields
def main():
db = sqlite3.connect(str(DB))
db.row_factory = sqlite3.Row
db.execute("PRAGMA journal_mode=WAL")
db.execute("PRAGMA foreign_keys=OFF")
# Build set of valid (itemTypeID, fieldID) pairs
valid_fields: set[tuple[int, int]] = set()
for r in db.execute("SELECT itemTypeID, fieldID FROM itemTypeFieldsCombined"):
valid_fields.add((r["itemTypeID"], r["fieldID"]))
# Build base field mapping: (itemTypeID, baseFieldID) -> mappedFieldID
field_map: dict[tuple[int, int], int] = {}
for r in db.execute("SELECT itemTypeID, baseFieldID, fieldID FROM baseFieldMappingsCombined"):
field_map[(r["itemTypeID"], r["baseFieldID"])] = r["fieldID"]
# Find all invalid itemData rows
# An itemData row is invalid if (itemTypeID, fieldID) is not in valid_fields
invalid = db.execute("""
SELECT id.itemID, id.fieldID, id.valueID, i.itemTypeID
FROM itemData id
JOIN items i ON id.itemID = i.itemID
WHERE i.libraryID = 1
AND (i.itemTypeID, id.fieldID) NOT IN (
SELECT itemTypeID, fieldID FROM itemTypeFieldsCombined
)
""").fetchall()
print(f"Found {len(invalid)} invalid itemData rows")
remapped = 0
deleted = 0
conflicts = 0
for row in invalid:
item_id = row["itemID"]
old_field = row["fieldID"]
value_id = row["valueID"]
item_type = row["itemTypeID"]
# Check if there's a mapping for this base field to a type-specific field
mapped_field = field_map.get((item_type, old_field))
if mapped_field and (item_type, mapped_field) in valid_fields:
# Check if the mapped field already has data for this item
existing = db.execute(
"SELECT valueID FROM itemData WHERE itemID=? AND fieldID=?",
(item_id, mapped_field),
).fetchone()
if existing:
# Mapped field already has data — delete the base field entry
db.execute(
"DELETE FROM itemData WHERE itemID=? AND fieldID=?",
(item_id, old_field),
)
conflicts += 1
else:
# Remap: update fieldID from base to mapped
db.execute(
"UPDATE itemData SET fieldID=? WHERE itemID=? AND fieldID=?",
(mapped_field, item_id, old_field),
)
remapped += 1
else:
# No valid mapping exists — delete the orphan data
db.execute(
"DELETE FROM itemData WHERE itemID=? AND fieldID=?",
(item_id, old_field),
)
deleted += 1
db.commit()
db.close()
print(f"Remapped: {remapped}")
print(f"Deleted (no mapping): {deleted}")
print(f"Deleted (conflict): {conflicts}")
print(f"Total fixed: {remapped + deleted + conflicts}")
db = (
Path(sys.argv[1])
if len(sys.argv) > 1
else Path("/home/ubuntu/Zotero/zotero.sqlite")
)
out = fix_fields(db)
for k, v in out.items():
print(f"{k}: {v}")
return 0
if __name__ == "__main__":
main()
sys.exit(main())

145
infra/droplets/mail-setup.sh Executable file
View File

@@ -0,0 +1,145 @@
#!/bin/bash
# Provision a DigitalOcean Maddy mail droplet.
#
# Runs under cloud-init with these env vars already exported by the
# builder (see src/cli/mail.py::_cloud_init):
# HOSTNAME fqdn of this mail server, e.g. mail.corwins.media
# PRIMARY_DOMAIN bare domain, e.g. corwins.media
# POSTMASTER_PASSWORD initial password for postmaster@<domain>
# GITEA_SMTP_PASSWORD initial password for gitea@<domain> (app-only account)
#
# Idempotent — safe to re-run. Installs Docker + Maddy, lays down config,
# provisions the two seed accounts, and opens the firewall for SMTP/IMAP.
set -euo pipefail
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y -qq curl gettext-base ca-certificates ufw
# ── Docker Engine (upstream, not the ancient distro package) ──────
if ! command -v docker >/dev/null 2>&1; then
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
gpg --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
> /etc/apt/sources.list.d/docker.list
apt-get update -qq
apt-get install -y -qq docker-ce docker-ce-cli containerd.io docker-compose-plugin
systemctl enable --now docker
fi
# ── Maddy config + data dir ───────────────────────────────────────
install -d -m 0750 /srv/mail /srv/mail/tls /srv/mail/data
cat > /srv/mail/maddy.conf.tpl <<'MADDY_CONF_EOF'
__MADDY_CONF_TPL_PLACEHOLDER__
MADDY_CONF_EOF
# Render config atomically and detect changes so we only restart maddy
# when the rendered file actually differs.
envsubst '$HOSTNAME $PRIMARY_DOMAIN' \
< /srv/mail/maddy.conf.tpl > /srv/mail/data/maddy.conf.new
if ! cmp -s /srv/mail/data/maddy.conf /srv/mail/data/maddy.conf.new 2>/dev/null; then
mv /srv/mail/data/maddy.conf.new /srv/mail/data/maddy.conf
MADDY_CONF_CHANGED=1
else
rm -f /srv/mail/data/maddy.conf.new
fi
# Empty aliases file so the local_aliases chain doesn't error on first boot
[ -f /srv/mail/data/aliases ] || : > /srv/mail/data/aliases
# ── docker-compose.yml ────────────────────────────────────────────
cat > /srv/mail/docker-compose.yml <<COMPOSE_EOF
services:
mail:
image: foxcpp/maddy:latest
container_name: mail
restart: unless-stopped
hostname: ${HOSTNAME}
environment:
MADDY_HOSTNAME: ${HOSTNAME}
MADDY_DOMAIN: ${PRIMARY_DOMAIN}
ports:
- "25:25"
- "143:143"
- "465:465"
- "587:587"
- "993:993"
volumes:
- ./data:/data
- ./tls:/data/tls:ro
COMPOSE_EOF
# ── Let's Encrypt via DNS-01 (Cloudflare plugin) ──────────────────
# DNS-01 not HTTP-01 because:
# 1. The mail.$(domain) A record doesn't yet point at this droplet
# when cloud-init runs (the CLI writes DNS *after* the droplet is
# up), so HTTP-01 would race and fail.
# 2. Port 80 doesn't need to be internet-facing on a mail server.
# 3. DNS-01 works even when the droplet has no inbound HTTP at all.
#
# Requires CLOUDFLARE_API_TOKEN to be exported by the user_data wrapper.
apt-get install -y -qq certbot python3-certbot-dns-cloudflare
mkdir -p /etc/letsencrypt/cloudflare
cat > /etc/letsencrypt/cloudflare/credentials.ini <<CF_EOF
dns_cloudflare_api_token = ${CLOUDFLARE_API_TOKEN}
CF_EOF
chmod 600 /etc/letsencrypt/cloudflare/credentials.ini
# --keep-until-expiring makes this safe to re-run.
certbot certonly --dns-cloudflare \
--dns-cloudflare-credentials /etc/letsencrypt/cloudflare/credentials.ini \
--dns-cloudflare-propagation-seconds 20 \
--non-interactive --agree-tos --keep-until-expiring \
-m "postmaster@${PRIMARY_DOMAIN}" -d "${HOSTNAME}" || true
if [ -f "/etc/letsencrypt/live/${HOSTNAME}/fullchain.pem" ]; then
cp /etc/letsencrypt/live/${HOSTNAME}/fullchain.pem /srv/mail/tls/fullchain.pem
cp /etc/letsencrypt/live/${HOSTNAME}/privkey.pem /srv/mail/tls/privkey.pem
chmod 644 /srv/mail/tls/fullchain.pem
chmod 600 /srv/mail/tls/privkey.pem
fi
# Daily renew hook — DNS-01 doesn't need port 80.
cat > /etc/cron.daily/maddy-cert-renew <<'CRON_EOF'
#!/bin/bash
set -e
certbot renew --quiet --deploy-hook "cp /etc/letsencrypt/live/${HOSTNAME}/fullchain.pem /srv/mail/tls/ && \
cp /etc/letsencrypt/live/${HOSTNAME}/privkey.pem /srv/mail/tls/ && \
docker restart mail"
CRON_EOF
chmod +x /etc/cron.daily/maddy-cert-renew
# ── Firewall ──────────────────────────────────────────────────────
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw allow 25/tcp
ufw allow 143/tcp
ufw allow 465/tcp
ufw allow 587/tcp
ufw allow 993/tcp
ufw --force enable
# ── Bring up maddy ────────────────────────────────────────────────
cd /srv/mail
docker compose pull
docker compose up -d
# Only restart when config actually drifted — keeps re-runs cheap.
if [ "${MADDY_CONF_CHANGED:-0}" = "1" ]; then
docker restart mail
fi
# Mailbox seeding is owned by the orchestrator (`stack mail` /
# `corwins mail`), not this script. The orchestrator SSHes in after
# Maddy is up and runs `maddy creds create` per mailbox. Keeping
# seeding out of cloud-init means adding a new mailbox is one CLI
# call (`stack mail rotate-creds <name>`) — no droplet rebuild, no
# template edit, no env-var threading.
echo "==> Maddy provisioning complete for ${HOSTNAME}"

2
infra/gitea/.dockerignore Normal file
View File

@@ -0,0 +1,2 @@
custom/
templates/

725
infra/gitea/CROSSWALK.md Normal file
View File

@@ -0,0 +1,725 @@
# Fhirworx ⇄ Gitea v1.25.4 CSS Variable Crosswalk
This report cross-references every CSS variable Gitea expects against
what fhirworx provides. Format:
- ✅ defined and value matches upstream intent
- ⚠️ defined with different value (intentional remap or risk)
- ❌ NOT defined → resolves to `initial` (transparent bg, black text)
- 🔍 used in source but not declared anywhere
---
## Summary
| Set | Count |
|---|---|
| Upstream theme-gitea-light.css declares | 259 |
| Fhirworx light declares | 253 |
| Fhirworx dark declares | 253 |
| Total distinct --vars referenced in src | 186 |
| Vars used in src but NOT declared anywhere | 6 |
## Coverage verdict
**Light variant: ✅ defines every `--color-*` variable upstream uses (full coverage)**
**Dark variant: ✅ defines every `--color-*` variable upstream uses (full coverage)**
Structural vars (`--font-*`, `--border-radius`, `--gap-*`, etc.) come from `base.css`'s
`:root` and are inherited automatically; themes don't need to override them.
## 🔍 Variables used in source but NOT declared in upstream OR fhirworx
- `--fonts-default-override-ja` — referenced by 1 rules; first: `font_i18n.css::root :lang(ja)`
- `--fonts-default-override-ko` — referenced by 1 rules; first: `font_i18n.css::root :lang(ko)`
- `--fonts-default-override-zh-cn` — referenced by 1 rules; first: `font_i18n.css::root :lang(zh-CN)`
- `--fonts-default-override-zh-hk` — referenced by 1 rules; first: `font_i18n.css::root :lang(zh-HK)`
- `--fonts-default-override-zh-tw` — referenced by 1 rules; first: `font_i18n.css::root :lang(zh-TW)`
- `--fonts-override` — referenced by 1 rules; first: `base.css:} :root *`
## Extra vars defined by fhirworx LIGHT (fhirworx-only conveniences)
- `--color-error` = `var(--color-red)`
- `--color-info` = `var(--color-blue)`
- `--color-primary-foreground` = `var(--fw-primary-foreground)`
- `--color-success` = `var(--color-green)`
- `--color-warning` = `#7F5E08`
- `--fw-background` = `#F7F5F0`
- `--fw-border` = `#B8B3A4`
- `--fw-card` = `#FFFFFF`
- `--fw-card-hover` = `#F3F0EA`
- `--fw-chart-3` = `#A88B5C`
- `--fw-chart-5` = `#C8702A`
- `--fw-destructive` = `#C0392B`
- `--fw-foreground` = `#1A1A18`
- `--fw-link` = `#2D5F7F`
- `--fw-muted-foreground` = `#6B6B68`
- `--fw-nav-active-bg` = `#2E4359`
- `--fw-nav-bg` = `#1C2B3A`
- `--fw-nav-hover-bg` = `#253748`
- `--fw-nav-text` = `#F0F4F8`
- `--fw-primary` = `#1C2B3A`
- `--fw-primary-foreground` = `#F7F5F0`
- `--fw-secondary` = `#EDEBE6`
## Extra vars defined by fhirworx DARK
- `--color-error` = `var(--color-red)`
- `--color-info` = `var(--color-blue)`
- `--color-primary-foreground` = `var(--fw-primary-foreground)`
- `--color-success` = `var(--color-green)`
- `--color-warning` = `var(--color-yellow)`
- `--fonts-override` = `"Source Serif 4", Georgia, serif`
- `--fw-background` = `#0F1419`
- `--fw-border` = `#3A4452`
- `--fw-card` = `#1A2028`
- `--fw-card-hover` = `#232A34`
- `--fw-chart-3` = `#B8A078`
- `--fw-chart-5` = `#E08A4A`
- `--fw-destructive` = `#E07A6E`
- `--fw-foreground` = `#E8E5DD`
- `--fw-link` = `#5C9CC0`
- `--fw-muted-foreground` = `#8A8A87`
- `--fw-nav-active-bg` = `#25384C`
- `--fw-nav-bg` = `#0A0E13`
- `--fw-nav-hover-bg` = `#1C2B3A`
- `--fw-nav-text` = `#E8F0F8`
- `--fw-primary` = `#5684A3`
- `--fw-primary-foreground` = `#0F1419`
- `--fw-secondary` = `#1F262E`
## Variable crosswalk — every var referenced in source
| Var | Upstream value | Fhirworx light | Fhirworx dark | Source consumers (top 3) |
|---|---|---|---|---|
| ❌ `--background-view-image` | `url("data:image/png` | `—` | `—` | features/imagediff.css, repo/file-view.css |
| ❌ `--border-radius` | `4px` | `—` | `—` | base.css, codemirror/base.css, features/codeedito… |
| ❌ `--border-radius-full` | `99999px` | `—` | `—` | base.css, modules/animations.css, repo.css |
| ❌ `--border-radius-medium` | `6px` | `—` | `—` | features/codeeditor.css, features/dropzone.css, r… |
| ❌ `--checkbox-mask-checked` | `url('data:image/svg+xml` | `—` | `—` | markup/content.css |
| ❌ `--checkbox-mask-indeterminate` | `url('data:image/svg+xml` | `—` | `—` | markup/content.css |
| ❌ `--checkbox-size` | `15px` | `—` | `—` | modules/checkbox.css |
| ⚠️ `--color-accent` | `var(--color-primary-light-1)` | `var(--fw-primary)` | `var(--color-primary-dark-1)` | base.css, review.css |
| ⚠️ `--color-active` | `#00001714` | `#0000171F` | `#E8F3FF24` | base.css, editor/fileeditor.css, helpers.css |
| ⚠️ `--color-ansi-black` | `#1e2327` | `#1E2327` | `#1E2327` | features/console.css |
| ⚠️ `--color-ansi-blue` | `#3a8ac6` | `#3A8AC6` | `#3A8AC6` | features/console.css |
| ⚠️ `--color-ansi-bright-black` | `#46494d` | `#46494D` | `#424851` | features/console.css |
| ⚠️ `--color-ansi-bright-blue` | `#4e96cc` | `#4E96CC` | `#4E96CC` | features/console.css |
| ⚠️ `--color-ansi-bright-cyan` | `#00b6ad` | `#00B6AD` | `#00B6AD` | features/console.css |
| ⚠️ `--color-ansi-bright-green` | `#93b373` | `#93B373` | `#93B373` | features/console.css |
| ⚠️ `--color-ansi-bright-magenta` | `#d74397` | `#D74397` | `#D74397` | features/console.css |
| ⚠️ `--color-ansi-bright-red` | `#d15a5a` | `#D15A5A` | `#D15A5A` | features/console.css |
| ✅ `--color-ansi-bright-white` | `var(--color-console-fg)` | `var(--color-console-fg)` | `var(--color-console-fg)` | features/console.css |
| ⚠️ `--color-ansi-bright-yellow` | `#eaaf03` | `#EAAF03` | `#EAAF03` | features/console.css |
| ⚠️ `--color-ansi-cyan` | `#00918a` | `#00918A` | `#00918A` | features/console.css |
| ⚠️ `--color-ansi-green` | `#87ab63` | `#87AB63` | `#87AB63` | features/console.css |
| ⚠️ `--color-ansi-magenta` | `#d22e8b` | `#D22E8B` | `#D22E8B` | features/console.css |
| ⚠️ `--color-ansi-red` | `#cc4848` | `#CC4848` | `#CC4848` | features/console.css |
| ✅ `--color-ansi-white` | `var(--color-console-fg-subtle)` | `var(--color-console-fg-subtle)` | `var(--color-console-fg-subtle)` | features/console.css |
| ⚠️ `--color-ansi-yellow` | `#cc9903` | `#CC9903` | `#CC9903` | features/console.css |
| ⚠️ `--color-blue` | `#2185d0` | `var(--fw-link)` | `var(--fw-link)` | actions.css, base.css, modules/message.css |
| ⚠️ `--color-blue-dark-1` | `#1e78bb` | `#244B63` | `#487FA1` | base.css |
| ⚠️ `--color-body` | `#ffffff` | `var(--fw-background) …` | `var(--fw-background) !importa…` | base.css, editor/fileeditor.css, modules/menu.css |
| ⚠️ `--color-box-body` | `#ffffff` | `var(--fw-card) …` | `var(--fw-card) !importa…` | markup/content.css, modules/menu.css, modules/mes… |
| ⚠️ `--color-box-body-highlight` | `#ecf5fd` | `var(--color-primary-light-7)` | `#1E2630` | repo.css |
| ⚠️ `--color-box-header` | `#f1f3f5` | `var(--fw-secondary)` | `var(--fw-secondary)` | modules/header.css, modules/menu.css, modules/tab… |
| ⚠️ `--color-button` | `#f8f9fb` | `var(--fw-card)` | `var(--fw-card)` | modules/button.css, modules/label.css |
| ⚠️ `--color-card` | `#f8f9fb` | `var(--fw-card) …` | `var(--fw-card) !importa…` | modules/card.css, repo/issue-card.css, user.css |
| ✅ `--color-caret` | `var(--color-text-dark)` | `var(--color-text-dark)` | `var(--color-text)` | base.css, codemirror/base.css |
| ⚠️ `--color-code-bg` | `#fafdff` | `var(--fw-card)` | `#0B0F14` | repo.css, repo/file-view.css |
| ⚠️ `--color-console-bg` | `#171b1e` | `#0F1419` | `#050910` | features/console.css |
| ⚠️ `--color-console-fg` | `#f7f8f9` | `#F7F5F0` | `#F7F8F9` | features/console.css |
| ⚠️ `--color-diff-added-linenum-bg` | `#d1f8d9` | `#CDEED6` | `#1E3C2A` | repo.css |
| ⚠️ `--color-diff-added-row-bg` | `#e6ffed` | `#E2F5E8` | `#18301E` | repo.css |
| ⚠️ `--color-diff-added-row-border` | `#e6ffed` | `#C2E0CB` | `#2A4A32` | repo.css |
| ⚠️ `--color-diff-added-word-bg` | `#acf2bd` | `#A8E0B7` | `#2E5C3E` | repo.css |
| ⚠️ `--color-diff-inactive` | `#f0f2f4` | `#EEECE7` | `#1A1F26` | repo.css |
| ⚠️ `--color-diff-moved-row-bg` | `#f1f8d1` | `#EEF4D8` | `#423E20` | repo.css |
| ⚠️ `--color-diff-removed-linenum-bg` | `#ffcecb` | `#F5C8C5` | `#402020` | repo.css |
| ⚠️ `--color-diff-removed-row-bg` | `#ffeef0` | `#FAE8E8` | `#2C1818` | repo.css |
| ⚠️ `--color-diff-removed-row-border` | `#f1c0c0` | `#EBBFBF` | `#5A3434` | repo.css |
| ⚠️ `--color-diff-removed-word-bg` | `#fdb8c0` | `#F3B5BC` | `#5C2D2D` | repo.css |
| ⚠️ `--color-error-bg` | `#fff6f6` | `#F6E5E4` | `#3C1F1F` | form.css, modules/header.css, modules/input.css |
| ⚠️ `--color-error-bg-active` | `#fbb` | `#E8ADAD` | `#5C2E2E` | form.css |
| ⚠️ `--color-error-bg-hover` | `#fdd` | `#F0C8C8` | `#4A2525` | form.css |
| ⚠️ `--color-error-border` | `#e0b4b4` | `#D48B8B` | `#8A3B3B` | form.css, modules/header.css, modules/input.css |
| ⚠️ `--color-error-text` | `#9f3a38` | `#8E2E2E` | `#F0B0B0` | form.css, modules/header.css, modules/input.css |
| ⚠️ `--color-expand-button` | `#cfe8fa` | `var(--color-primary-light-6)` | `#2A3845` | review.css |
| ✅ `--color-footer` | `var(--color-nav-bg)` | `var(--color-nav-bg)` | `var(--color-nav-bg)` | home.css |
| ⚠️ `--color-git` | `#f05133` | `#F05133` | `#F05133` | base.css |
| ⚠️ `--color-gold` | `#a1882b` | `var(--fw-chart-3)` | `var(--fw-chart-3)` | base.css |
| ⚠️ `--color-green` | `#21ba45` | `#247058` | `#4CB28A` | actions.css, base.css, features/imagediff.css |
| ⚠️ `--color-green-badge` | `#21ba45` | `var(--color-green)` | `var(--color-green)` | repo/commit-sign.css |
| ⚠️ `--color-green-badge-bg` | `#21ba451a` | `#2470581A` | `#4CB28A1A` | repo/commit-sign.css |
| ⚠️ `--color-green-badge-hover-bg` | `#21ba454d` | `#2470584D` | `#4CB28A4D` | repo/commit-sign.css |
| ⚠️ `--color-green-dark-1` | `#1ea73e` | `#1E5E49` | `#3A9876` | modules/button.css, modules/label.css |
| ⚠️ `--color-green-dark-2` | `#1a9537` | `#17483A` | `#2D7C5F` | modules/button.css |
| ⚠️ `--color-grey` | `#697077` | `var(--fw-muted-foreground)` | `var(--fw-muted-foreground)` | repo.css |
| ⚠️ `--color-grey-light` | `#7c838a` | `#8A8A87` | `#98948C` | base.css |
| ⚠️ `--color-highlight-bg` | `#fffbdd` | `#FFF3C4` | `#3A2E1A` | repo/file-view.css |
| ⚠️ `--color-highlight-fg` | `#eed200` | `var(--fw-chart-3)` | `#C9A347` | repo/file-view.css |
| ⚠️ `--color-hover` | `#00001708` | `#00001710` | `#E8F3FF14` | base.css, editor/fileeditor.css, helpers.css |
| ⚠️ `--color-hover-opaque` | `#f1f3f5` | `var(--fw-card-hover)` | `var(--fw-card-hover)` | repo/home-file-list.css |
| ⚠️ `--color-info-bg` | `#f8ffff` | `#E8F0F6` | `#1A2E44` | modules/message.css |
| ⚠️ `--color-info-border` | `#a9d5de` | `#A5BCCD` | `#3A6A8F` | modules/message.css |
| ⚠️ `--color-info-text` | `#276f86` | `var(--color-blue)` | `var(--color-blue)` | modules/message.css |
| ⚠️ `--color-input-background` | `#fff` | `var(--fw-card)` | `#12171E` | codemirror/base.css, form.css, markup/content.css |
| ✅ `--color-input-border` | `var(--color-secondary)` | `var(--color-secondary)` | `var(--color-secondary)` | form.css, modules/input.css |
| ⚠️ `--color-input-border-hover` | `var(--color-secondary-dark-1)` | `var(--color-secondary-dark-2)` | `var(--color-secondary-dark-1)` | form.css |
| ✅ `--color-input-text` | `var(--color-text-dark)` | `var(--color-text-dark)` | `var(--color-text-dark)` | codemirror/base.css, form.css, modules/input.css |
| ⚠️ `--color-input-toggle-background` | `#d0d7de` | `var(--fw-border)` | `#2A323D` | modules/checkbox.css |
| ⚠️ `--color-label-bg` | `#949da64b` | `#94908648` | `#6A7A8E4B` | base.css, modules/label.css, modules/menu.css |
| ⚠️ `--color-label-hover-bg` | `#949da6a0` | `#9490869E` | `#6A7A8EA0` | modules/label.css |
| ⚠️ `--color-label-text` | `var(--color-text)` | `var(--fw-foreground)` | `var(--fw-foreground)` | modules/label.css, modules/menu.css |
| ⚠️ `--color-light` | `#00001706` | `#EFECE5` | `#00001728` | base.css, modules/table.css, repo.css |
| ⚠️ `--color-light-border` | `#0000171d` | `#00001728` | `#E8F3FF28` | base.css, modules/button.css, modules/label.css |
| ⚠️ `--color-logo` | `#609926` | `var(--fw-primary)` | `var(--fw-primary)` | home.css |
| ✅ `--color-markup-code-block` | `#00306010` | `#00306010` | `#E8F3FF12` | markup/content.css, repo.css |
| ⚠️ `--color-markup-code-inline` | `#00306012` | `#00306014` | `#E8F3FF20` | markup/content.css |
| ⚠️ `--color-markup-table-row` | `#0030600a` | `#00306008` | `#E8F3FF0F` | markup/content.css |
| ⚠️ `--color-menu` | `#f8f9fb` | `var(--fw-card) …` | `var(--fw-card) !importa…` | base.css, features/expander.css, modules/menu.css |
| ⚠️ `--color-nav-bg` | `#f6f7fa` | `var(--fw-nav-bg) …` | `var(--fw-nav-bg) !importa…` | modules/navbar.css |
| ⚠️ `--color-nav-hover-bg` | `var(--color-secondary-light-1)` | `var(--fw-nav-hover-bg)` | `var(--fw-nav-hover-bg)` | modules/navbar.css |
| ⚠️ `--color-nav-text` | `var(--color-text)` | `var(--fw-nav-text) …` | `var(--fw-nav-text) !importa…` | modules/navbar.css |
| ⚠️ `--color-olive` | `#b5cc18` | `#8B9A4C` | `#ABC075` | modules/label.css |
| ⚠️ `--color-orange` | `#f2711c` | `var(--fw-chart-5)` | `var(--fw-chart-5)` | base.css, modules/label.css |
| ⚠️ `--color-orange-badge` | `#f2711c` | `var(--color-orange)` | `var(--color-orange)` | repo/commit-sign.css |
| ⚠️ `--color-orange-badge-bg` | `#f2711c1a` | `#C8702A1A` | `#E08A4A1A` | repo/commit-sign.css |
| ⚠️ `--color-orange-badge-hover-bg` | `#f2711c4d` | `#C8702A4D` | `#E08A4A4D` | repo/commit-sign.css |
| ⚠️ `--color-orange-dark-1` | `#e6630d` | `#B4621F` | `#C87239` | modules/label.css |
| ⚠️ `--color-overlay-backdrop` | `#080808c0` | `#000017C0` | `#000B17D0` | modules/dimmer.css |
| ✅ `--color-placeholder-text` | `var(--color-text-light-3)` | `var(--color-text-light-3)` | `var(--color-text-light-3)` | base.css, codemirror/base.css |
| ⚠️ `--color-primary` | `#4183c4` | `var(--fw-primary)` | `var(--fw-primary)` | base.css, codemirror/base.css, editor/combomarkdo… |
| ⚠️ `--color-primary-active` | `var(--color-primary-dark-2)` | `var(--color-primary-dark-1)` | `var(--color-primary-dark-2)` | helpers.css, modules/button.css |
| ⚠️ `--color-primary-alpha-30` | `#4183c44b` | `#1C2B3A4B` | `#5684A34B` | repo.css |
| ⚠️ `--color-primary-contrast` | `#ffffff` | `var(--fw-primary-foreground)` | `var(--fw-primary-foreground)` | base.css, features/expander.css, modules/button.c… |
| ⚠️ `--color-primary-dark-1` | `#3876b3` | `#172530` | `#6D97B3` | modules/label.css |
| ⚠️ `--color-primary-dark-2` | `#31699f` | `#121F28` | `#85ABC2` | modules/label.css |
| ⚠️ `--color-primary-dark-3` | `#2b5c8b` | `#0E1920` | `#9CBED1` | modules/label.css |
| ⚠️ `--color-primary-hover` | `var(--color-primary-dark-1)` | `var(--color-primary-light-1)` | `var(--color-primary-dark-1)` | modules/button.css |
| ⚠️ `--color-primary-light-1` | `#548fca` | `#2E4A63` | `#4A7590` | base.css, codemirror/base.css |
| ⚠️ `--color-primary-light-4` | `#8db5dc` | `#7BA0BC` | `#294557` | form.css, review.css |
| ⚠️ `--color-primary-light-5` | `#b3cde7` | `#A5BCCD` | `#203644` | review.css |
| ⚠️ `--color-primary-light-6` | `#d9e6f3` | `#D2DDE6` | `#162531` | form.css |
| ⚠️ `--color-primary-light-7` | `#f4f8fb` | `#EEF2F6` | `#0D1620` | base.css, features/imagediff.css |
| ✅ `--color-project-column-bg` | `var(--color-secondary-light-4)` | `var(--color-secondary-light-4)` | `var(--color-secondary-light-2)` | features/projects.css |
| ⚠️ `--color-purple` | `#a333c8` | `#8A4BA8` | `#B478D4` | base.css, modules/label.css |
| ⚠️ `--color-purple-dark-1` | `#932eb4` | `#6E388A` | `#9860B7` | modules/label.css |
| ⚠️ `--color-reaction-active-bg` | `var(--color-primary-light-6)` | `var(--color-primary-light-5)` | `var(--color-primary-light-5)` | repo/reactions.css |
| ⚠️ `--color-reaction-hover-bg` | `var(--color-primary-light-5)` | `var(--color-primary-light-6)` | `var(--color-primary-light-4)` | repo/reactions.css |
| ⚠️ `--color-red` | `#db2828` | `var(--fw-destructive)` | `var(--fw-destructive)` | base.css, dashboard.css, features/imagediff.css |
| ⚠️ `--color-red-badge` | `#db2828` | `var(--color-red)` | `var(--color-red)` | repo/commit-sign.css |
| ⚠️ `--color-red-badge-bg` | `#db28281a` | `#C0392B1A` | `#E07A6E1A` | repo/commit-sign.css |
| ⚠️ `--color-red-badge-hover-bg` | `#db28284d` | `#C0392B4D` | `#E07A6E4D` | repo/commit-sign.css |
| ⚠️ `--color-red-dark-1` | `#c82121` | `#A93024` | `#C86559` | base.css, modules/button.css, modules/label.css |
| ⚠️ `--color-red-dark-2` | `#b11e1e` | `#8D2720` | `#A74D42` | modules/button.css |
| ⚠️ `--color-red-light` | `#e45e5e` | `#D66A5C` | `#E89891` | actions.css |
| ⚠️ `--color-secondary` | `#d0d7de` | `var(--fw-border)` | `var(--fw-border)` | admin.css, base.css, codemirror/base.css |
| ⚠️ `--color-secondary-alpha-20` | `#d0d7de33` | `#B8B3A433` | `#3A445233` | repo.css |
| ⚠️ `--color-secondary-alpha-50` | `#d0d7de80` | `#B8B3A480` | `#3A445280` | modules/table.css, repo/issue-list.css |
| ⚠️ `--color-secondary-bg` | `#f2f5f8` | `var(--fw-secondary)` | `var(--fw-secondary)` | modules/modal.css, modules/segment.css |
| ⚠️ `--color-secondary-dark-1` | `#c7ced5` | `#A49F90` | `#45505E` | base.css, home.css, markup/codecopy.css |
| ⚠️ `--color-secondary-dark-2` | `#b9c0c7` | `#908B7D` | `#505C6A` | base.css, modules/button.css, modules/label.css |
| ⚠️ `--color-secondary-dark-4` | `#899097` | `#6A6558` | `#6E7A89` | features/projects.css, repo/issue-list.css |
| ⚠️ `--color-secondary-dark-5` | `#7a8188` | `#5A554A` | `#7E8A99` | features/gitgraph.css, form.css |
| ⚠️ `--color-secondary-dark-7` | `#5b6269` | `#3A372F` | `#9CA7B4` | base.css |
| ⚠️ `--color-secondary-dark-8` | `#4b5259` | `#2C2924` | `#ABB5C0` | features/imagediff.css, modules/animations.css |
| ⚠️ `--color-secondary-light-1` | `#dee5ec` | `#C7C3B6` | `#2F3845` | modules/card.css |
| ⚠️ `--color-secondary-nav-bg` | `#f9fafb` | `var(--color-secondary-light-4)` | `var(--color-secondary-light-3)` | modules/navbar.css |
| ✅ `--color-shadow` | `#00001726` | `#00001726` | `#00001758` | base.css, features/expander.css, modules/checkbox… |
| ✅ `--color-small-accent` | `var(--color-primary-light-6)` | `var(--color-primary-light-6)` | `var(--color-primary-light-4)` | review.css |
| ⚠️ `--color-success-bg` | `#fcfff5` | `#E8F2DD` | `#1F3A2A` | modules/message.css |
| ⚠️ `--color-success-border` | `#a3c293` | `#A3C293` | `#3F7D4F` | modules/message.css |
| ⚠️ `--color-success-text` | `#2c662d` | `#264F26` | `#7EC89A` | base.css, modules/message.css |
| ⚠️ `--color-teal` | `#00b5ad` | `#3A9A94` | `#66C7C0` | repo.css |
| ⚠️ `--color-text` | `#181c21` | `var(--fw-foreground) …` | `var(--fw-foreground) !importa…` | base.css, editor/combomarkdowneditor.css, feature… |
| ⚠️ `--color-text-dark` | `#01050a` | `#0D0D0C` | `#F5F2EA` | base.css, modules/list.css, modules/menu.css |
| ⚠️ `--color-text-light` | `#30363b` | `#2E2E2C` | `#CFCCC5` | actions.css, base.css, editor/fileeditor.css |
| ⚠️ `--color-text-light-1` | `#40474d` | `#454542` | `#B4B0A8` | base.css, features/expander.css, form.css |
| ⚠️ `--color-text-light-2` | `#5b6167` | `#595955` | `#9A968E` | base.css, markup/content.css, modules/breadcrumb.… |
| ⚠️ `--color-text-light-3` | `#747c84` | `#6E6E6A` | `#807C74` | base.css, features/gitgraph.css, repo.css |
| ⚠️ `--color-timeline` | `#d0d7de` | `var(--fw-border)` | `var(--fw-border)` | repo.css |
| ⚠️ `--color-tooltip-bg` | `#000017f0` | `#111B25F0` | `#000B17F0` | modules/tippy.css |
| ⚠️ `--color-tooltip-text` | `#fbfdff` | `var(--fw-primary-foreground)` | `#F5F2EA` | modules/tippy.css |
| ⚠️ `--color-violet-dark-1` | `#5a30b5` | `#55418A` | `#7E65BD` | base.css |
| ⚠️ `--color-warning-bg` | `#fffaf3` | `#F8F0DC` | `#3A3220` | modules/header.css, modules/message.css |
| ⚠️ `--color-warning-border` | `#c9ba9b` | `#C9BA9B` | `#8A7A30` | modules/header.css, modules/message.css, modules/… |
| ⚠️ `--color-warning-text` | `#573a08` | `#5A3D10` | `#E5BE5A` | base.css, modules/header.css, modules/message.css |
| ⚠️ `--color-white` | `#ffffff` | `#FFFFFF` | `#FFFFFF` | actions.css, base.css, codemirror/base.css |
| ⚠️ `--color-yellow` | `#fbbd08` | `#D4A017` | `#E5BE5A` | actions.css, base.css, modules/label.css |
| ⚠️ `--color-yellow-badge` | `#fbbd08` | `var(--color-yellow)` | `var(--color-yellow)` | repo/commit-sign.css |
| ⚠️ `--color-yellow-badge-bg` | `#fbbd081a` | `#D4A0171A` | `#E5BE5A1A` | repo/commit-sign.css |
| ⚠️ `--color-yellow-badge-hover-bg` | `#fbbd084d` | `#D4A0174D` | `#E5BE5A4D` | repo/commit-sign.css |
| ⚠️ `--color-yellow-dark-1` | `#e5ac04` | `#B38712` | `#C9A347` | modules/label.css |
| ❌ `--font-size-label` | `12px` | `—` | `—` | modules/label.css |
| ❌ `--font-weight-bold` | `700` | `—` | `—` | base.css, modules/navbar.css |
| ❌ `--font-weight-medium` | `500` | `—` | `—` | base.css, features/expander.css, modules/card.css |
| ❌ `--font-weight-normal` | `400` | `—` | `—` | base.css, features/expander.css, modules/button.c… |
| ❌ `--font-weight-semibold` | `600` | `—` | `—` | admin.css, base.css, chroma/base.css |
| ❌ `--fonts-default-override-ja` | `—` | `—` | `—` | font_i18n.css |
| ❌ `--fonts-default-override-ko` | `—` | `—` | `—` | font_i18n.css |
| ❌ `--fonts-default-override-zh-cn` | `—` | `—` | `—` | font_i18n.css |
| ❌ `--fonts-default-override-zh-hk` | `—` | `—` | `—` | font_i18n.css |
| ❌ `--fonts-default-override-zh-tw` | `—` | `—` | `—` | font_i18n.css |
| ❌ `--fonts-emoji` | `-emoji-fallback` | `—` | `—` | base.css, review.css |
| ⚠️ `--fonts-monospace` | `ui-monospace, SFMono-Regular,…` | `"JetBrains Mono", "Fira Code"…` | `"JetBrains Mono", "Fira Code"…` | base.css, codemirror/base.css, features/console.c… |
| ❌ `--fonts-override` | `—` | `—` | `"Source Serif 4", Georgia, se…` | base.css |
| ❌ `--fonts-proportional` | `-apple-system, "Segoe UI", sy…` | `—` | `—` | base.css, font_i18n.css |
| ⚠️ `--fonts-regular` | `var(--fonts-override, var(--f…` | `var(--fonts-override, var(--f…` | `—` | base.css, font_i18n.css, modules/button.css |
| ❌ `--gap-block` | `0.5rem` | `—` | `—` | base.css |
| ❌ `--gap-inline` | `0.25rem` | `—` | `—` | base.css, modules/label.css, repo.css |
| ❌ `--height-loading` | `16rem` | `—` | `—` | markup/content.css, modules/animations.css |
| ❌ `--line-height-default` | `normal` | `—` | `—` | admin.css, base.css, features/gitgraph.css |
| ❌ `--min-height-textarea` | `132px` | `—` | `—` | editor/combomarkdowneditor.css, form.css |
| ❌ `--octicon-chevron-right` | `url('data:image/svg+xml` | `—` | `—` | shared/settings.css |
| ❌ `--opacity-disabled` | `0.55` | `—` | `—` | form.css, modules/button.css, modules/input.css |
| ❌ `--page-margin-x` | `8px` | `—` | `—` | modules/container.css |
| ❌ `--page-space-bottom` | `64px` | `—` | `—` | base.css |
| ❌ `--page-spacing` | `16px` | `—` | `—` | base.css, modules/flexcontainer.css, repo/home.css |
| ❌ `--tab-size` | `4` | `—` | `—` | base.css |
| ❌ `--z-index-toast` | `1002` | `—` | `—` | modules/toast.css |
## Top selectors by var-reference count
- `modules/label.css` `/* based on Fomantic UI label module, with just the parts extracted th` — 6 vars: --border-radius, --color-label-bg, --color-label-text, --font-size-label, --font-weight-medium...
- `modules/button.css` `.ui.button` — 6 vars: --border-radius, --color-button, --color-light-border, --color-text, --font-weight-normal...
- `repo.css` `} .repository.file.editor .commit-form-wrapper .commit-form .quick-pul` — 4 vars: --border-radius, --color-secondary, --color-text, --fonts-monospace
- `codemirror/base.css` `.EasyMDEContainer .CodeMirror` — 4 vars: --color-input-background, --color-input-text, --color-secondary, --fonts-monospace
- `markup/content.css` `.markup kbd` — 4 vars: --border-radius, --color-markup-code-inline, --color-secondary, --color-text-light
- `repo/issue-card.css` `.issue-card` — 4 vars: --border-radius, --color-card, --color-secondary, --color-text
- `features/expander.css` `text-expander .suggestions, .tribute-container` — 4 vars: --border-radius, --color-menu, --color-secondary, --color-shadow
- `modules/menu.css` `.ui.menu` — 4 vars: --color-menu, --color-secondary, --font-weight-normal, --fonts-regular
- `modules/message.css` `/* based on Fomantic UI message module, with just the parts extracted ` — 4 vars: --border-radius, --color-box-body, --color-secondary, --color-text
- `modules/modal.css` `.ui.modal > .header` — 4 vars: --border-radius, --color-body, --color-secondary, --color-text-dark
- `modules/tippy.css` `.tippy-box` — 4 vars: --border-radius, --color-menu, --color-secondary, --color-text
- `modules/table.css` `.ui.table > thead > tr > th` — 4 vars: --color-box-header, --color-secondary, --color-text, --font-weight-normal
- `modules/table.css` `.ui.table > tfoot > tr > th, .ui.table > tfoot > tr > td` — 4 vars: --color-box-body, --color-secondary, --color-text, --font-weight-normal
- `base.css` `::file-selector-button` — 4 vars: --border-radius, --color-light, --color-light-border, --color-text-light
- `features/console.css` `/* Based on https://github.com/buildkite/terminal-to-html/blob/697ff23` — 4 vars: --border-radius, --color-console-bg, --color-console-fg, --fonts-monospace
- `modules/toast.css` `.toastify` — 4 vars: --border-radius, --color-shadow, --color-white, --z-index-toast
- `base.css` `body` — 4 vars: --color-body, --color-text, --fonts-regular, --tab-size
- `modules/input.css` `.ui.input > input` — 4 vars: --color-input-border, --color-input-text, --fonts-regular, --line-height-default
- `shared/flex-list.css` `.flex-item .flex-item-title` — 3 vars: --color-text, --font-weight-semibold, --gap-inline
- `user.css` `#readme_profile` — 3 vars: --border-radius, --color-card, --color-secondary
- `user.css` `#notification_table` — 3 vars: --border-radius, --color-box-body, --color-secondary
- `repo.css` `.repository.file.editor .commit-form-wrapper .commit-form` — 3 vars: --border-radius, --color-box-body, --color-secondary
- `repo.css` `.repository.view.issue .comment-list .comment .comment-container` — 3 vars: --border-radius, --color-box-body, --color-secondary
- `repo.css` `.comment-header` — 3 vars: --color-box-header, --color-secondary, --color-text
- `repo.css` `.resolved-placeholder` — 3 vars: --border-radius, --color-box-header, --color-secondary
---
## Per-file source CSS consumption summary
For each Gitea source CSS file, the `--color-*` variables it consumes
(top 10 by reference count). Use this to know which file to read when a
specific variable misbehaves.
### `actions.css` (11 refs across 6 unique vars)
- `--color-white` × 6
- `--color-green` × 1
- `--color-red-light` × 1
- `--color-blue` × 1
- `--color-yellow` × 1
- `--color-text-light` × 1
### `admin.css` (1 refs across 1 unique vars)
- `--color-secondary` × 1
### `base.css` (104 refs across 40 unique vars)
- `--color-text` × 14
- `--color-secondary` × 13
- `--color-text-light-2` × 10
- `--color-primary` × 7
- `--color-hover` × 6
- `--color-text-light` × 4
- `--color-body` × 3
- `--color-label-bg` × 2
- `--color-secondary-dark-1` × 2
- `--color-accent` × 2
### `codemirror/base.css` (9 refs across 8 unique vars)
- `--color-primary` × 2
- `--color-input-text` × 1
- `--color-input-background` × 1
- `--color-secondary` × 1
- `--color-caret` × 1
- `--color-primary-light-1` × 1
- `--color-white` × 1
- `--color-placeholder-text` × 1
### `dashboard.css` (1 refs across 1 unique vars)
- `--color-red` × 1
### `editor/combomarkdowneditor.css` (4 refs across 3 unique vars)
- `--color-secondary` × 2
- `--color-text` × 1
- `--color-primary` × 1
### `editor/fileeditor.css` (9 refs across 5 unique vars)
- `--color-secondary` × 3
- `--color-body` × 2
- `--color-text-light` × 2
- `--color-hover` × 1
- `--color-active` × 1
### `explore.css` (1 refs across 1 unique vars)
- `--color-text-light` × 1
### `features/codeeditor.css` (1 refs across 1 unique vars)
- `--color-secondary` × 1
### `features/console.css` (55 refs across 19 unique vars)
- `--color-ansi-bright-black` × 5
- `--color-ansi-red` × 4
- `--color-ansi-green` × 4
- `--color-ansi-bright-red` × 4
- `--color-ansi-bright-green` × 4
- `--color-ansi-black` × 3
- `--color-ansi-yellow` × 3
- `--color-ansi-blue` × 3
- `--color-ansi-magenta` × 3
- `--color-ansi-cyan` × 3
### `features/dropzone.css` (3 refs across 3 unique vars)
- `--color-secondary` × 1
- `--color-text-light` × 1
- `--color-text` × 1
### `features/expander.css` (7 refs across 6 unique vars)
- `--color-secondary` × 2
- `--color-menu` × 1
- `--color-shadow` × 1
- `--color-text-light-1` × 1
- `--color-primary` × 1
- `--color-primary-contrast` × 1
### `features/gitgraph.css` (4 refs across 3 unique vars)
- `--color-secondary-dark-5` × 2
- `--color-text-light` × 1
- `--color-text-light-3` × 1
### `features/heatmap.css` (1 refs across 1 unique vars)
- `--color-text` × 1
### `features/imagediff.css` (5 refs across 4 unique vars)
- `--color-secondary-dark-8` × 2
- `--color-primary-light-7` × 1
- `--color-red` × 1
- `--color-green` × 1
### `features/projects.css` (3 refs across 3 unique vars)
- `--color-project-column-bg` × 1
- `--color-secondary` × 1
- `--color-secondary-dark-4` × 1
### `form.css` (24 refs across 16 unique vars)
- `--color-input-text` × 4
- `--color-error-border` × 3
- `--color-input-background` × 3
- `--color-text` × 2
- `--color-error-bg` × 1
- `--color-error-text` × 1
- `--color-error-bg-hover` × 1
- `--color-error-bg-active` × 1
- `--color-primary-light-6` × 1
- `--color-primary-light-4` × 1
### `helpers.css` (4 refs across 4 unique vars)
- `--color-primary` × 1
- `--color-primary-active` × 1
- `--color-hover` × 1
- `--color-active` × 1
### `home.css` (5 refs across 4 unique vars)
- `--color-logo` × 2
- `--color-footer` × 1
- `--color-secondary` × 1
- `--color-secondary-dark-1` × 1
### `install.css` (2 refs across 2 unique vars)
- `--color-secondary` × 1
- `--color-red` × 1
### `markup/codecopy.css` (2 refs across 2 unique vars)
- `--color-secondary` × 1
- `--color-secondary-dark-1` × 1
### `markup/codepreview.css` (3 refs across 2 unique vars)
- `--color-secondary` × 2
- `--color-text-light-1` × 1
### `markup/content.css` (23 refs across 11 unique vars)
- `--color-secondary` × 10
- `--color-text-light-2` × 2
- `--color-text` × 2
- `--color-markup-code-inline` × 2
- `--color-red` × 1
- `--color-input-background` × 1
- `--color-primary` × 1
- `--color-markup-table-row` × 1
- `--color-box-body` × 1
- `--color-markup-code-block` × 1
### `modules/animations.css` (4 refs across 2 unique vars)
- `--color-secondary` × 2
- `--color-secondary-dark-8` × 2
### `modules/breadcrumb.css` (1 refs across 1 unique vars)
- `--color-text-light-2` × 1
### `modules/button.css` (61 refs across 19 unique vars)
- `--color-light-border` × 6
- `--color-text` × 5
- `--color-secondary-dark-2` × 4
- `--color-green` × 4
- `--color-white` × 4
- `--color-green-dark-1` × 4
- `--color-red` × 4
- `--color-hover` × 3
- `--color-primary` × 3
- `--color-primary-hover` × 3
### `modules/card.css` (13 refs across 6 unique vars)
- `--color-text` × 5
- `--color-secondary` × 3
- `--color-card` × 2
- `--color-primary` × 1
- `--color-text-light-2` × 1
- `--color-secondary-light-1` × 1
### `modules/checkbox.css` (5 refs across 5 unique vars)
- `--color-white` × 1
- `--color-shadow` × 1
- `--color-input-toggle-background` × 1
- `--color-text` × 1
- `--color-primary` × 1
### `modules/dimmer.css` (1 refs across 1 unique vars)
- `--color-overlay-backdrop` × 1
### `modules/divider.css` (4 refs across 2 unique vars)
- `--color-secondary` × 3
- `--color-text` × 1
### `modules/header.css` (12 refs across 10 unique vars)
- `--color-text` × 2
- `--color-secondary` × 2
- `--color-text-light-1` × 1
- `--color-box-header` × 1
- `--color-error-bg` × 1
- `--color-error-text` × 1
- `--color-error-border` × 1
- `--color-warning-bg` × 1
- `--color-warning-text` × 1
- `--color-warning-border` × 1
### `modules/input.css` (13 refs across 6 unique vars)
- `--color-input-border` × 3
- `--color-primary` × 3
- `--color-error-border` × 3
- `--color-input-text` × 2
- `--color-error-bg` × 1
- `--color-error-text` × 1
### `modules/label.css` (85 refs across 26 unique vars)
- `--color-white` × 12
- `--color-label-hover-bg` × 6
- `--color-label-bg` × 4
- `--color-label-text` × 4
- `--color-red` × 4
- `--color-red-dark-1` × 4
- `--color-orange` × 4
- `--color-orange-dark-1` × 4
- `--color-yellow` × 4
- `--color-yellow-dark-1` × 4
### `modules/list.css` (5 refs across 3 unique vars)
- `--color-text` × 2
- `--color-text-dark` × 2
- `--color-secondary` × 1
### `modules/menu.css` (49 refs across 13 unique vars)
- `--color-text` × 13
- `--color-secondary` × 9
- `--color-active` × 7
- `--color-hover` × 4
- `--color-menu` × 3
- `--color-text-light-2` × 3
- `--color-text-dark` × 3
- `--color-body` × 2
- `--color-box-header` × 1
- `--color-label-bg` × 1
### `modules/message.css` (19 refs across 19 unique vars)
- `--color-box-body` × 1
- `--color-text` × 1
- `--color-secondary` × 1
- `--color-blue` × 1
- `--color-info-bg` × 1
- `--color-info-text` × 1
- `--color-info-border` × 1
- `--color-green` × 1
- `--color-success-bg` × 1
- `--color-success-text` × 1
### `modules/modal.css` (10 refs across 5 unique vars)
- `--color-body` × 3
- `--color-text-dark` × 2
- `--color-secondary` × 2
- `--color-shadow` × 2
- `--color-secondary-bg` × 1
### `modules/navbar.css` (10 refs across 7 unique vars)
- `--color-nav-bg` × 3
- `--color-nav-hover-bg` × 2
- `--color-secondary` × 1
- `--color-nav-text` × 1
- `--color-active` × 1
- `--color-primary` × 1
- `--color-secondary-nav-bg` × 1
### `modules/segment.css` (17 refs across 7 unique vars)
- `--color-secondary` × 7
- `--color-box-body` × 3
- `--color-text` × 3
- `--color-secondary-bg` × 1
- `--color-text-light` × 1
- `--color-error-border` × 1
- `--color-warning-border` × 1
### `modules/table.css` (19 refs across 7 unique vars)
- `--color-secondary` × 7
- `--color-text` × 3
- `--color-box-body` × 2
- `--color-secondary-alpha-50` × 2
- `--color-hover` × 2
- `--color-light` × 2
- `--color-box-header` × 1
### `modules/tippy.css` (21 refs across 10 unique vars)
- `--color-menu` × 4
- `--color-shadow` × 4
- `--color-text` × 2
- `--color-secondary` × 2
- `--color-tooltip-bg` × 2
- `--color-hover` × 2
- `--color-box-body` × 2
- `--color-tooltip-text` × 1
- `--color-active` × 1
- `--color-box-header` × 1
### `modules/toast.css` (4 refs across 4 unique vars)
- `--color-white` × 1
- `--color-shadow` × 1
- `--color-hover` × 1
- `--color-active` × 1
### `org.css` (2 refs across 2 unique vars)
- `--color-secondary` × 1
- `--color-box-body` × 1
### `repo/clone.css` (2 refs across 2 unique vars)
- `--color-text-light-2` × 1
- `--color-text-dark` × 1
### `repo/commit-sign.css` (13 refs across 13 unique vars)
- `--color-light-border` × 1
- `--color-green-badge` × 1
- `--color-green-badge-bg` × 1
- `--color-green-badge-hover-bg` × 1
- `--color-yellow-badge` × 1
- `--color-yellow-badge-bg` × 1
- `--color-yellow-badge-hover-bg` × 1
- `--color-orange-badge` × 1
- `--color-orange-badge-bg` × 1
- `--color-orange-badge-hover-bg` × 1
### `repo/file-view.css` (6 refs across 5 unique vars)
- `--color-secondary` × 2
- `--color-highlight-bg` × 1
- `--color-highlight-fg` × 1
- `--color-code-bg` × 1
- `--color-text-dark` × 1
### `repo/home-file-list.css` (7 refs across 5 unique vars)
- `--color-secondary` × 2
- `--color-text-light-1` × 2
- `--color-box-body` × 1
- `--color-hover-opaque` × 1
- `--color-box-header` × 1
### `repo/home.css` (1 refs across 1 unique vars)
- `--color-secondary` × 1
### `repo/issue-card.css` (3 refs across 3 unique vars)
- `--color-secondary` × 1
- `--color-card` × 1
- `--color-text` × 1
### `repo/issue-label.css` (1 refs across 1 unique vars)
- `--color-secondary` × 1
### `repo/issue-list.css` (3 refs across 2 unique vars)
- `--color-secondary-dark-4` × 2
- `--color-secondary-alpha-50` × 1
### `repo/reactions.css` (2 refs across 2 unique vars)
- `--color-reaction-active-bg` × 1
- `--color-reaction-hover-bg` × 1
### `repo/release-tag.css` (2 refs across 2 unique vars)
- `--color-secondary` × 1
- `--color-text-light-1` × 1
### `repo/wiki.css` (3 refs across 2 unique vars)
- `--color-secondary` × 2
- `--color-hover` × 1
### `repo.css` (98 refs across 37 unique vars)
- `--color-secondary` × 18
- `--color-text` × 8
- `--color-red` × 5
- `--color-primary` × 5
- `--color-box-body` × 5
- `--color-box-header` × 5
- `--color-body` × 4
- `--color-yellow` × 3
- `--color-text-light-2` × 3
- `--color-timeline` × 3
### `review.css` (14 refs across 11 unique vars)
- `--color-primary-contrast` × 2
- `--color-accent` × 2
- `--color-small-accent` × 2
- `--color-red` × 1
- `--color-secondary` × 1
- `--color-text` × 1
- `--color-text-light` × 1
- `--color-expand-button` × 1
- `--color-primary` × 1
- `--color-primary-light-4` × 1
### `shared/flex-list.css` (5 refs across 4 unique vars)
- `--color-text` × 2
- `--color-primary` × 1
- `--color-text-light-2` × 1
- `--color-secondary` × 1
### `shared/milestone.css` (4 refs across 3 unique vars)
- `--color-text-light-2` × 2
- `--color-secondary` × 1
- `--color-text` × 1
### `shared/settings.css` (1 refs across 1 unique vars)
- `--color-body` × 1
### `user.css` (6 refs across 4 unique vars)
- `--color-secondary` × 3
- `--color-card` × 1
- `--color-box-body` × 1
- `--color-hover` × 1

96
infra/gitea/Dockerfile Normal file
View File

@@ -0,0 +1,96 @@
# syntax=docker/dockerfile:1.7
# =============================================================================
# Gitea with fhirworx theme compiled in.
#
# Clones go-gitea/gitea at the pinned tag, drops our theme CSS into
# web_src/css/themes/, then runs Gitea's own webpack+go build so the theme is
# baked into `bindata`. Runtime stage is the same alpine+dumb-init layout as
# the upstream rootless image.
# =============================================================================
ARG GITEA_VERSION=v1.25.4
# BUILD_TAG must be unique per build; defaults to current epoch when unset.
# Gitea uses main.Version to construct the `?v=...` query string on every
# asset URL. Without changing it, browsers cache theme-fhirworx.css for 6h
# (Cache-Control: max-age=21600) — so theme edits never reach the user
# until the URL key changes. Setting it per-build forces every cache layer
# (browser, CDN, Cloudflare) to refetch.
ARG BUILD_TAG
# ---- build stage ------------------------------------------------------------
FROM docker.io/library/golang:1.25-alpine3.22 AS build
ARG GITEA_VERSION
ARG BUILD_TAG
ENV GOPROXY=https://proxy.golang.org,direct \
GOSUMDB=sum.golang.org \
TAGS="bindata timetzdata sqlite sqlite_unlock_notify" \
CGO_ENABLED=1
RUN apk add --no-cache build-base git nodejs npm \
&& npm install -g pnpm@10
WORKDIR /src
RUN git clone --depth 1 --branch ${GITEA_VERSION} \
https://github.com/go-gitea/gitea.git . \
&& git log -1 --format='%H %s'
# Drop fhirworx theme into Gitea's theme dir before the frontend build so
# webpack/bindata pick it up.
COPY theme/theme-fhirworx.css web_src/css/themes/theme-fhirworx.css
COPY theme/theme-fhirworx-dark.css web_src/css/themes/theme-fhirworx-dark.css
# Replace Gitea's source SVGs with fhirworx branding. tools/generate-images.ts
# reads these to produce logo.svg/.png, favicon.svg/.png, apple-touch-icon.png,
# avatar_default.png — all baked into bindata.
COPY brand/logo.svg assets/logo.svg
COPY brand/favicon.svg assets/favicon.svg
# Full build. Order matters: deps → generate-images (uses our brand SVGs) →
# webpack frontend → go binary with bindata embedding everything in public/.
RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.local/share/pnpm/store \
make deps-frontend \
&& make generate-images \
&& BUILD_TAG="${BUILD_TAG:-$(date +%s)}" \
&& export LDFLAGS="-X 'main.Version=${GITEA_VERSION}-fhirworx.${BUILD_TAG}'" \
&& echo "Building with version: ${GITEA_VERSION}-fhirworx.${BUILD_TAG}" \
&& make clean-all build LDFLAGS="${LDFLAGS}" \
&& go build contrib/environment-to-ini/environment-to-ini.go
# Upstream rootless overlay files (entrypoint, setup, gitea wrapper).
RUN chmod 755 docker/rootless/usr/local/bin/docker-entrypoint.sh \
docker/rootless/usr/local/bin/docker-setup.sh \
docker/rootless/usr/local/bin/gitea \
/src/gitea \
/src/environment-to-ini
# ---- runtime stage ----------------------------------------------------------
FROM docker.io/library/alpine:3.22
LABEL org.opencontainers.image.source="https://github.com/go-gitea/gitea"
LABEL fhirworx.theme="fhirworx"
EXPOSE 2222 3000
RUN apk add --no-cache bash ca-certificates dumb-init gettext git curl gnupg openssh-keygen \
&& addgroup -S -g 1000 git \
&& adduser -S -H -D -h /var/lib/gitea/git -s /bin/bash -u 1000 -G git git \
&& mkdir -p /var/lib/gitea /etc/gitea \
&& chown git:git /var/lib/gitea /etc/gitea
COPY --from=build /src/docker/rootless /
COPY --from=build --chown=root:root /src/gitea /app/gitea/gitea
COPY --from=build --chown=root:root /src/environment-to-ini /usr/local/bin/environment-to-ini
USER 1000:1000
ENV GITEA_WORK_DIR=/var/lib/gitea \
GITEA_CUSTOM=/var/lib/gitea/custom \
GITEA_TEMP=/tmp/gitea \
TMPDIR=/tmp/gitea \
GITEA_APP_INI=/etc/gitea/app.ini \
HOME=/var/lib/gitea/git
VOLUME ["/var/lib/gitea", "/etc/gitea"]
WORKDIR /var/lib/gitea
ENTRYPOINT ["/usr/bin/dumb-init", "--", "/usr/local/bin/docker-entrypoint.sh"]
CMD []

132
infra/gitea/README.md Normal file
View File

@@ -0,0 +1,132 @@
# Fhirworx Gitea image
This directory builds `fhirworx/gitea:<tag>` — upstream `go-gitea/gitea` at a
pinned tag with the fhirworx theme baked in via `bindata`. **No source
changes to upstream**; we only inject one CSS file and two brand SVGs into
the upstream tree before its own build runs.
## Layout
```
infra/gitea/
├── Dockerfile # multi-stage build, see below
├── .dockerignore
├── theme/
│ └── theme-fhirworx.css # ONE css file = the entire theme
├── brand/
│ ├── logo.svg # → assets/logo.svg → make generate-images
│ └── favicon.svg # → assets/favicon.svg
├── custom/ # bind-mounted at runtime as /var/lib/gitea/custom
│ └── templates/
│ └── home.tmpl # custom anonymous landing page
└── README.md # this file
```
## Staying downstream from upstream Gitea
Goal: track upstream cleanly, never fork the source.
The build does only three things to the upstream tree:
1. `COPY theme/theme-fhirworx.css web_src/css/themes/theme-fhirworx.css`
2. `COPY brand/logo.svg assets/logo.svg`
3. `COPY brand/favicon.svg assets/favicon.svg`
After these, the upstream `make clean-all build` runs unmodified — webpack
processes our theme into `public/assets/css/theme-fhirworx.css`, the image
generator regenerates every PNG/SVG variant from our SVGs, and `bindata`
embeds the whole `public/` tree into the Go binary.
### Bumping Gitea
```
# 1. Update the pinned tag
sed -i 's/GITEA_VERSION=v1\.[0-9.]\+/GITEA_VERSION=v1.NEW.VER/' Dockerfile
# 2. Diff our theme against upstream's reference
diff theme/theme-fhirworx.css \
<(curl -sL https://raw.githubusercontent.com/go-gitea/gitea/v1.NEW.VER/web_src/css/themes/theme-gitea-light.css)
# 3. Add any new --color-* vars upstream introduced
# 4. Rebuild
docker compose build gitea && docker compose up -d gitea
```
The theme file's variable order **mirrors upstream's `theme-gitea-light.css`
1:1**, on purpose, so step 2 produces a clean readable diff. Add variables
upstream added; refresh values you've remapped.
## Theme contract
The theme is a complete drop-in replacement for upstream's
`theme-gitea-light.css`. It defines:
- **All ~140 `--color-*` variables** Gitea references. Any var left undefined
resolves to CSS `initial` (transparent bg / black text), which breaks
surfaces like the navbar, secondary-nav, footer, clone panel, menu hover
states. Historical bug: an early version defined ~30 vars and many
components broke.
- **`--fonts-override`** — Gitea's `base.css` composes
`--fonts-regular: var(--fonts-override, var(--fonts-proportional)), ...`.
Setting `--fonts-override` propagates the editorial type (Source Serif 4)
through every Fomantic UI component (menus, buttons, tabs, inputs).
- **A defensive `#navbar` color sweep**. Fhirworx is the only design
(compared against awesome-gitea's full theme list — Catppuccin, Rainnny
GitHub, lutinglt, Earl Grey, Dark Arc, etc.) that puts a *dark* navbar
over a *light* body. Every reference theme keeps both surfaces in the
same luminance class. Because of that, Gitea's base CSS doesn't anticipate
the inversion: any Fomantic class with `color: var(--color-text)` (e.g.
`.ui.button` for the hamburger `#navbar-expand-toggle`) leaks near-black
text into the dark navbar. The sweep at the end of theme-fhirworx.css
forces nav-text on every text/icon element inside `#navbar`, and flips
dropdown popouts back to the light body palette since they float over
the page, not the bar.
## Custom templates (still bind-mounted)
`custom/templates/home.tmpl` is the anonymous landing page. It overrides
upstream's stock dashboard for unauthenticated visitors and renders the
homelab service grid. It's bind-mounted via `compose.yml`, not baked in,
because it's content not theme — easier to edit without rebuilding.
## Compose wiring
```yaml
gitea:
build:
context: ./infra/gitea
args:
GITEA_VERSION: v1.25.4
image: fhirworx/gitea:v1.25.4
environment:
- GITEA__ui__THEMES=fhirworx
- GITEA__ui__DEFAULT_THEME=fhirworx
volumes:
- gitea_data:/var/lib/gitea
- gitea_config:/etc/gitea
- ./infra/gitea/custom:/var/lib/gitea/custom
```
`THEMES=fhirworx` (single option) — no gitea-auto/light/dark/protanopia
variants are exposed in the user appearance dropdown. fhirworx is the only
choice and it's the default. Per-user `theme` column in the postgres
`"user"` table should be set to `'fhirworx'`.
## What goes wrong if you bypass the build
Earlier iterations bind-mounted raw CSS files into stock
`gitea/gitea:1.25.4-rootless`. Two persistent failures:
- **Inode drift**: Docker's single-file bind mount tracks by inode. The
`Edit`/`Write` tools rewrite atomically, replacing the inode. The container
keeps pointing at the now-orphaned old inode and sees nothing change.
Required `docker restart` after every edit.
- **Cache + version-pinning**: The asset URL embeds Gitea's version
(`?v=1.25.4`). Browsers cache aggressively for 6h. Edits to the CSS file
don't change the URL → cache hit serves stale CSS forever (or until the
user knows to hard-refresh). On Cloudflare it's even longer.
Baking the theme into bindata sidesteps both. The CSS only changes when you
rebuild the image, which means the version string actually changes in the
binary's metadata, and there's no inode tracking to drift.

BIN
infra/gitea/brand/favicon-180.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

1
infra/gitea/brand/favicon.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 58 KiB

BIN
infra/gitea/brand/logo-512.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 74 KiB

1
infra/gitea/brand/logo.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 58 KiB

0
infra/gitea/custom/public/assets/css/theme-fhirworx.css
View File

12
infra/gitea/custom/templates/custom/header.tmpl Normal file
View File

@@ -0,0 +1,12 @@
{{/*
Fhirworx: declare the page color-scheme as light-only so client-side
dark-mode features (Chrome Auto Dark Mode for Web Contents, Firefox
`layout.css.prefers-color-scheme.content-override`, Safari Reader dark)
refuse to recolor the served palette.
This is in addition to `color-scheme: light only` in theme-fhirworx.css.
The <meta> form is honored by browsers BEFORE CSS evaluates, blocking
the auto-dark heuristic at the earliest possible point.
*/}}
<meta name="color-scheme" content="light dark">
<meta name="supported-color-schemes" content="light dark">

17
infra/gitea/custom/templates/home.tmpl
View File

@@ -33,6 +33,17 @@
gap: 0.5rem;
margin-bottom: 1rem;
}
.homelab-dashboard .masthead-brand {
display: flex;
align-items: center;
gap: 1rem;
}
.homelab-dashboard .masthead-logo {
width: clamp(48px, 6vw, 80px);
height: clamp(48px, 6vw, 80px);
display: block;
flex: 0 0 auto;
}
.homelab-dashboard .masthead-row h1 {
font-family: "Playfair Display", Georgia, serif !important;
font-size: clamp(2.5rem, 5vw, 3.5rem) !important;
@@ -41,6 +52,7 @@
letter-spacing: -0.02em !important;
line-height: 1.1 !important;
border: none !important;
margin: 0 !important;
}
.homelab-dashboard .masthead-eyebrow {
font-family: "JetBrains Mono", "Fira Code", monospace;
@@ -232,7 +244,10 @@
<div class="masthead">
<div class="rule-line"></div>
<div class="masthead-row">
<h1>Homelab</h1>
<div class="masthead-brand">
<img src="/assets/img/logo.svg" alt="fhirworx" class="masthead-logo">
<h1>Homelab</h1>
</div>
<span class="masthead-eyebrow">git.fhirworx.io</span>
</div>
<p class="tagline">

564
infra/gitea/theme/theme-fhirworx-dark.css Normal file
View File

@@ -0,0 +1,564 @@
/* =============================================================================
Fhirworx — Gitea theme (dark variant)
=============================================================================
Structural reference: Gitea v1.25.4 web_src/css/themes/theme-gitea-dark.css.
Variable order mirrors upstream line-for-line (clean-diff rule). Only
VALUES differ from upstream. Shares variable NAMES with theme-fhirworx.css.
========================================================================== */
@import url('https://fonts.googleapis.com/css2?family=Playfair+Display:wght@400;600;700;800&family=Source+Serif+4:ital,wght@0,300;0,400;0,600;1,400&family=JetBrains+Mono:wght@400;500;600&display=swap');
gitea-theme-meta-info {
--theme-display-name: "Fhirworx Dark";
}
:root {
/* --- Fhirworx dark palette (source of truth) ---------------------------- */
--fw-background: #0F1419; /* deep dark body */
--fw-foreground: #E8E5DD; /* warm off-white body text */
--fw-card: #1A2028; /* card/panel surface, lifted from body */
--fw-card-hover: #232A34;
--fw-primary: #5684A3; /* muted blue; lifted so it beats --card contrast */
--fw-primary-foreground: #0F1419;
--fw-secondary: #1F262E;
--fw-muted-foreground: #8A8A87;
--fw-border: #3A4452;
--fw-nav-bg: #0A0E13; /* even darker than body (visible separation) */
--fw-nav-text: #E8F0F8;
--fw-nav-hover-bg: #1C2B3A;
--fw-nav-active-bg: #25384C;
--fw-destructive: #E07A6E;
--fw-link: #5C9CC0;
--fw-chart-3: #B8A078;
--fw-chart-5: #E08A4A;
--fonts-override: "Source Serif 4", Georgia, serif;
--fonts-monospace: "JetBrains Mono", "Fira Code", ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace;
/* ======================================================================= */
/* BELOW: Gitea variables in upstream order (theme-gitea-dark.css). */
/* ======================================================================= */
--is-dark-theme: true;
--color-primary: var(--fw-primary);
--color-primary-contrast: var(--fw-primary-foreground);
--color-primary-dark-1: #6D97B3;
--color-primary-dark-2: #85ABC2;
--color-primary-dark-3: #9CBED1;
--color-primary-dark-4: #B3D0DF;
--color-primary-dark-5: #C8DDE8;
--color-primary-dark-6: #DEEAF1;
--color-primary-dark-7: #F1F6F9;
--color-primary-light-1: #4A7590;
--color-primary-light-2: #3E657D;
--color-primary-light-3: #33556A;
--color-primary-light-4: #294557;
--color-primary-light-5: #203644;
--color-primary-light-6: #162531;
--color-primary-light-7: #0D1620;
--color-primary-alpha-10: #5684A319;
--color-primary-alpha-20: #5684A333;
--color-primary-alpha-30: #5684A34B;
--color-primary-alpha-40: #5684A366;
--color-primary-alpha-50: #5684A380;
--color-primary-alpha-60: #5684A399;
--color-primary-alpha-70: #5684A3B3;
--color-primary-alpha-80: #5684A3CC;
--color-primary-alpha-90: #5684A3E1;
--color-primary-hover: var(--color-primary-dark-1);
--color-primary-active: var(--color-primary-dark-2);
--color-secondary: var(--fw-border);
--color-secondary-dark-1: #45505E;
--color-secondary-dark-2: #505C6A;
--color-secondary-dark-3: #5F6B7A;
--color-secondary-dark-4: #6E7A89;
--color-secondary-dark-5: #7E8A99;
--color-secondary-dark-6: #8D99A7;
--color-secondary-dark-7: #9CA7B4;
--color-secondary-dark-8: #ABB5C0;
--color-secondary-dark-9: #B7C0CA;
--color-secondary-dark-10: #C3CBD4;
--color-secondary-dark-11: #CFD6DE;
--color-secondary-dark-12: #DAE0E7;
--color-secondary-dark-13: #E3E8EE;
--color-secondary-light-1: #2F3845;
--color-secondary-light-2: #262D38;
--color-secondary-light-3: #1C222B;
--color-secondary-light-4: #161B22;
--color-secondary-alpha-10: #3A445219;
--color-secondary-alpha-20: #3A445233;
--color-secondary-alpha-30: #3A44524B;
--color-secondary-alpha-40: #3A445266;
--color-secondary-alpha-50: #3A445280;
--color-secondary-alpha-60: #3A445299;
--color-secondary-alpha-70: #3A4452B3;
--color-secondary-alpha-80: #3A4452CC;
--color-secondary-alpha-90: #3A4452E1;
--color-secondary-button: var(--color-secondary-dark-4);
--color-secondary-hover: var(--color-secondary-dark-3);
--color-secondary-active: var(--color-secondary-dark-2);
/* console */
--color-console-fg: #F7F8F9;
--color-console-fg-subtle: #BDC4CC;
--color-console-bg: #050910;
--color-console-border: #2A323A;
--color-console-hover-bg: #121820;
--color-console-active-bg: #1A2028;
--color-console-menu-bg: #161B22;
--color-console-menu-border: #2E353D;
/* named colors — lifted ~15% lightness from the light variant so each hue
reads on dark bg. */
--color-red: var(--fw-destructive);
--color-orange: var(--fw-chart-5);
--color-yellow: #E5BE5A;
--color-olive: #ABC075;
--color-green: #4CB28A;
--color-teal: #66C7C0;
--color-blue: var(--fw-link);
--color-violet: #9B7FDB;
--color-purple: #B478D4;
--color-pink: #D878B0;
--color-brown: var(--fw-chart-3);
--color-black: #0A0E13;
--color-red-light: #E89891;
--color-orange-light: #E8A576;
--color-yellow-light: #EDD084;
--color-olive-light: #BCCF95;
--color-green-light: #77C6A5;
--color-teal-light: #8BD4CE;
--color-blue-light: #7CB0CB;
--color-violet-light: #AE97E0;
--color-purple-light: #C494DD;
--color-pink-light: #DF96C0;
--color-brown-light: #C6B28E;
--color-black-light: #2C333C;
--color-red-dark-1: #C86559;
--color-orange-dark-1: #C87239;
--color-yellow-dark-1: #C9A347;
--color-olive-dark-1: #92A65F;
--color-green-dark-1: #3A9876;
--color-teal-dark-1: #4FA9A2;
--color-blue-dark-1: #487FA1;
--color-violet-dark-1: #7E65BD;
--color-purple-dark-1: #9860B7;
--color-pink-dark-1: #BA6097;
--color-brown-dark-1: #9A8260;
--color-black-dark-1: #05080B;
--color-red-dark-2: #A74D42;
--color-orange-dark-2: #A35824;
--color-yellow-dark-2: #A8862F;
--color-olive-dark-2: #788B47;
--color-green-dark-2: #2D7C5F;
--color-teal-dark-2: #3B8C86;
--color-blue-dark-2: #3A688A;
--color-violet-dark-2: #634E9E;
--color-purple-dark-2: #7C4E99;
--color-pink-dark-2: #9B4C7E;
--color-brown-dark-2: #7B684C;
--color-black-dark-2: #020406;
/* ansi */
--color-ansi-black: #1E2327;
--color-ansi-red: #CC4848;
--color-ansi-green: #87AB63;
--color-ansi-yellow: #CC9903;
--color-ansi-blue: #3A8AC6;
--color-ansi-magenta: #D22E8B;
--color-ansi-cyan: #00918A;
--color-ansi-white: var(--color-console-fg-subtle);
--color-ansi-bright-black: #424851;
--color-ansi-bright-red: #D15A5A;
--color-ansi-bright-green: #93B373;
--color-ansi-bright-yellow: #EAAF03;
--color-ansi-bright-blue: #4E96CC;
--color-ansi-bright-magenta: #D74397;
--color-ansi-bright-cyan: #00B6AD;
--color-ansi-bright-white: var(--color-console-fg);
/* other */
--color-grey: var(--fw-muted-foreground);
--color-grey-light: #98948C;
--color-gold: var(--fw-chart-3);
--color-white: #FFFFFF;
/* diff colors */
--color-diff-added-linenum-bg: #1E3C2A;
--color-diff-added-row-bg: #18301E;
--color-diff-added-row-border: #2A4A32;
--color-diff-added-word-bg: #2E5C3E;
--color-diff-moved-row-bg: #423E20;
--color-diff-moved-row-border: #6A6836;
--color-diff-removed-linenum-bg: #402020;
--color-diff-removed-row-bg: #2C1818;
--color-diff-removed-row-border: #5A3434;
--color-diff-removed-word-bg: #5C2D2D;
--color-diff-inactive: #1A1F26;
--color-error-border: #8A3B3B;
--color-error-bg: #3C1F1F;
--color-error-bg-active:#5C2E2E;
--color-error-bg-hover: #4A2525;
--color-error-text: #F0B0B0;
--color-success-border: #3F7D4F;
--color-success-bg: #1F3A2A;
--color-success-text: #7EC89A;
--color-warning-border: #8A7A30;
--color-warning-bg: #3A3220;
--color-warning-text: #E5BE5A;
--color-info-border: #3A6A8F;
--color-info-bg: #1A2E44;
--color-info-text: var(--color-blue);
--color-red-badge: var(--color-red);
--color-red-badge-bg: #E07A6E1A;
--color-red-badge-hover-bg: #E07A6E4D;
--color-green-badge: var(--color-green);
--color-green-badge-bg: #4CB28A1A;
--color-green-badge-hover-bg:#4CB28A4D;
--color-yellow-badge: var(--color-yellow);
--color-yellow-badge-bg: #E5BE5A1A;
--color-yellow-badge-hover-bg:#E5BE5A4D;
--color-orange-badge: var(--color-orange);
--color-orange-badge-bg: #E08A4A1A;
--color-orange-badge-hover-bg:#E08A4A4D;
--color-git: #F05133;
--color-logo: var(--fw-primary);
/* target-based */
--color-body: var(--fw-background);
--color-box-header: var(--fw-secondary);
--color-box-body: var(--fw-card);
--color-box-body-highlight: #1E2630;
--color-text-dark: #F5F2EA;
--color-text: var(--fw-foreground);
--color-text-light: #CFCCC5;
--color-text-light-1: #B4B0A8;
--color-text-light-2: #9A968E;
--color-text-light-3: #807C74;
--color-footer: var(--color-nav-bg);
--color-timeline: var(--fw-border);
--color-input-text: var(--color-text-dark);
--color-input-background: #12171E;
--color-input-toggle-background: #2A323D;
--color-input-border: var(--color-secondary);
--color-input-border-hover: var(--color-secondary-dark-1);
--color-light: #00001728;
--color-light-mimic-enabled: rgba(0, 0, 0, calc(40 / 255 * 222 / 255 / var(--opacity-disabled)));
--color-light-border: #E8F3FF28;
--color-hover: #E8F3FF14;
--color-hover-opaque: var(--fw-card-hover);
--color-active: #E8F3FF24;
/* Dropdown/menu panel bg — must visibly separate from body, even when
floating over nav. In dark we use --card (slightly lifted from body). */
--color-menu: var(--fw-card);
--color-card: var(--fw-card);
--color-markup-table-row: #E8F3FF0F;
--color-markup-code-block: #E8F3FF12;
--color-markup-code-inline:#E8F3FF20;
--color-button: var(--fw-card);
--color-code-bg: #0B0F14;
--color-shadow: #00001758;
--color-shadow-opaque: #000017;
--color-secondary-bg: var(--fw-secondary);
--color-expand-button: #2A3845;
--color-placeholder-text: var(--color-text-light-3);
--color-editor-line-highlight: var(--color-primary-light-5);
--color-project-column-bg: var(--color-secondary-light-2);
--color-caret: var(--color-text);
--color-reaction-bg: #E8F3FF12;
--color-reaction-hover-bg: var(--color-primary-light-4);
--color-reaction-active-bg: var(--color-primary-light-5);
--color-tooltip-text: #F5F2EA;
--color-tooltip-bg: #000B17F0;
/* NAVBAR — still darker than body for separation */
--color-nav-bg: var(--fw-nav-bg);
--color-nav-hover-bg: var(--fw-nav-hover-bg);
--color-nav-text: var(--fw-nav-text);
--color-secondary-nav-bg: var(--color-secondary-light-3);
--color-label-text: var(--fw-foreground);
--color-label-bg: #6A7A8E4B;
--color-label-hover-bg: #6A7A8EA0;
--color-label-active-bg: #6A7A8EFF;
--color-accent: var(--color-primary-dark-1);
--color-small-accent: var(--color-primary-light-4);
--color-highlight-fg: #C9A347;
--color-highlight-bg: #3A2E1A;
--color-overlay-backdrop: #000B17D0;
/* Convenience aliases (fhirworx custom templates) */
--color-warning: var(--color-yellow);
--color-success: var(--color-green);
--color-info: var(--color-blue);
--color-error: var(--color-red);
--color-primary-foreground: var(--fw-primary-foreground);
accent-color: var(--color-accent);
color-scheme: dark only;
}
/* ============================================================================
Component polish — same structural rules as light variant.
============================================================================ */
h1, h2, h3, h4, h5, h6 {
font-family: "Playfair Display", Georgia, serif;
letter-spacing: -0.01em;
}
#navbar {
border-bottom: 3px solid var(--fw-border);
}
/* ---------- #navbar sweep ---------------------------------------------------
Same logic as light variant: Fomantic classes leak default text color into
nav. In dark theme the leak is from --color-text (warm off-white) onto
--color-nav-bg (near-black) — that's actually readable, but we still force
consistent nav-text color because some Fomantic state rules flip to white/
black directly, not through variables. */
#navbar,
#navbar a,
#navbar button,
#navbar .item,
#navbar .ui.button,
#navbar .ui.label,
#navbar .ui.menu,
#navbar .ui.menu .item,
#navbar .ui.dropdown,
#navbar .ui.dropdown > .text,
#navbar .ui.input,
#navbar #navbar-expand-toggle,
#navbar .navbar-left > *,
#navbar .navbar-right > *,
#navbar .navbar-mobile-right > * {
color: var(--color-nav-text);
}
#navbar svg,
#navbar svg path,
#navbar svg circle,
#navbar svg rect,
#navbar svg polygon {
fill: currentcolor;
color: currentcolor;
}
#navbar a:hover,
#navbar button:hover,
#navbar .ui.button:hover,
#navbar .item:hover,
#navbar .item.active,
#navbar #navbar-expand-toggle:hover {
background: var(--color-nav-hover-bg);
color: var(--color-nav-text);
}
#navbar .ui.button,
#navbar .item.button {
background: transparent;
}
/* Navbar dropdown popouts: panels float over body (--fw-background); use
card surface so they visually detach from the nav. */
#navbar .ui.dropdown > .menu {
background: var(--color-menu);
color: var(--color-text);
border: 1px solid var(--color-secondary);
}
#navbar .ui.dropdown > .menu .item,
#navbar .ui.dropdown > .menu a.item,
#navbar .ui.dropdown > .menu .header.item {
color: var(--color-text);
background: transparent;
}
#navbar .ui.dropdown > .menu .item:hover,
#navbar .ui.dropdown > .menu .item.selected,
#navbar .ui.dropdown > .menu .item.active {
background: var(--color-hover);
color: var(--color-text);
}
#navbar .ui.dropdown > .menu .item svg,
#navbar .ui.dropdown > .menu .item .svg {
color: var(--color-text);
fill: currentcolor;
}
#navbar .ui.dropdown > .menu .divider {
border-color: var(--color-secondary);
}
/* Notification badge: same override as light — upstream sets
color:var(--color-nav-bg), which here is near-black = same as badge bg. */
#navbar a.item .notification_count,
#navbar a.item .header-stopwatch-dot {
color: var(--fw-nav-bg);
background: var(--fw-chart-5);
border-color: var(--fw-nav-bg);
}
#navbar a.item:hover .notification_count,
#navbar a.item:hover .header-stopwatch-dot {
border-color: var(--fw-nav-hover-bg);
}
#navbar input,
#navbar .ui.input input {
background: var(--fw-nav-hover-bg);
color: var(--fw-nav-text);
border-color: var(--fw-nav-active-bg);
}
#navbar input::placeholder,
#navbar .ui.input input::placeholder {
color: var(--color-text-light-3);
}
/* Footer + monospace */
.page-footer {
border-top: 3px solid var(--fw-border);
font-family: var(--fonts-monospace);
font-size: 12px;
}
.commit-sha, .sha, [class*="sha"],
.branch-name, .tag-name, .file-name, code.ref {
font-family: var(--fonts-monospace);
}
/* Yellow/olive/orange labels & buttons: on light-hue fills the default white
fg is still okay on dark-theme yellow/orange when the hue is lifted, but
to stay consistent across variants we pin to foreground color in both. */
.ui.ui.ui.yellow.label,
.ui.ui.ui.olive.label,
.ui.ui.ui.orange.label,
.ui.yellow.button,
.ui.olive.button,
.ui.orange.button,
.ui.yellow.button:hover,
.ui.olive.button:hover,
.ui.orange.button:hover {
color: #0F1419; /* dark text on light-hue fills; explicit for WCAG AA */
}
/* Basic-green label on dark bg — use the bright green directly, it has AA */
.ui.basic.green.label,
.ui.basic.green.labels .label {
color: var(--color-green);
border-color: var(--color-green);
}
a.ui.ui.ui.grey.label:hover,
a.ui.ui.ui.grey.label:focus {
color: var(--color-label-text);
background: var(--color-label-hover-bg);
}
.review-comments-counter {
background-color: var(--color-primary);
color: var(--color-primary-contrast);
}
.ui.card > .extra,
.ui.cards .card > .extra {
border-top-color: var(--color-secondary);
}
.text.small,
.flex-item-body .text.small {
font-family: var(--fonts-proportional), sans-serif;
}
/* `.text.COLOR` helpers — on dark bg, use the *light* variants so hue stays
saturated and contrast is on the right side of 4.5:1. base.css uses
!important, so we must match it. */
.text.orange { color: var(--color-orange-light) !important; /* AA: base.css uses !important */ }
.text.yellow { color: var(--color-yellow-light) !important; /* AA: base.css uses !important */ }
.text.olive { color: var(--color-olive-light) !important; /* AA: base.css uses !important */ }
.text.teal { color: var(--color-teal-light) !important; /* AA: base.css uses !important */ }
.text.brown { color: var(--color-brown-light) !important; /* AA: base.css uses !important */ }
.text.pink { color: var(--color-pink-light) !important; /* AA: base.css uses !important */ }
.text.gold { color: var(--color-brown-light) !important; /* AA: base.css uses !important */ }
.text.green { color: var(--color-green) !important; /* AA: base.css uses !important */ }
.text.red { color: var(--color-red) !important; /* AA: base.css uses !important */ }
.text.blue { color: var(--color-blue) !important; /* AA: base.css uses !important */ }
.ui.positive.message, .ui.success.message { color: var(--color-success-text); }
.ui.negative.message, .ui.error.message { color: var(--color-error-text); }
.ui.warning.message { color: var(--color-warning-text); }
.ui.info.message { color: var(--color-info-text); }
.secondary-nav {
background: var(--color-secondary-nav-bg) !important; /* beats .ui.secondary.menu !important */
border-bottom: 1px solid var(--color-secondary);
}
/* Link color: upstream `a { color: var(--color-primary) }`; primary in dark
is the muted blue which IS our link color already, so this is mostly a
no-op — kept for consistency with light variant. Exclude UI atoms. */
a:not(.item):not(.button):not(.label):not(.tab) {
color: var(--color-blue);
}
a:not(.item):not(.button):not(.label):not(.tab):hover {
color: var(--color-blue-light);
}
a.muted, a.suppressed, a.silenced, .muted-links a {
color: inherit;
}
.repository .diff-detail-box .diff-detail-stats strong:nth-of-type(1) {
color: var(--color-yellow);
}
/* ============================================================================
prefers-color-scheme:light defense — for users who set UA to light and
somehow land on this theme; re-assert our dark palette.
============================================================================ */
@media (prefers-color-scheme: light) {
:root {
color-scheme: dark only !important;
--color-body: var(--fw-background) !important;
--color-text: var(--fw-foreground) !important;
--color-nav-bg: var(--fw-nav-bg) !important;
--color-nav-text: var(--fw-nav-text) !important;
--color-box-body: var(--fw-card) !important;
--color-card: var(--fw-card) !important;
--color-menu: var(--fw-card) !important;
}
body {
background: var(--fw-background) !important; /* beats UA light override */
color: var(--fw-foreground) !important; /* beats UA light override */
}
}
/* invert emojis that upstream identifies as hard-to-read on dark bg */
.emoji[aria-label="check mark"],
.emoji[aria-label="currency exchange"],
.emoji[aria-label="TOP arrow"],
.emoji[aria-label="END arrow"],
.emoji[aria-label="ON! arrow"],
.emoji[aria-label="SOON arrow"],
.emoji[aria-label="heavy dollar sign"],
.emoji[aria-label="copyright"],
.emoji[aria-label="registered"],
.emoji[aria-label="trade mark"],
.emoji[aria-label="multiply"],
.emoji[aria-label="plus"],
.emoji[aria-label="minus"],
.emoji[aria-label="divide"],
.emoji[aria-label="curly loop"],
.emoji[aria-label="double curly loop"],
.emoji[aria-label="wavy dash"],
.emoji[aria-label="paw prints"],
.emoji[aria-label="musical note"],
.emoji[aria-label="musical notes"] {
filter: invert(100%) hue-rotate(180deg);
}

606
infra/gitea/theme/theme-fhirworx.css Normal file
View File

@@ -0,0 +1,606 @@
/* =============================================================================
Fhirworx — Gitea theme (light variant)
=============================================================================
Structural reference: Gitea v1.25.4 web_src/css/themes/theme-gitea-light.css.
Variable ORDER in :root below mirrors upstream line-for-line so future Gitea
version bumps produce clean 3-way diffs (git merge). Only the VALUES change.
Component overrides (below :root) target selectors that can't be reshaped by
variable values alone (e.g. #navbar dark-surface sweep).
========================================================================== */
@import url('https://fonts.googleapis.com/css2?family=Playfair+Display:wght@400;600;700;800&family=Source+Serif+4:ital,wght@0,300;0,400;0,600;1,400&family=JetBrains+Mono:wght@400;500;600&display=swap');
gitea-theme-meta-info {
--theme-display-name: "Fhirworx";
}
:root {
/* --- Fhirworx source-of-truth palette ----------------------------------- */
--fw-background: #F7F5F0; /* cream body */
--fw-foreground: #1A1A18; /* near-black body text */
--fw-card: #FFFFFF; /* pure-white surfaces (separates from body) */
--fw-card-hover: #F3F0EA;
--fw-primary: #1C2B3A; /* dark navy; nav + primary button */
--fw-primary-foreground: #F7F5F0;
--fw-secondary: #EDEBE6; /* warm beige subtle bg */
--fw-muted-foreground: #6B6B68;
--fw-border: #B8B3A4; /* warm beige-darker (visible on cream) */
--fw-nav-bg: #1C2B3A;
--fw-nav-text: #F0F4F8;
--fw-nav-hover-bg: #253748;
--fw-nav-active-bg: #2E4359;
--fw-destructive: #C0392B;
--fw-link: #2D5F7F; /* editorial blue link */
--fw-chart-3: #A88B5C;
--fw-chart-5: #C8702A;
/* Font pipeline: Gitea composes
--fonts-regular: var(--fonts-override, var(--fonts-proportional)), ...
in base.css:59. Setting --fonts-override here propagates to every .ui.*
component, tab, button, dropdown, menu — single hook, whole UI. */
--fonts-override: "Source Serif 4", Georgia, serif;
--fonts-monospace: "JetBrains Mono", "Fira Code", ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace;
/* ======================================================================= */
/* BELOW: Gitea variables in upstream order (theme-gitea-light.css). */
/* Any variable present upstream MUST be present here (clean-diff rule). */
/* ======================================================================= */
--is-dark-theme: false;
--color-primary: var(--fw-primary);
--color-primary-contrast: var(--fw-primary-foreground);
--color-primary-dark-1: #172530;
--color-primary-dark-2: #121F28;
--color-primary-dark-3: #0E1920;
--color-primary-dark-4: #0A1418;
--color-primary-dark-5: #060D11;
--color-primary-dark-6: #030609;
--color-primary-dark-7: #010203;
--color-primary-light-1: #2E4A63;
--color-primary-light-2: #3F6683;
--color-primary-light-3: #5684A3;
--color-primary-light-4: #7BA0BC;
--color-primary-light-5: #A5BCCD;
--color-primary-light-6: #D2DDE6;
--color-primary-light-7: #EEF2F6;
--color-primary-alpha-10: #1C2B3A19;
--color-primary-alpha-20: #1C2B3A33;
--color-primary-alpha-30: #1C2B3A4B;
--color-primary-alpha-40: #1C2B3A66;
--color-primary-alpha-50: #1C2B3A80;
--color-primary-alpha-60: #1C2B3A99;
--color-primary-alpha-70: #1C2B3AB3;
--color-primary-alpha-80: #1C2B3ACC;
--color-primary-alpha-90: #1C2B3AE1;
--color-primary-hover: var(--color-primary-light-1);
--color-primary-active: var(--color-primary-dark-1);
/* Secondary drives nearly every 1px border in Gitea (segments, tables,
menus, inputs, dropdowns). Must be distinct from body. */
--color-secondary: var(--fw-border);
--color-secondary-dark-1: #A49F90;
--color-secondary-dark-2: #908B7D;
--color-secondary-dark-3: #7D7869;
--color-secondary-dark-4: #6A6558;
--color-secondary-dark-5: #5A554A;
--color-secondary-dark-6: #49453C;
--color-secondary-dark-7: #3A372F;
--color-secondary-dark-8: #2C2924;
--color-secondary-dark-9: #221F1B;
--color-secondary-dark-10: #181613;
--color-secondary-dark-11: #0F0E0C;
--color-secondary-dark-12: #050504;
--color-secondary-dark-13: #010100;
--color-secondary-light-1: #C7C3B6;
--color-secondary-light-2: #D4D0C4;
--color-secondary-light-3: #E0DCD1;
--color-secondary-light-4: #ECE9E1;
--color-secondary-alpha-10: #B8B3A419;
--color-secondary-alpha-20: #B8B3A433;
--color-secondary-alpha-30: #B8B3A44B;
--color-secondary-alpha-40: #B8B3A466;
--color-secondary-alpha-50: #B8B3A480;
--color-secondary-alpha-60: #B8B3A499;
--color-secondary-alpha-70: #B8B3A4B3;
--color-secondary-alpha-80: #B8B3A4CC;
--color-secondary-alpha-90: #B8B3A4E1;
--color-secondary-button: var(--color-secondary-dark-4);
--color-secondary-hover: var(--color-secondary-dark-5);
--color-secondary-active: var(--color-secondary-dark-6);
/* console colors */
--color-console-fg: #F7F5F0;
--color-console-fg-subtle: #BDC4CC;
--color-console-bg: #0F1419;
--color-console-border: #2A323A;
--color-console-hover-bg: #1E2730;
--color-console-active-bg: #2A323A;
--color-console-menu-bg: #1A2028;
--color-console-menu-border: #3A4552;
/* named colors (darkened where upstream hue fails AA on cream) */
--color-red: var(--fw-destructive);
--color-orange: var(--fw-chart-5);
--color-yellow: #D4A017;
--color-olive: #8B9A4C;
--color-green: #247058; /* 5.45:1 on cream (upstream #21ba45 = 2.9:1) */
--color-teal: #3A9A94;
--color-blue: var(--fw-link);
--color-violet: #6B52A3;
--color-purple: #8A4BA8;
--color-pink: #B85890;
--color-brown: var(--fw-chart-3);
--color-black: var(--fw-foreground);
--color-red-light: #D66A5C;
--color-orange-light: #DB8A4C;
--color-yellow-light: #E5C04A;
--color-olive-light: #B3C45C;
--color-green-light: #4EA988;
--color-teal-light: #5FB5AF;
--color-blue-light: #4E82A3;
--color-violet-light: #8974B8;
--color-purple-light: #A26BC0;
--color-pink-light: #CE7AAA;
--color-brown-light: #C0A280;
--color-black-light: #4A4A47;
--color-red-dark-1: #A93024;
--color-orange-dark-1: #B4621F;
--color-yellow-dark-1: #B38712;
--color-olive-dark-1: #6E7A38;
--color-green-dark-1: #1E5E49;
--color-teal-dark-1: #2C7E78;
--color-blue-dark-1: #244B63;
--color-violet-dark-1: #55418A;
--color-purple-dark-1: #6E388A;
--color-pink-dark-1: #963F70;
--color-brown-dark-1: #876F49;
--color-black-dark-1: #26252B;
--color-red-dark-2: #8D2720;
--color-orange-dark-2: #9A5018;
--color-yellow-dark-2: #8F6C0E;
--color-olive-dark-2: #54602A;
--color-green-dark-2: #17483A;
--color-teal-dark-2: #225F5B;
--color-blue-dark-2: #1C3A4E;
--color-violet-dark-2: #42336C;
--color-purple-dark-2: #562B6C;
--color-pink-dark-2: #723057;
--color-brown-dark-2: #6B583A;
--color-black-dark-2: #18181C;
/* ansi (terminal) — leave upstream values, these render on dark console bg */
--color-ansi-black: #1E2327;
--color-ansi-red: #CC4848;
--color-ansi-green: #87AB63;
--color-ansi-yellow: #CC9903;
--color-ansi-blue: #3A8AC6;
--color-ansi-magenta: #D22E8B;
--color-ansi-cyan: #00918A;
--color-ansi-white: var(--color-console-fg-subtle);
--color-ansi-bright-black: #46494D;
--color-ansi-bright-red: #D15A5A;
--color-ansi-bright-green: #93B373;
--color-ansi-bright-yellow: #EAAF03;
--color-ansi-bright-blue: #4E96CC;
--color-ansi-bright-magenta: #D74397;
--color-ansi-bright-cyan: #00B6AD;
--color-ansi-bright-white: var(--color-console-fg);
/* other */
--color-grey: var(--fw-muted-foreground);
--color-grey-light: #8A8A87;
--color-gold: var(--fw-chart-3);
--color-white: #FFFFFF;
/* diff colors — softened for cream body */
--color-diff-added-linenum-bg: #CDEED6;
--color-diff-added-row-bg: #E2F5E8;
--color-diff-added-row-border: #C2E0CB;
--color-diff-added-word-bg: #A8E0B7;
--color-diff-moved-row-bg: #EEF4D8;
--color-diff-moved-row-border: #D4DE93;
--color-diff-removed-linenum-bg: #F5C8C5;
--color-diff-removed-row-bg: #FAE8E8;
--color-diff-removed-row-border: #EBBFBF;
--color-diff-removed-word-bg: #F3B5BC;
--color-diff-inactive: #EEECE7;
--color-error-border: #D48B8B;
--color-error-bg: #F6E5E4;
--color-error-bg-active:#E8ADAD;
--color-error-bg-hover: #F0C8C8;
--color-error-text: #8E2E2E;
--color-success-border: #A3C293;
--color-success-bg: #E8F2DD;
--color-success-text: #264F26;
--color-warning-border: #C9BA9B;
--color-warning-bg: #F8F0DC;
--color-warning-text: #5A3D10;
--color-info-border: #A5BCCD;
--color-info-bg: #E8F0F6;
--color-info-text: var(--color-blue); /* distinct from primary navy */
--color-red-badge: var(--color-red);
--color-red-badge-bg: #C0392B1A;
--color-red-badge-hover-bg: #C0392B4D;
--color-green-badge: var(--color-green);
--color-green-badge-bg: #2470581A;
--color-green-badge-hover-bg:#2470584D;
--color-yellow-badge: var(--color-yellow);
--color-yellow-badge-bg: #D4A0171A;
--color-yellow-badge-hover-bg:#D4A0174D;
--color-orange-badge: var(--color-orange);
--color-orange-badge-bg: #C8702A1A;
--color-orange-badge-hover-bg:#C8702A4D;
--color-git: #F05133;
--color-logo: var(--fw-primary);
/* target-based colors */
--color-body: var(--fw-background);
--color-box-header: var(--fw-secondary);
--color-box-body: var(--fw-card);
--color-box-body-highlight: var(--color-primary-light-7);
--color-text-dark: #0D0D0C;
--color-text: var(--fw-foreground);
--color-text-light: #2E2E2C;
--color-text-light-1: #454542;
--color-text-light-2: #595955;
--color-text-light-3: #6E6E6A;
--color-footer: var(--color-nav-bg);
--color-timeline: var(--fw-border);
--color-input-text: var(--color-text-dark);
--color-input-background: var(--fw-card);
--color-input-toggle-background: var(--fw-border);
--color-input-border: var(--color-secondary);
--color-input-border-hover: var(--color-secondary-dark-2);
/* Striped-table "--color-light": upstream is ~3% black on white; on cream
that's ~1.5% darker = invisible stripe. Opaque light-tan instead. */
--color-light: #EFECE5;
--color-light-mimic-enabled: rgba(0, 0, 0, calc(16 / 255 * 222 / 255 / var(--opacity-disabled)));
--color-light-border: #00001728;
--color-hover: #00001710;
--color-hover-opaque: var(--fw-card-hover);
--color-active: #0000171F;
/* Dropdown/menu panel bg: upstream uses #f8f9fb (off-white). We promote to
pure white so panels over the cream body have a real surface break.
This is THE variable the user's recent "transparent dropdown" bug hit —
if it resolves to `initial`/transparent the panel disappears. */
--color-menu: var(--fw-card);
--color-card: var(--fw-card);
--color-markup-table-row: #00306008;
--color-markup-code-block: #00306010;
--color-markup-code-inline:#00306014;
--color-button: var(--fw-card);
--color-code-bg: var(--fw-card);
--color-shadow: #00001726;
--color-shadow-opaque: var(--fw-border);
--color-secondary-bg: var(--fw-secondary);
--color-expand-button: var(--color-primary-light-6);
--color-placeholder-text: var(--color-text-light-3);
--color-editor-line-highlight: var(--color-primary-light-7);
--color-project-column-bg: var(--color-secondary-light-4);
--color-caret: var(--color-text-dark);
--color-reaction-bg: #0000170A;
--color-reaction-hover-bg: var(--color-primary-light-6);
--color-reaction-active-bg: var(--color-primary-light-5);
--color-tooltip-text: var(--fw-primary-foreground);
--color-tooltip-bg: #111B25F0;
/* NAVBAR — fhirworx signature: dark navy bar over cream body. */
--color-nav-bg: var(--fw-nav-bg);
--color-nav-hover-bg: var(--fw-nav-hover-bg);
--color-nav-text: var(--fw-nav-text);
--color-secondary-nav-bg: var(--color-secondary-light-4);
--color-label-text: var(--fw-foreground);
--color-label-bg: #94908648;
--color-label-hover-bg: #9490869E;
--color-label-active-bg: #949086E6;
--color-accent: var(--fw-primary);
--color-small-accent: var(--color-primary-light-6);
--color-highlight-fg: var(--fw-chart-3);
--color-highlight-bg: #FFF3C4;
--color-overlay-backdrop: #000017C0;
/* Convenience aliases consumed by fhirworx custom templates. */
--color-warning: #7F5E08; /* AA both as bg and fg on cream (5.5:1) */
--color-success: var(--color-green);
--color-info: var(--color-blue);
--color-error: var(--color-red);
--color-primary-foreground: var(--fw-primary-foreground);
accent-color: var(--color-accent);
/* `light only` opts out of Chrome auto-dark, Firefox content-override, and
Safari Reader dark. `only` keyword is critical — plain `light` allows UA
overrides; `light only` forbids them. */
color-scheme: light only;
}
/* ============================================================================
Component polish — rules the variable system cannot express.
============================================================================ */
h1, h2, h3, h4, h5, h6 {
font-family: "Playfair Display", Georgia, serif;
letter-spacing: -0.01em;
}
/* Dark navbar accent line (Gitea nav id is #navbar, not .navbar). */
#navbar {
border-bottom: 3px solid var(--fw-foreground);
}
/* ---------- #navbar dark-surface sweep --------------------------------------
Fhirworx inverts the nav: DARK bar over LIGHT body. Gitea base CSS assumes
nav luminance ≈ body luminance, so any Fomantic class with its own color
rule (.ui.button, .ui.label, .ui.input, .ui.dropdown, .ui.menu .item) leaks
near-black text into the dark bar — invisible. Sweep forces nav-text color
on every interactive descendant. `#navbar` id-specificity (1-1-0) beats
Fomantic's single-class rules (0-1-0) without !important. */
#navbar,
#navbar a,
#navbar button,
#navbar .item,
#navbar .ui.button,
#navbar .ui.label,
#navbar .ui.menu,
#navbar .ui.menu .item,
#navbar .ui.dropdown,
#navbar .ui.dropdown > .text,
#navbar .ui.input,
#navbar #navbar-expand-toggle,
#navbar .navbar-left > *,
#navbar .navbar-right > *,
#navbar .navbar-mobile-right > * {
color: var(--color-nav-text);
}
/* SVG icons in nav: follow currentcolor so the sweep above reaches them. */
#navbar svg,
#navbar svg path,
#navbar svg circle,
#navbar svg rect,
#navbar svg polygon {
fill: currentcolor;
color: currentcolor;
}
/* Hover/active state across all interactive elements in nav */
#navbar a:hover,
#navbar button:hover,
#navbar .ui.button:hover,
#navbar .item:hover,
#navbar .item.active,
#navbar #navbar-expand-toggle:hover {
background: var(--color-nav-hover-bg);
color: var(--fw-primary-foreground);
}
/* Fomantic backgrounds inside #navbar must clear so nav bg shows through. */
#navbar .ui.button,
#navbar .item.button {
background: transparent;
}
/* Dropdown popouts FROM the navbar are floating panels OVER the body; they
should NOT inherit the dark nav palette. Flip to body (cream surface, dark
text). This is the main acute pain point — without these 4 rules the menu
panels render with transparent bg + near-black text = invisible. */
#navbar .ui.dropdown > .menu {
background: var(--color-menu);
color: var(--color-text);
border: 1px solid var(--color-secondary);
}
#navbar .ui.dropdown > .menu .item,
#navbar .ui.dropdown > .menu a.item,
#navbar .ui.dropdown > .menu .header.item {
color: var(--color-text);
background: transparent;
}
#navbar .ui.dropdown > .menu .item:hover,
#navbar .ui.dropdown > .menu .item.selected,
#navbar .ui.dropdown > .menu .item.active {
background: var(--color-hover);
color: var(--color-text);
}
#navbar .ui.dropdown > .menu .item svg,
#navbar .ui.dropdown > .menu .item .svg {
color: var(--color-text);
fill: currentcolor;
}
#navbar .ui.dropdown > .menu .divider {
border-color: var(--color-secondary);
}
/* Notification badge: Gitea sets color:var(--color-nav-bg) on the count span
(navbar.css:111), assuming nav-bg is light enough to read on the primary-
colored badge bg. In fhirworx nav-bg == primary — same color = invisible.
Force badge text to primary-contrast. */
#navbar a.item .notification_count,
#navbar a.item .header-stopwatch-dot {
color: var(--fw-primary-foreground);
background: var(--fw-chart-5); /* orange badge, visible on navy */
border-color: var(--fw-nav-bg);
}
#navbar a.item:hover .notification_count,
#navbar a.item:hover .header-stopwatch-dot {
border-color: var(--fw-nav-hover-bg);
}
/* Navbar search input on dark bg: use a slightly-lifted navy so the input is
visible against the bar, with near-white placeholder. */
#navbar input,
#navbar .ui.input input {
background: var(--fw-nav-hover-bg);
color: var(--fw-nav-text);
border-color: var(--fw-nav-active-bg);
}
#navbar input::placeholder,
#navbar .ui.input input::placeholder {
color: var(--color-console-fg-subtle);
}
/* Footer — same dark-surface inversion as navbar. Footer bg defaults to
var(--color-nav-bg) (= dark navy in fhirworx); without overriding, every
text descendant inherits var(--color-text) (= near-black) → contrast 1.2:1.
Force every text/icon descendant to nav-text, same as the #navbar sweep. */
.page-footer {
border-top: 3px solid var(--fw-foreground);
font-family: var(--fonts-monospace);
font-size: 12px;
color: var(--color-nav-text);
}
.page-footer,
.page-footer a,
.page-footer span,
.page-footer strong,
.page-footer .left-links,
.page-footer .right-links,
.page-footer .flex-text-inline,
.page-footer .ui.dropdown,
.page-footer .item {
color: var(--color-nav-text);
}
.page-footer a:hover {
color: var(--fw-primary-foreground);
}
.page-footer svg,
.page-footer svg path {
fill: currentcolor;
}
.commit-sha, .sha, [class*="sha"],
.branch-name, .tag-name, .file-name, code.ref {
font-family: var(--fonts-monospace);
}
/* Yellow / olive / orange labels + buttons: white on these hues fails WCAG
AA. Use --fw-foreground (near-black). The .ui.ui.ui chain matches
Fomantic's triple-class specificity for state-modified labels. */
.ui.ui.ui.yellow.label,
.ui.ui.ui.olive.label,
.ui.ui.ui.orange.label,
.ui.yellow.button,
.ui.olive.button,
.ui.orange.button,
.ui.yellow.button:hover,
.ui.olive.button:hover,
.ui.orange.button:hover {
color: var(--fw-foreground);
}
/* Basic-green label text must darken for AA on cream (base green is 3.8:1). */
.ui.basic.green.label,
.ui.basic.green.labels .label {
color: var(--color-green-dark-1);
border-color: var(--color-green-dark-1);
}
/* Grey-label hover: Fomantic flips text to var(--color-white) while bg stays
semi-transparent beige → invisible. Preserve label-text on hover. */
a.ui.ui.ui.grey.label:hover,
a.ui.ui.ui.grey.label:focus {
color: var(--color-label-text);
background: var(--color-label-hover-bg);
}
/* PR review-comments-counter uses --color-primary-light-4 (pale navy) bg
with cream text → fails contrast. Use opaque primary instead. */
.review-comments-counter {
background-color: var(--color-primary);
color: var(--color-primary-contrast);
}
/* Card .extra divider uses --color-secondary-light-1 (paler than secondary
on cream = invisible divider). Force visible. */
.ui.card > .extra,
.ui.cards .card > .extra {
border-top-color: var(--color-secondary);
}
/* `.text.small` at 0.75em serif renders poorly; keep system font for it. */
.text.small,
.flex-item-body .text.small {
font-family: var(--fonts-proportional), sans-serif;
}
/* base.css uses `color: var(--color-COLOR) !important` for .text.COLOR
helpers. Several hues fail AA on cream; route them through darker shades.
We must use !important here because base.css line 920-ish does too —
without !important we lose the cascade. */
.text.orange { color: var(--color-orange-dark-2) !important; /* AA: base.css uses !important */ }
.text.yellow { color: #7F5E08 !important; /* AA: base.css uses !important */ }
.text.olive { color: var(--color-olive-dark-2) !important; /* AA: base.css uses !important */ }
.text.teal { color: var(--color-teal-dark-2) !important; /* AA: base.css uses !important */ }
.text.brown { color: var(--color-brown-dark-2) !important; /* AA: base.css uses !important */ }
.text.pink { color: var(--color-pink-dark-1) !important; /* AA: base.css uses !important */ }
.text.gold { color: var(--color-brown-dark-2) !important; /* AA: base.css uses !important */ }
.text.green { color: var(--color-green-dark-1) !important; /* AA: base.css uses !important */ }
/* Flash messages: force readable text on pastel bg tints. */
.ui.positive.message, .ui.success.message { color: var(--color-success-text); }
.ui.negative.message, .ui.error.message { color: var(--color-error-text); }
.ui.warning.message { color: var(--color-warning-text); }
.ui.info.message { color: var(--color-info-text); }
/* Secondary-nav (repo breadcrumb strip) — use !important because Fomantic's
.ui.secondary.menu rule uses it too; matching specificity. */
.secondary-nav {
background: var(--color-secondary-nav-bg) !important; /* beats .ui.secondary.menu !important in Fomantic */
border-bottom: 1px solid var(--color-secondary);
}
/* Link color: base.css sets `a { color: var(--color-primary) }`. In fhirworx
--color-primary = dark navy = same family as --color-text, so links look
identical to prose. Route real hyperlinks through a brighter blue. Exclude
UI atoms (.item/.button/.label/.tab) which carry Fomantic color logic. */
a:not(.item):not(.button):not(.label):not(.tab) {
color: var(--color-blue);
}
a:not(.item):not(.button):not(.label):not(.tab):hover {
color: var(--color-blue-dark-1);
}
a.muted, a.suppressed, a.silenced, .muted-links a {
color: inherit;
}
/* Diff detail yellow count fails AA on cream */
.repository .diff-detail-box .diff-detail-stats strong:nth-of-type(1) {
color: var(--color-yellow-dark-1);
}
/* ============================================================================
Defense against client-side dark mode (Chrome auto-dark, Firefox content-
override). Even with `color-scheme: light only`, some UAs still apply
per-element dark adjustments when prefers-color-scheme:dark. Re-assert.
============================================================================ */
@media (prefers-color-scheme: dark) {
:root {
color-scheme: light only !important; /* beats UA-injected dark scheme */
--color-body: var(--fw-background) !important;
--color-text: var(--fw-foreground) !important;
--color-nav-bg: var(--fw-nav-bg) !important;
--color-nav-text: var(--fw-nav-text) !important;
--color-box-body: var(--fw-card) !important;
--color-card: var(--fw-card) !important;
--color-menu: var(--fw-card) !important;
}
body {
background: var(--fw-background) !important; /* beats UA dark bg */
color: var(--fw-foreground) !important; /* beats UA dark text */
}
#navbar {
background: var(--fw-nav-bg) !important; /* beats UA dark bg */
color: var(--fw-nav-text) !important; /* beats UA dark text */
}
}
@media (forced-colors: active) {
#navbar { border-bottom: 3px solid CanvasText; }
}

9
infra/images/api.Dockerfile
View File

@@ -6,8 +6,11 @@ WORKDIR /app
# Local package registry (Gitea) — set via --build-arg to pull from mirror
ARG PYPI_INDEX_URL=""
# Patch base image CVEs
RUN apt-get update && apt-get upgrade -y && rm -rf /var/lib/apt/lists/*
# Patch base image CVEs + install curl for healthcheck.
# (Python cold-start on this image is 10-13s — too slow for the 5s
# healthcheck timeout, so Docker marks the container unhealthy even
# though /health responds in milliseconds.)
RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
# Copy project files for install
COPY pyproject.toml uv.lock README.md ./
@@ -26,7 +29,7 @@ COPY stack.toml ./
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" || exit 1
CMD curl -sf http://localhost:8000/health || exit 1
CMD ["uv", "run", "--no-sync", "uvicorn", "api.server:app", \
"--host", "0.0.0.0", "--port", "8000", \

9
infra/images/docs.Dockerfile
View File

@@ -10,8 +10,13 @@ COPY docs/scripts/ docs/scripts/
# Extract API docs via griffe (pure AST, no imports)
RUN uv run --with griffe python docs/scripts/extract_docs.py
# Export bibliography (gracefully handles missing bib.sqlite)
COPY data/bib.sqlit[e] data/
# Export bibliography. The COPY uses a multi-source form so the
# bib.sqlite is optional — `data/.gitkeep` guarantees there's always
# at least one matching source so BuildKit doesn't error on an empty
# glob like `data/bib.sqlit[e]` did. The export script in turn
# tolerates a missing bib.sqlite and writes an empty library.json.
RUN mkdir -p data
COPY data/.gitkeep data/bib.sqlit[e] data/
RUN uv run --with pydantic python docs/scripts/export_library.py

85
infra/images/notebooks.Dockerfile
View File

@@ -1,4 +1,69 @@
# syntax=docker/dockerfile:1
#
# Notebooks runtime — fhirworx-themed marimo on CUDA + Python 3.13.
#
# Stage 1 (fe): node + pnpm. Overlay fhirworx theme onto a pinned
# marimo source tree and compile frontend + lsp into
# marimo/_static/ and marimo/_lsp/.
# Stage 2 (wheel): uv builds a marimo wheel with the fhirworx bundle
# baked in.
# Stage 3 (final): CUDA runtime + Python stack; installs the wheel into
# the workspace venv.
#
# Iterate by editing infra/marimo/theme/**. BuildKit cache mounts keep
# pnpm and turbo warm between rebuilds, so CSS/icon edits land in under
# a minute after first bootstrap.
ARG MARIMO_VERSION=0.23.1
# ---- stage 1: frontend ------------------------------------------------------
FROM node:22-bookworm-slim AS fe
ARG MARIMO_VERSION
WORKDIR /src
RUN apt-get update \
&& apt-get install -y --no-install-recommends git ca-certificates python3 jq \
&& rm -rf /var/lib/apt/lists/* \
&& corepack enable \
&& corepack prepare pnpm@latest --activate
# Shallow clone marimo at the pinned tag.
RUN git clone --depth 1 --branch ${MARIMO_VERSION} --filter=blob:none \
https://github.com/marimo-team/marimo /src
COPY infra/marimo/theme /overlay
RUN /overlay/scripts/apply-overlay.sh /src /overlay
RUN --mount=type=cache,target=/root/.cache/pnpm,sharing=locked \
pnpm install --no-frozen-lockfile
RUN --mount=type=cache,target=/root/.cache/turbo,sharing=locked \
--mount=type=cache,target=/src/frontend/node_modules/.cache,sharing=locked \
NODE_ENV=production pnpm turbo build --filter @marimo-team/frontend --output-logs=full
RUN --mount=type=cache,target=/root/.cache/turbo,sharing=locked \
NODE_ENV=production pnpm turbo build --filter @marimo-team/lsp --output-logs=full
RUN rm -rf marimo/_static marimo/_lsp \
&& mkdir -p marimo/_static marimo/_lsp \
&& cp -R frontend/dist/. marimo/_static/ \
&& rm -f marimo/_static/files/wasm-intro.py \
&& cp docs/_static/CLAUDE.md marimo/_static/CLAUDE.md 2>/dev/null || true \
&& cp packages/lsp/dist/index.cjs marimo/_lsp/ \
&& { [ -d packages/lsp/dist/copilot/dist ] \
&& cp -R packages/lsp/dist/copilot/dist/. marimo/_lsp/copilot/ \
|| true; }
# ---- stage 2: wheel ---------------------------------------------------------
FROM ghcr.io/astral-sh/uv:python3.13-bookworm-slim AS wheel
WORKDIR /src
COPY --from=fe /src/ /src/
RUN uv build --wheel
# ---- stage 3: runtime -------------------------------------------------------
FROM nvidia/cuda:12.6.0-runtime-ubuntu24.04
ARG USERNAME=kert
@@ -15,32 +80,30 @@ ENV DEBIAN_FRONTEND=noninteractive \
NVIDIA_VISIBLE_DEVICES=all \
NVIDIA_DRIVER_CAPABILITIES=compute,utility
# System dependencies
RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends \
curl \
ca-certificates \
git \
build-essential \
curl ca-certificates git build-essential \
&& rm -rf /var/lib/apt/lists/*
# Rename existing ubuntu user/group to kert and fix home ownership
RUN groupmod -n ${USERNAME} ubuntu \
&& usermod -l ${USERNAME} -d /home/${USERNAME} -m -s /bin/bash ubuntu \
&& chown -R ${USER_UID}:${USER_GID} /home/${USERNAME}
# Install uv
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
COPY --from=ghcr.io/astral-sh/uv:latest /uvx /usr/local/bin/uvx
# Stay as root for rootless Docker compatibility
# (root in container = host user in rootless Docker)
WORKDIR /home/${USERNAME}
# Initialize uv project and install dependencies
# Bring in the fhirworx marimo wheel from the wheel stage.
COPY --from=wheel /src/dist/ /tmp/marimo-wheel/
# Install python + init workspace venv. Use the local wheel for marimo so
# the fhirworx frontend ships in the image; no upstream PyPI fetch for
# marimo itself.
RUN uv python install ${PYTHON_VERSION} \
&& uv init workspace --python ${PYTHON_VERSION} \
&& cd workspace \
&& uv add "marimo[recommended]" polars cudf-polars-cu12 pandas numpy pyarrow \
&& MARIMO_WHL=$(ls /tmp/marimo-wheel/marimo-*.whl | head -1) \
&& uv add "${MARIMO_WHL}[recommended]" polars cudf-polars-cu12 pandas numpy pyarrow \
"pyiceberg[s3,pyarrow]>=0.7.0" "duckdb>=1.0.0" "narwhals>=1.0.0" "trino>=0.328.0" \
"sqlglot>=26.0.0" \
vega_datasets pyzotero obstore s3fs \

147
infra/mail/maddy.conf.tpl Normal file
View File

@@ -0,0 +1,147 @@
# Maddy mail server config — minimal single-domain setup with DKIM
# autosigning. Tracks the shape of the upstream reference config so
# upgrades track cleanly.
#
# Placeholders (substituted by mail-setup.sh via envsubst):
# $HOSTNAME -> mail.corwins.media
# $PRIMARY_DOMAIN -> corwins.media
hostname $HOSTNAME
$(primary_domain) = $PRIMARY_DOMAIN
# Accept mail at both the apex (e.g. git@fhirworx.io for outbound
# SMTP submission and DKIM signing) and the host subdomain (e.g.
# cmsupdates@mail.fhirworx.io, which is what actually MX-routes to
# this droplet when another provider holds the apex MX).
$(local_domains) = $(primary_domain) $HOSTNAME
tls file /data/tls/fullchain.pem /data/tls/privkey.pem
# ── Authentication / storage ─────────────────────────────────────
#
# Accounts live in credentials.db; IMAP mailboxes live in imapsql.db.
# Both are SQLite files under /data and survive container restarts.
auth.pass_table local_authdb {
table sql_table {
driver sqlite3
dsn credentials.db
table_name passwords
}
}
storage.imapsql local_mailboxes {
driver sqlite3
dsn imapsql.db
}
# ── SMTP receive (port 25, from the world) ───────────────────────
#
# Plain SMTP inbound, STARTTLS opportunistically, no auth (inbound
# MTAs shouldn't authenticate). DMARC + DKIM + SPF checks gate the
# inbound stream; anything forging our domains is rejected.
smtp tcp://0.0.0.0:25 {
limits {
all rate 20 1s
all concurrency 10
}
dmarc yes
check {
require_mx_record
dkim
spf
}
source $(local_domains) {
reject 501 5.1.8 "Use Submission (port 587) to send as us"
}
default_source {
destination postmaster $(local_domains) {
deliver_to &local_routing
}
default_destination {
reject 550 5.1.1 "User not found"
}
}
}
# ── SMTP submission (ports 465 TLS, 587 STARTTLS, authenticated) ──
#
# Our own outbound send path. Clients (Gitea, Thunderbird, mail-merge
# scripts) authenticate against local_authdb, and outbound mail is
# DKIM-signed before hitting the remote queue.
submission tls://0.0.0.0:465 tcp://0.0.0.0:587 {
limits {
all rate 50 1s
}
auth &local_authdb
source $(local_domains) {
check {
authorize_sender {
prepare_email identity
user_to_email identity
}
}
destination postmaster $(local_domains) {
deliver_to &local_routing
}
default_destination {
modify {
dkim $(primary_domain) default
}
deliver_to &remote_queue
}
}
default_source {
reject 501 5.1.8 "Unknown sender domain"
}
}
# Outbound goes through an external SMTP relay (smarthost), not
# direct MX — DigitalOcean silently blocks outbound port 25 on new
# droplets. The real `target.smtp outbound_smarthost { … }` block is
# injected at the top of this file by `stack mail attach-smarthost`.
# Until then, the stub below lets maddy start cleanly so inbound,
# IMAP, and DKIM key generation all work — outbound just queues
# locally and never delivers (intentional).
target.smtp outbound_smarthost {
targets tcp://127.0.0.1:9
auth plain "stub" "stub"
}
target.queue remote_queue {
target &outbound_smarthost
autogenerated_msg_domain $(primary_domain)
bounce {
destination postmaster $(local_domains) {
deliver_to &local_routing
}
default_destination {
reject 550 5.0.0 "Refusing to send DSNs to non-local addresses"
}
}
}
# ── IMAP (port 993 TLS, 143 STARTTLS) ────────────────────────────
imap tls://0.0.0.0:993 tcp://0.0.0.0:143 {
auth &local_authdb
storage &local_mailboxes
}
# ── Local delivery pipeline ──────────────────────────────────────
msgpipeline local_routing {
destination postmaster $(local_domains) {
deliver_to &local_mailboxes
}
default_destination {
reject 550 5.1.1 "User doesn't exist"
}
}

4
infra/marimo/home-page-patched.js
View File

File diff suppressed because one or more lines are too long

BIN
infra/marimo/theme/frontend/public/android-chrome-192x192.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 28 KiB

BIN
infra/marimo/theme/frontend/public/android-chrome-512x512.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 28 KiB

BIN
infra/marimo/theme/frontend/public/apple-touch-icon.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

BIN
infra/marimo/theme/frontend/public/favicon-16x16.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 741 B

BIN
infra/marimo/theme/frontend/public/favicon-32x32.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.0 KiB

3
infra/marimo/theme/frontend/public/favicon.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 29 KiB

BIN
infra/marimo/theme/frontend/public/logo.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 28 KiB

148
infra/marimo/theme/frontend/src/css/fhirworx.css Normal file
View File

@@ -0,0 +1,148 @@
/* fhirworx theme for marimo — ProPublica-editorial × Federal-Register-civic.
*
* Keep overrides minimal and load AFTER marimo's globals.css so our
* cascade wins without !important. Variables first, then component
* touch-ups. Matches the Gitea theme's variable-completeness rule:
* define everything marimo references or the cascade falls through.
*/
/* Force light mode. The Radix color scales that marimo uses for the home
* page (--blue-1, --blue-2, --slate-3, --red-1, …) resolve to dark values
* when the browser reports prefers-color-scheme: dark. Without this lock,
* the cream --background we set below fights the dark Radix panels and
* produces "slate text on dark-blue card" — unreadable. `only light` tells
* the UA to ignore the system pref entirely, so light-dark() and the
* radix @import'ed -dark files don't kick in. */
:root {
color-scheme: only light;
}
:root,
.marimo {
/* Typography: editorial serif + Federal-Register humanist monospace */
--marimo-heading-font: "Source Serif 4", "Source Serif Pro", ui-serif, Georgia, serif;
--marimo-text-font: "Source Serif 4", "Source Serif Pro", ui-serif, Georgia, serif;
--marimo-monospace-font: "IBM Plex Mono", "Berkeley Mono", ui-monospace, SFMono-Regular, Menlo, monospace;
/* Paper stock + ink */
--fw-paper: #F6F2E8; /* warm cream */
--fw-paper-sunk: #EDE7D6;
--fw-ink: #141B2D; /* midnight navy */
--fw-ink-soft: #2A3349;
--fw-ink-faint: #5A6378;
--fw-rule: #B8B3A4; /* beige-dark — warm border that reads against cream */
--fw-accent: #9A2A2A; /* signal red, sparingly */
--fw-accent-soft: #D9AD5C; /* brass */
/* Override marimo's palette in both schemes */
--background: var(--fw-paper);
--foreground: var(--fw-ink);
--muted: var(--fw-paper-sunk);
--muted-foreground: var(--fw-ink-faint);
--popover: var(--fw-paper);
--popover-foreground: var(--fw-ink);
--card: var(--fw-paper);
--card-foreground: var(--fw-ink);
--border: var(--fw-rule);
--input: var(--fw-rule);
--primary: var(--fw-ink);
--primary-foreground: var(--fw-paper);
--secondary: var(--fw-paper-sunk);
--secondary-foreground: var(--fw-ink);
--accent: var(--fw-accent-soft);
--accent-foreground: var(--fw-ink);
--destructive: var(--fw-accent);
--destructive-foreground: var(--fw-paper);
--radius: 2px; /* civic forms, not rounded chrome */
--markdown-max-width: 72ch; /* ProPublica measure */
/* Radix color scales used directly by marimo's home page UI. The stock
* scales don't fit our cream/navy palette; these give notebook panels
* and hover states a coherent look. Keep the variable names because
* the source references them as `bg-(--blue-2)`, `divide-(--slate-3)`,
* etc. via Tailwind arbitrary-value syntax. */
--blue-1: var(--fw-paper); /* create-new card rest bg */
--blue-2: #EDE1C4; /* card/row hover — warm straw */
--blue-3: #E5D6AE;
--slate-1: var(--fw-paper);
--slate-2: var(--fw-paper-sunk);
--slate-3: var(--fw-rule); /* divider between notebook rows */
--slate-4: var(--fw-rule);
--red-1: #F4E6E0; /* shutdown button hover bg */
--red-2: #E8CCC1;
}
/* Body & headings — serif first, tight leading, generous measure */
body, .marimo {
font-family: var(--marimo-text-font);
font-feature-settings: "onum" 1, "ss01" 1; /* oldstyle figures, stylistic set 1 */
letter-spacing: 0;
color: var(--fw-ink);
background: var(--fw-paper);
}
.marimo h1, .marimo h2, .marimo h3, .marimo h4, .marimo h5, .marimo h6 {
font-family: var(--marimo-heading-font);
font-weight: 700;
letter-spacing: -0.01em;
color: var(--fw-ink);
}
/* Icons: Tabler defaults to stroke-width 2 — thin them for editorial feel.
* Scoped to .tabler-icon so our utility icons elsewhere aren't affected. */
.tabler-icon,
svg.tabler-icon {
stroke-width: 1.5;
}
/* Rules & borders — single hairline, warm. Kill the subtle gray-on-white
* that looks muddy against cream. */
.marimo hr,
.marimo [role="separator"] {
border-color: var(--fw-rule);
}
/* Code cells — no rounded corners, let the typewriter feel breathe */
.marimo .cm-editor {
font-family: var(--marimo-monospace-font);
border-radius: 0;
}
/* Buttons — flatten, no gradients, clear hit target */
.marimo button,
.marimo [role="button"] {
border-radius: var(--radius);
font-weight: 500;
}
/* Links — underline on hover only, ink color always */
.marimo a {
color: var(--fw-ink);
text-decoration-color: var(--fw-rule);
text-underline-offset: 2px;
}
.marimo a:hover {
text-decoration-color: var(--fw-ink);
}
/* Home-page panels. Notebook list rows use `hover:bg-(--blue-2)` and
* `hover:text-primary` — that means text color on hover switches to
* --primary, which we set to --fw-ink. Make sure every text color in
* those rows resolves against our palette, not an inherited slate. */
.marimo a,
.marimo a:visited,
.marimo .text-muted-foreground,
.marimo .text-primary {
color: var(--fw-ink);
}
.marimo .text-muted-foreground {
color: var(--fw-ink-faint);
}
/* "Create a new notebook" hero card — marimo classes it `bg-(--blue-1)`
* with an accent shadow. Keep it on cream with a subtle brass border so
* the eye lands there without shouting. */
.marimo [class*="bg-(--blue-1)"] {
background: var(--fw-paper);
border-color: var(--fw-accent-soft);
}

514
infra/marimo/theme/frontend/src/lucide-shim.tsx Normal file
View File

@@ -0,0 +1,514 @@
// Auto-generated: Tabler Icons as a drop-in for lucide-react.
// Build-time replacement; do not edit by hand — see gen-lucide-shim.py
import type { ComponentType, SVGProps } from 'react';
import {
IconActivity,
IconAlertCircle,
IconAlertOctagon,
IconAlertTriangle,
IconAlignCenter,
IconAlignJustified,
IconAlignLeft,
IconAlignRight,
IconAlphabetLatin,
IconArrowBackUp,
IconArrowBarRight,
IconArrowBarToDown,
IconArrowBarToRight,
IconArrowBarToUp,
IconArrowDown,
IconArrowLeft,
IconArrowRight,
IconArrowsMaximize,
IconArrowsShuffle,
IconArrowsSort,
IconArrowsUpDown,
IconArticle,
IconAt,
IconBan,
IconBaselineDensitySmall,
IconBinary,
IconBolt,
IconBoltOff,
IconBook,
IconBookmark,
IconBox,
IconBraces,
IconBrackets,
IconBrain,
IconBrandGithub,
IconBrandYoutube,
IconBrush,
IconBug,
IconCalendar,
IconCalendarTime,
IconChartArea,
IconChartBar,
IconChartDots,
IconChartLine,
IconChartPie,
IconCheck,
IconChevronDown,
IconChevronLeft,
IconChevronRight,
IconChevronUp,
IconChevronsDown,
IconChevronsLeft,
IconChevronsRight,
IconChevronsUp,
IconCircle,
IconCircleCheck,
IconCircleChevronDown,
IconCircleChevronRight,
IconCircleDashed,
IconCirclePlus,
IconCircleX,
IconClipboardCopy,
IconClipboardText,
IconClock,
IconCloudDownload,
IconCode,
IconColumns,
IconColumns2,
IconCommand,
IconConfetti,
IconCopy,
IconCopyOff,
IconCornerLeftUp,
IconCpu,
IconCrosshair,
IconCursorText,
IconDatabase,
IconDeviceDesktop,
IconDeviceFloppy,
IconDeviceSdCard,
IconDiamond,
IconDots,
IconDotsCircleHorizontal,
IconDotsVertical,
IconDownload,
IconEdit,
IconEqual,
IconEraser,
IconExternalLink,
IconEye,
IconEyeOff,
IconFile,
IconFileCode,
IconFileMusic,
IconFilePencil,
IconFilePlus,
IconFileSpreadsheet,
IconFileSymlink,
IconFileText,
IconFileTypography,
IconFiles,
IconFilter,
IconFilterOff,
IconFilterPlus,
IconFlask,
IconFolder,
IconFolderCog,
IconFolderDown,
IconFolderPlus,
IconFolders,
IconGrid3x3,
IconGripHorizontal,
IconGripVertical,
IconHash,
IconHelpCircle,
IconHistory,
IconHome,
IconHourglass,
IconInfoCircle,
IconJson,
IconKey,
IconKeyboard,
IconLayout,
IconLayoutBoard,
IconLayoutRows,
IconLayoutSidebar,
IconLayoutSidebarRight,
IconLetterCase,
IconLink,
IconList,
IconListNumbers,
IconListTree,
IconLoader,
IconLoader2,
IconLock,
IconMail,
IconMapPin,
IconMathFunction,
IconMenu2,
IconMessageCircle,
IconMessageCircleQuestion,
IconMessages,
IconMinus,
IconMoodHappy,
IconNetwork,
IconNotebook,
IconPackage,
IconPackageOff,
IconPaperclip,
IconPencil,
IconPhoto,
IconPin,
IconPinnedOff,
IconPlayerPlay,
IconPlayerSkipForward,
IconPlayerStop,
IconPlayerTrackNext,
IconPlug,
IconPlus,
IconPointer,
IconPointerShare,
IconPower,
IconPresentation,
IconRefresh,
IconRegex,
IconRobot,
IconRotate,
IconRotateClockwise,
IconRuler,
IconSchool,
IconScissors,
IconSearch,
IconSelector,
IconSend,
IconSettings,
IconShare,
IconShieldCheck,
IconSitemap,
IconSparkles,
IconSquare,
IconSquareArrowRight,
IconSquareCheck,
IconSquareDashed,
IconSquareLetterM,
IconSquarePlus,
IconStack,
IconStackPush,
IconSum,
IconTable,
IconTerminal,
IconTerminal2,
IconTextCaption,
IconTextWrap,
IconToggleLeft,
IconTool,
IconTrash,
IconTriangle,
IconTypography,
IconUnlink,
IconUpload,
IconUsers,
IconUsersGroup,
IconVariable,
IconVideo,
IconWall,
IconWifi,
IconWifiOff,
IconWorld,
IconX,
IconZoomCheck,
IconZoomCode,
type IconProps,
} from '@tabler/icons-react';
// Prop shapes mirror lucide's public types so Omit<LucideProps, 'ref'>
// and similar patterns in upstream marimo code keep typechecking.
export type LucideProps = SVGProps<SVGSVGElement> & {
size?: number | string;
absoluteStrokeWidth?: boolean;
};
export type LucideIcon = ComponentType<LucideProps & IconProps>;
export const ActivityIcon: LucideIcon = IconActivity as LucideIcon;
export const AlertCircle: LucideIcon = IconAlertCircle as LucideIcon;
export const AlertCircleIcon: LucideIcon = IconAlertCircle as LucideIcon;
export const AlertOctagonIcon: LucideIcon = IconAlertOctagon as LucideIcon;
export const AlertTriangle: LucideIcon = IconAlertTriangle as LucideIcon;
export const AlertTriangleIcon: LucideIcon = IconAlertTriangle as LucideIcon;
export const AlignCenterVerticalIcon: LucideIcon = IconAlignCenter as LucideIcon;
export const AlignEndVerticalIcon: LucideIcon = IconAlignRight as LucideIcon;
export const AlignHorizontalSpaceAroundIcon: LucideIcon = IconAlignJustified as LucideIcon;
export const AlignJustifyIcon: LucideIcon = IconAlignJustified as LucideIcon;
export const AlignStartVerticalIcon: LucideIcon = IconAlignLeft as LucideIcon;
export const AreaChartIcon: LucideIcon = IconChartArea as LucideIcon;
export const ArrowDownIcon: LucideIcon = IconArrowDown as LucideIcon;
export const ArrowDownToLineIcon: LucideIcon = IconArrowBarToDown as LucideIcon;
export const ArrowDownWideNarrowIcon: LucideIcon = IconArrowsSort as LucideIcon;
export const ArrowLeftIcon: LucideIcon = IconArrowLeft as LucideIcon;
export const ArrowRightFromLineIcon: LucideIcon = IconArrowBarRight as LucideIcon;
export const ArrowRightIcon: LucideIcon = IconArrowRight as LucideIcon;
export const ArrowRightSquareIcon: LucideIcon = IconSquareArrowRight as LucideIcon;
export const ArrowRightToLineIcon: LucideIcon = IconArrowBarToRight as LucideIcon;
export const ArrowUpDownIcon: LucideIcon = IconArrowsUpDown as LucideIcon;
export const ArrowUpNarrowWideIcon: LucideIcon = IconArrowsSort as LucideIcon;
export const ArrowUpToLineIcon: LucideIcon = IconArrowBarToUp as LucideIcon;
export const ArrowUpWideNarrowIcon: LucideIcon = IconArrowsSort as LucideIcon;
export const AtSignIcon: LucideIcon = IconAt as LucideIcon;
export const BanIcon: LucideIcon = IconBan as LucideIcon;
export const BarChart2Icon: LucideIcon = IconChartBar as LucideIcon;
export const BarChartBigIcon: LucideIcon = IconChartBar as LucideIcon;
export const BarChartIcon: LucideIcon = IconChartBar as LucideIcon;
export const BaselineIcon: LucideIcon = IconBaselineDensitySmall as LucideIcon;
export const BetweenHorizontalStartIcon: LucideIcon = IconLayoutRows as LucideIcon;
export const BinaryIcon: LucideIcon = IconBinary as LucideIcon;
export const BookMarkedIcon: LucideIcon = IconBookmark as LucideIcon;
export const BookOpenIcon: LucideIcon = IconBook as LucideIcon;
export const BookPlusIcon: LucideIcon = IconBook as LucideIcon;
export const BookTextIcon: LucideIcon = IconBook as LucideIcon;
export const BotIcon: LucideIcon = IconRobot as LucideIcon;
export const BotMessageSquareIcon: LucideIcon = IconRobot as LucideIcon;
export const BoxIcon: LucideIcon = IconBox as LucideIcon;
export const BracesIcon: LucideIcon = IconBraces as LucideIcon;
export const BracketsIcon: LucideIcon = IconBrackets as LucideIcon;
export const BrainIcon: LucideIcon = IconBrain as LucideIcon;
export const BrickWallIcon: LucideIcon = IconWall as LucideIcon;
export const BugPlayIcon: LucideIcon = IconBug as LucideIcon;
export const CalendarClockIcon: LucideIcon = IconCalendarTime as LucideIcon;
export const CalendarIcon: LucideIcon = IconCalendar as LucideIcon;
export const CaseSensitiveIcon: LucideIcon = IconLetterCase as LucideIcon;
export const ChartColumn: LucideIcon = IconChartBar as LucideIcon;
export const ChartColumnIcon: LucideIcon = IconChartBar as LucideIcon;
export const ChartNoAxesColumn: LucideIcon = IconChartBar as LucideIcon;
export const ChartPieIcon: LucideIcon = IconChartPie as LucideIcon;
export const ChartScatterIcon: LucideIcon = IconChartDots as LucideIcon;
export const ChartSplineIcon: LucideIcon = IconChartLine as LucideIcon;
export const Check: LucideIcon = IconCheck as LucideIcon;
export const CheckCircle2Icon: LucideIcon = IconCircleCheck as LucideIcon;
export const CheckCircleIcon: LucideIcon = IconCircleCheck as LucideIcon;
export const CheckIcon: LucideIcon = IconCheck as LucideIcon;
export const CheckSquareIcon: LucideIcon = IconSquareCheck as LucideIcon;
export const ChevronDown: LucideIcon = IconChevronDown as LucideIcon;
export const ChevronDownCircleIcon: LucideIcon = IconCircleChevronDown as LucideIcon;
export const ChevronDownIcon: LucideIcon = IconChevronDown as LucideIcon;
export const ChevronLeft: LucideIcon = IconChevronLeft as LucideIcon;
export const ChevronLeftIcon: LucideIcon = IconChevronLeft as LucideIcon;
export const ChevronRight: LucideIcon = IconChevronRight as LucideIcon;
export const ChevronRightCircleIcon: LucideIcon = IconCircleChevronRight as LucideIcon;
export const ChevronRightIcon: LucideIcon = IconChevronRight as LucideIcon;
export const ChevronUp: LucideIcon = IconChevronUp as LucideIcon;
export const ChevronUpIcon: LucideIcon = IconChevronUp as LucideIcon;
export const ChevronsDownIcon: LucideIcon = IconChevronsDown as LucideIcon;
export const ChevronsDownUpIcon: LucideIcon = IconSelector as LucideIcon;
export const ChevronsLeft: LucideIcon = IconChevronsLeft as LucideIcon;
export const ChevronsRight: LucideIcon = IconChevronsRight as LucideIcon;
export const ChevronsUpDown: LucideIcon = IconSelector as LucideIcon;
export const ChevronsUpDownIcon: LucideIcon = IconSelector as LucideIcon;
export const ChevronsUpIcon: LucideIcon = IconChevronsUp as LucideIcon;
export const Circle: LucideIcon = IconCircle as LucideIcon;
export const CircleCheck: LucideIcon = IconCircleCheck as LucideIcon;
export const CircleCheckIcon: LucideIcon = IconCircleCheck as LucideIcon;
export const CircleEllipsis: LucideIcon = IconDotsCircleHorizontal as LucideIcon;
export const CircleHelpIcon: LucideIcon = IconHelpCircle as LucideIcon;
export const CircleIcon: LucideIcon = IconCircle as LucideIcon;
export const CirclePlayIcon: LucideIcon = IconPlayerPlay as LucideIcon;
export const CircleX: LucideIcon = IconCircleX as LucideIcon;
export const ClipboardCopyIcon: LucideIcon = IconClipboardCopy as LucideIcon;
export const ClipboardPasteIcon: LucideIcon = IconClipboardText as LucideIcon;
export const ClockIcon: LucideIcon = IconClock as LucideIcon;
export const Code2Icon: LucideIcon = IconCode as LucideIcon;
export const CodeIcon: LucideIcon = IconCode as LucideIcon;
export const Cog: LucideIcon = IconSettings as LucideIcon;
export const Columns2Icon: LucideIcon = IconColumns2 as LucideIcon;
export const ColumnsIcon: LucideIcon = IconColumns as LucideIcon;
export const CombineIcon: LucideIcon = IconStackPush as LucideIcon;
export const CommandIcon: LucideIcon = IconCommand as LucideIcon;
export const Copy: LucideIcon = IconCopy as LucideIcon;
export const CopyIcon: LucideIcon = IconCopy as LucideIcon;
export const CopyMinusIcon: LucideIcon = IconCopy as LucideIcon;
export const CopySlashIcon: LucideIcon = IconCopyOff as LucideIcon;
export const CornerLeftUp: LucideIcon = IconCornerLeftUp as LucideIcon;
export const CpuIcon: LucideIcon = IconCpu as LucideIcon;
export const CrosshairIcon: LucideIcon = IconCrosshair as LucideIcon;
export const CurlyBracesIcon: LucideIcon = IconBraces as LucideIcon;
export const DatabaseIcon: LucideIcon = IconDatabase as LucideIcon;
export const DatabaseZap: LucideIcon = IconDatabase as LucideIcon;
export const DatabaseZapIcon: LucideIcon = IconDatabase as LucideIcon;
export const DiamondPlusIcon: LucideIcon = IconDiamond as LucideIcon;
export const DownloadCloudIcon: LucideIcon = IconCloudDownload as LucideIcon;
export const DownloadIcon: LucideIcon = IconDownload as LucideIcon;
export const Edit3Icon: LucideIcon = IconEdit as LucideIcon;
export const EditIcon: LucideIcon = IconEdit as LucideIcon;
export const EllipsisIcon: LucideIcon = IconDots as LucideIcon;
export const EraserIcon: LucideIcon = IconEraser as LucideIcon;
export const ExpandIcon: LucideIcon = IconArrowsMaximize as LucideIcon;
export const ExternalLinkIcon: LucideIcon = IconExternalLink as LucideIcon;
export const EyeIcon: LucideIcon = IconEye as LucideIcon;
export const EyeOffIcon: LucideIcon = IconEyeOff as LucideIcon;
export const FastForwardIcon: LucideIcon = IconPlayerTrackNext as LucideIcon;
export const FileAudio2Icon: LucideIcon = IconFileMusic as LucideIcon;
export const FileAudioIcon: LucideIcon = IconFileMusic as LucideIcon;
export const FileCodeIcon: LucideIcon = IconFileCode as LucideIcon;
export const FileIcon: LucideIcon = IconFile as LucideIcon;
export const FileImageIcon: LucideIcon = IconPhoto as LucideIcon;
export const FileJsonIcon: LucideIcon = IconJson as LucideIcon;
export const FilePenIcon: LucideIcon = IconFilePencil as LucideIcon;
export const FilePlus2Icon: LucideIcon = IconFilePlus as LucideIcon;
export const FileSpreadsheetIcon: LucideIcon = IconFileSpreadsheet as LucideIcon;
export const FileSymlink: LucideIcon = IconFileSymlink as LucideIcon;
export const FileTextIcon: LucideIcon = IconFileText as LucideIcon;
export const FileVideoCameraIcon: LucideIcon = IconFileTypography as LucideIcon;
export const FileVideoIcon: LucideIcon = IconVideo as LucideIcon;
export const Files: LucideIcon = IconFiles as LucideIcon;
export const FilterIcon: LucideIcon = IconFilter as LucideIcon;
export const FilterX: LucideIcon = IconFilterOff as LucideIcon;
export const FlaskConicalIcon: LucideIcon = IconFlask as LucideIcon;
export const FolderArchiveIcon: LucideIcon = IconFolder as LucideIcon;
export const FolderCog2: LucideIcon = IconFolderCog as LucideIcon;
export const FolderDownIcon: LucideIcon = IconFolderDown as LucideIcon;
export const FolderIcon: LucideIcon = IconFolder as LucideIcon;
export const FolderPlusIcon: LucideIcon = IconFolderPlus as LucideIcon;
export const FolderTreeIcon: LucideIcon = IconFolders as LucideIcon;
export const FunctionSquareIcon: LucideIcon = IconMathFunction as LucideIcon;
export const FunnelPlusIcon: LucideIcon = IconFilterPlus as LucideIcon;
export const GithubIcon: LucideIcon = IconBrandGithub as LucideIcon;
export const GlobeIcon: LucideIcon = IconWorld as LucideIcon;
export const GraduationCapIcon: LucideIcon = IconSchool as LucideIcon;
export const Grid3x3Icon: LucideIcon = IconGrid3x3 as LucideIcon;
export const GridIcon: LucideIcon = IconGrid3x3 as LucideIcon;
export const GripHorizontal: LucideIcon = IconGripHorizontal as LucideIcon;
export const GripHorizontalIcon: LucideIcon = IconGripHorizontal as LucideIcon;
export const GripVerticalIcon: LucideIcon = IconGripVertical as LucideIcon;
export const GroupIcon: LucideIcon = IconUsersGroup as LucideIcon;
export const HardDrive: LucideIcon = IconDeviceSdCard as LucideIcon;
export const HardDriveDownloadIcon: LucideIcon = IconDeviceSdCard as LucideIcon;
export const HardDriveIcon: LucideIcon = IconDeviceSdCard as LucideIcon;
export const HashIcon: LucideIcon = IconHash as LucideIcon;
export const HatGlasses: LucideIcon = IconMoodHappy as LucideIcon;
export const HelpCircleIcon: LucideIcon = IconHelpCircle as LucideIcon;
export const HistoryIcon: LucideIcon = IconHistory as LucideIcon;
export const Home: LucideIcon = IconHome as LucideIcon;
export const HomeIcon: LucideIcon = IconHome as LucideIcon;
export const HourglassIcon: LucideIcon = IconHourglass as LucideIcon;
export const ImageIcon: LucideIcon = IconPhoto as LucideIcon;
export const Info: LucideIcon = IconInfoCircle as LucideIcon;
export const InfoIcon: LucideIcon = IconInfoCircle as LucideIcon;
export const KeyIcon: LucideIcon = IconKey as LucideIcon;
export const KeyRoundIcon: LucideIcon = IconKey as LucideIcon;
export const KeyboardIcon: LucideIcon = IconKeyboard as LucideIcon;
export const LayersIcon: LucideIcon = IconStack as LucideIcon;
export const LayoutIcon: LucideIcon = IconLayout as LucideIcon;
export const LayoutTemplateIcon: LucideIcon = IconLayoutBoard as LucideIcon;
export const LineChartIcon: LucideIcon = IconChartLine as LucideIcon;
export const LinkIcon: LucideIcon = IconLink as LucideIcon;
export const ListFilterIcon: LucideIcon = IconFilter as LucideIcon;
export const ListFilterPlusIcon: LucideIcon = IconFilterPlus as LucideIcon;
export const ListIcon: LucideIcon = IconList as LucideIcon;
export const ListOrderedIcon: LucideIcon = IconListNumbers as LucideIcon;
export const ListTreeIcon: LucideIcon = IconListTree as LucideIcon;
export const Loader2: LucideIcon = IconLoader2 as LucideIcon;
export const Loader2Icon: LucideIcon = IconLoader2 as LucideIcon;
export const LoaderCircle: LucideIcon = IconLoader as LucideIcon;
export const LockIcon: LucideIcon = IconLock as LucideIcon;
export const Mail: LucideIcon = IconMail as LucideIcon;
export const MapPinIcon: LucideIcon = IconMapPin as LucideIcon;
export const MemoryStickIcon: LucideIcon = IconDeviceSdCard as LucideIcon;
export const MenuIcon: LucideIcon = IconMenu2 as LucideIcon;
export const MessageCircleIcon: LucideIcon = IconMessageCircle as LucideIcon;
export const MessageCircleQuestionIcon: LucideIcon = IconMessageCircleQuestion as LucideIcon;
export const MessagesSquareIcon: LucideIcon = IconMessages as LucideIcon;
export const MicrochipIcon: LucideIcon = IconCpu as LucideIcon;
export const MinusIcon: LucideIcon = IconMinus as LucideIcon;
export const MonitorIcon: LucideIcon = IconDeviceDesktop as LucideIcon;
export const MoreHorizontal: LucideIcon = IconDots as LucideIcon;
export const MoreHorizontalIcon: LucideIcon = IconDots as LucideIcon;
export const MoreVerticalIcon: LucideIcon = IconDotsVertical as LucideIcon;
export const MousePointerSquareDashedIcon: LucideIcon = IconPointerShare as LucideIcon;
export const NetworkIcon: LucideIcon = IconNetwork as LucideIcon;
export const NotebookIcon: LucideIcon = IconNotebook as LucideIcon;
export const NotebookPenIcon: LucideIcon = IconNotebook as LucideIcon;
export const NotebookText: LucideIcon = IconNotebook as LucideIcon;
export const OrbitIcon: LucideIcon = IconCircleDashed as LucideIcon;
export const PackageCheckIcon: LucideIcon = IconPackage as LucideIcon;
export const PackageIcon: LucideIcon = IconPackage as LucideIcon;
export const PackageXIcon: LucideIcon = IconPackageOff as LucideIcon;
export const PaintRollerIcon: LucideIcon = IconBrush as LucideIcon;
export const PanelLeftIcon: LucideIcon = IconLayoutSidebar as LucideIcon;
export const PanelRightIcon: LucideIcon = IconLayoutSidebarRight as LucideIcon;
export const PaperclipIcon: LucideIcon = IconPaperclip as LucideIcon;
export const PartyPopperIcon: LucideIcon = IconConfetti as LucideIcon;
export const PencilIcon: LucideIcon = IconPencil as LucideIcon;
export const PieChartIcon: LucideIcon = IconChartPie as LucideIcon;
export const PinIcon: LucideIcon = IconPin as LucideIcon;
export const PinOffIcon: LucideIcon = IconPinnedOff as LucideIcon;
export const PlayCircleIcon: LucideIcon = IconPlayerPlay as LucideIcon;
export const PlayIcon: LucideIcon = IconPlayerPlay as LucideIcon;
export const PlaySquareIcon: LucideIcon = IconPlayerPlay as LucideIcon;
export const PlugIcon: LucideIcon = IconPlug as LucideIcon;
export const PlusCircleIcon: LucideIcon = IconCirclePlus as LucideIcon;
export const PlusIcon: LucideIcon = IconPlus as LucideIcon;
export const PlusSquareIcon: LucideIcon = IconSquarePlus as LucideIcon;
export const PowerOffIcon: LucideIcon = IconPower as LucideIcon;
export const PowerSquareIcon: LucideIcon = IconPower as LucideIcon;
export const PresentationIcon: LucideIcon = IconPresentation as LucideIcon;
export const RefreshCcw: LucideIcon = IconRefresh as LucideIcon;
export const RefreshCcwIcon: LucideIcon = IconRefresh as LucideIcon;
export const RefreshCwIcon: LucideIcon = IconRefresh as LucideIcon;
export const RegexIcon: LucideIcon = IconRegex as LucideIcon;
export const RotateCcwIcon: LucideIcon = IconRotate as LucideIcon;
export const RotateCwIcon: LucideIcon = IconRotateClockwise as LucideIcon;
export const RulerDimensionLine: LucideIcon = IconRuler as LucideIcon;
export const SaveIcon: LucideIcon = IconDeviceFloppy as LucideIcon;
export const ScissorsIcon: LucideIcon = IconScissors as LucideIcon;
export const ScrollIcon: LucideIcon = IconArticle as LucideIcon;
export const ScrollTextIcon: LucideIcon = IconArticle as LucideIcon;
export const Search: LucideIcon = IconSearch as LucideIcon;
export const SearchCheck: LucideIcon = IconZoomCheck as LucideIcon;
export const SearchIcon: LucideIcon = IconSearch as LucideIcon;
export const SendHorizontalIcon: LucideIcon = IconSend as LucideIcon;
export const SettingsIcon: LucideIcon = IconSettings as LucideIcon;
export const Share2Icon: LucideIcon = IconShare as LucideIcon;
export const ShieldCheckIcon: LucideIcon = IconShieldCheck as LucideIcon;
export const ShuffleIcon: LucideIcon = IconArrowsShuffle as LucideIcon;
export const SigmaIcon: LucideIcon = IconSum as LucideIcon;
export const SkipForwardIcon: LucideIcon = IconPlayerSkipForward as LucideIcon;
export const SparklesIcon: LucideIcon = IconSparkles as LucideIcon;
export const SquareArrowOutUpRightIcon: LucideIcon = IconExternalLink as LucideIcon;
export const SquareCodeIcon: LucideIcon = IconCode as LucideIcon;
export const SquareDashedBottomCodeIcon: LucideIcon = IconSquareDashed as LucideIcon;
export const SquareEqualIcon: LucideIcon = IconEqual as LucideIcon;
export const SquareFunction: LucideIcon = IconMathFunction as LucideIcon;
export const SquareFunctionIcon: LucideIcon = IconMathFunction as LucideIcon;
export const SquareIcon: LucideIcon = IconSquare as LucideIcon;
export const SquareMIcon: LucideIcon = IconSquareLetterM as LucideIcon;
export const SquareMousePointerIcon: LucideIcon = IconPointer as LucideIcon;
export const SquareStack: LucideIcon = IconStack as LucideIcon;
export const StopCircleIcon: LucideIcon = IconPlayerStop as LucideIcon;
export const Table2Icon: LucideIcon = IconTable as LucideIcon;
export const TableIcon: LucideIcon = IconTable as LucideIcon;
export const TerminalIcon: LucideIcon = IconTerminal as LucideIcon;
export const TerminalSquareIcon: LucideIcon = IconTerminal2 as LucideIcon;
export const TextCursorInputIcon: LucideIcon = IconCursorText as LucideIcon;
export const TextIcon: LucideIcon = IconTextCaption as LucideIcon;
export const TextSearchIcon: LucideIcon = IconZoomCode as LucideIcon;
export const TextSelectionIcon: LucideIcon = IconTextCaption as LucideIcon;
export const ToggleLeftIcon: LucideIcon = IconToggleLeft as LucideIcon;
export const Trash2Icon: LucideIcon = IconTrash as LucideIcon;
export const TrashIcon: LucideIcon = IconTrash as LucideIcon;
export const TriangleAlert: LucideIcon = IconAlertTriangle as LucideIcon;
export const TriangleIcon: LucideIcon = IconTriangle as LucideIcon;
export const TypeIcon: LucideIcon = IconTypography as LucideIcon;
export const Undo2Icon: LucideIcon = IconArrowBackUp as LucideIcon;
export const UnlinkIcon: LucideIcon = IconUnlink as LucideIcon;
export const Upload: LucideIcon = IconUpload as LucideIcon;
export const UploadIcon: LucideIcon = IconUpload as LucideIcon;
export const UsersIcon: LucideIcon = IconUsers as LucideIcon;
export const VariableIcon: LucideIcon = IconVariable as LucideIcon;
export const ViewIcon: LucideIcon = IconEye as LucideIcon;
export const WholeWordIcon: LucideIcon = IconAlphabetLatin as LucideIcon;
export const WifiIcon: LucideIcon = IconWifi as LucideIcon;
export const WifiOffIcon: LucideIcon = IconWifiOff as LucideIcon;
export const WorkflowIcon: LucideIcon = IconSitemap as LucideIcon;
export const WrapTextIcon: LucideIcon = IconTextWrap as LucideIcon;
export const WrenchIcon: LucideIcon = IconTool as LucideIcon;
export const X: LucideIcon = IconX as LucideIcon;
export const XCircle: LucideIcon = IconCircleX as LucideIcon;
export const XCircleIcon: LucideIcon = IconCircleX as LucideIcon;
export const XIcon: LucideIcon = IconX as LucideIcon;
export const YoutubeIcon: LucideIcon = IconBrandYoutube as LucideIcon;
export const ZapIcon: LucideIcon = IconBolt as LucideIcon;
export const ZapOffIcon: LucideIcon = IconBoltOff as LucideIcon;

107
infra/marimo/theme/scripts/apply-overlay.sh Executable file
View File

@@ -0,0 +1,107 @@
#!/bin/sh
# Apply fhirworx overlay onto a vanilla marimo source tree.
# Idempotent: safe to run multiple times during iterative development.
#
# Invoked by the Dockerfile after `git clone`, before `pnpm install`.
# Run from the marimo source root (same dir as frontend/ and pyproject.toml).
set -eu
SRC="${1:?usage: apply-overlay.sh <marimo-source-root> <overlay-root>}"
OVR="${2:?}"
echo "== fhirworx overlay → $SRC"
# 1. Drop new files in place (lucide-shim, fhirworx.css, asset overrides).
cp -R "$OVR/frontend/." "$SRC/frontend/"
# 2. Add @tabler/icons-react to frontend dependencies via jq (idempotent).
# Pinned to a known-good major that matches our shim's export names.
jq '.dependencies["@tabler/icons-react"] = "^3.26.0"' \
"$SRC/frontend/package.json" > "$SRC/frontend/package.json.new"
mv "$SRC/frontend/package.json.new" "$SRC/frontend/package.json"
# 3. Rewrite pnpm-lock if it exists so pnpm install doesn't error on drift.
rm -f "$SRC/frontend/pnpm-lock.yaml" "$SRC/pnpm-lock.yaml"
# 4. Inject our CSS import into globals.css.
# `@import` must precede all other statements (postcss/CSS spec), so
# append-at-EOF would get rejected with a warning + dropped. Instead,
# insert our @import right after the last existing top-level @import.
if ! grep -q 'fhirworx.css' "$SRC/frontend/src/css/globals.css"; then
python3 - "$SRC/frontend/src/css/globals.css" <<'PY'
import sys, re
p = sys.argv[1]
s = open(p).read()
# Find the index right after the last top-level @import line.
matches = list(re.finditer(r'^@import[^;]*;\s*\n', s, flags=re.M))
if not matches:
sys.exit(f"overlay: no existing @import in {p}")
pos = matches[-1].end()
insert = '@import "./fhirworx.css";\n'
open(p, "w").write(s[:pos] + insert + s[pos:])
PY
fi
# 4b. Strip the "Resources" section from the home page.
# Touchless landing page: no upstream documentation links, just the
# user's notebooks + workspace.
HOME="$SRC/frontend/src/components/pages/home-page.tsx"
if [ -f "$HOME" ] && grep -q '<ResourceLinks />' "$HOME"; then
python3 - "$HOME" <<'PY'
import sys, re
p = sys.argv[1]
s = open(p).read()
# Remove the JSX element render.
s = re.sub(r'^\s*<ResourceLinks />\s*\n', '', s, flags=re.M)
# Remove it from the imports so TS doesn't complain about unused symbols.
s = re.sub(r'(\bResourceLinks,\s*)', '', s)
# Relabel the logo so it doesn't say "marimo".
s = s.replace('alt="marimo logo"', 'alt="fhirworx"')
open(p, 'w').write(s)
PY
fi
# 5. Inject the lucide-react → shim alias into vite.config.mts.
# Match the `resolve: {` block and insert an `alias` entry right after.
if ! grep -q 'lucide-shim' "$SRC/frontend/vite.config.mts"; then
python3 - "$SRC/frontend/vite.config.mts" <<'PY'
import sys, re
path = sys.argv[1]
s = open(path).read()
# Ensure node:path + node:url are imported for the alias resolution.
if "from \"node:path\"" not in s and "from 'node:path'" not in s:
s = 'import { dirname, resolve as pathResolve } from "node:path";\n' + \
'import { fileURLToPath } from "node:url";\n' + s
# Inject alias immediately after `resolve: {`. fileURLToPath() keeps the
# path Windows-safe and anchored to this config file, regardless of where
# the build is invoked from.
inject_alias = (
' alias: {\n'
' "lucide-react": pathResolve(\n'
' dirname(fileURLToPath(import.meta.url)),\n'
' "src/lucide-shim.tsx",\n'
' ),\n'
' },\n'
)
s2, n = re.subn(r'(resolve:\s*\{\n)', r'\1' + inject_alias, s, count=1)
if n != 1:
sys.exit(f"overlay: could not find 'resolve: {{' in {path}")
# Raise the build target to esnext. Default is es2020+old-browsers, and
# vite-plugin-top-level-await's esbuild pass can't downlevel Tabler's
# bundled destructuring patterns to that set. All currently-supported
# evergreen browsers handle esnext output fine; no need to constrain.
inject_target = (
' target: "esnext",\n'
)
s3, n2 = re.subn(r'(build:\s*\{\n)', r'\1' + inject_target, s2, count=1)
if n2 != 1:
sys.exit(f"overlay: could not find 'build: {{' in {path}")
open(path, "w").write(s3)
PY
fi
echo "== overlay applied"

331
infra/marimo/theme/scripts/gen-lucide-shim.py Normal file
View File

@@ -0,0 +1,331 @@
#!/usr/bin/env python3
"""Regenerate `theme/frontend/src/lucide-shim.tsx`.
Scans a marimo source tree for every `import {…} from 'lucide-react'`
and emits a TSX shim that re-exports Tabler icons under each Lucide
name. Runs outside the Docker build (the shim is committed), so you
only rerun this when bumping the pinned marimo version or when an
upstream change introduces new lucide icons.
Usage:
python3 gen-lucide-shim.py [MARIMO_SRC_ROOT]
Default source root is `infra/marimo/src` relative to the stack repo root.
"""
from __future__ import annotations
import os
import re
import sys
from pathlib import Path
# Lucide-name → Tabler export name. Populated with the divergent cases;
# straight-rename names are handled by `transform()` below.
SPECIAL: dict[str, str] = {
"Play": "IconPlayerPlay",
"PlayIcon": "IconPlayerPlay",
"PlayCircleIcon": "IconPlayerPlay",
"PlaySquareIcon": "IconPlayerPlay",
"CirclePlayIcon": "IconPlayerPlay",
"Pause": "IconPlayerPause",
"StopCircleIcon": "IconPlayerStop",
"SkipForwardIcon": "IconPlayerSkipForward",
"FastForwardIcon": "IconPlayerTrackNext",
"Loader2": "IconLoader2",
"Loader2Icon": "IconLoader2",
"LoaderCircle": "IconLoader",
"TriangleAlert": "IconAlertTriangle",
"ChartColumn": "IconChartBar",
"ChartColumnIcon": "IconChartBar",
"ChartColumnStacked": "IconChartBarOff",
"ChartNoAxesColumn": "IconChartBar",
"HatGlasses": "IconMoodHappy",
"RulerDimensionLine": "IconRuler",
"FunnelPlusIcon": "IconFilterPlus",
"FilterX": "IconFilterOff",
"AreaChartIcon": "IconChartArea",
"BarChartIcon": "IconChartBar",
"BarChart2Icon": "IconChartBar",
"BarChartBigIcon": "IconChartBar",
"LineChartIcon": "IconChartLine",
"PieChartIcon": "IconChartPie",
"ChartPieIcon": "IconChartPie",
"ChartScatterIcon": "IconChartDots",
"ChartSplineIcon": "IconChartLine",
"Grid3x3Icon": "IconGrid3x3",
"Columns2Icon": "IconColumns2",
"Edit3Icon": "IconEdit",
"Code2Icon": "IconCode",
"FileAudio2Icon": "IconFileMusic",
"FileAudioIcon": "IconFileMusic",
"FileVideoCameraIcon": "IconFileTypography",
"SquareMIcon": "IconSquareLetterM",
"SquareCodeIcon": "IconCode",
"SquareDashedBottomCodeIcon": "IconSquareDashed",
"CircleHelpIcon": "IconHelpCircle",
"HelpCircleIcon": "IconHelpCircle",
"MessageCircleQuestionIcon": "IconMessageCircleQuestion",
"AlertOctagonIcon": "IconAlertOctagon",
"CheckCircle2Icon": "IconCircleCheck",
"CheckCircleIcon": "IconCircleCheck",
"CheckSquareIcon": "IconSquareCheck",
"XCircle": "IconCircleX",
"XCircleIcon": "IconCircleX",
"DatabaseZap": "IconDatabase",
"DatabaseZapIcon": "IconDatabase",
"ArrowDownWideNarrowIcon": "IconArrowsSort",
"ArrowUpWideNarrowIcon": "IconArrowsSort",
"ArrowUpNarrowWideIcon": "IconArrowsSort",
"ChevronsUpDown": "IconChevronUpDown",
"ChevronsUpDownIcon": "IconChevronUpDown",
"ChevronsDownUpIcon": "IconChevronDownUp",
"CircleEllipsis": "IconDotsCircleHorizontal",
"MousePointerSquareDashedIcon": "IconPointerShare",
"SquareMousePointerIcon": "IconPointer",
"SquareArrowOutUpRightIcon": "IconExternalLink",
"SquareFunction": "IconMathFunction",
"SquareFunctionIcon": "IconMathFunction",
"FunctionSquareIcon": "IconMathFunction",
"SquareEqualIcon": "IconEqual",
"Share2Icon": "IconShare",
"Undo2Icon": "IconArrowBackUp",
"Trash2Icon": "IconTrash",
"BetweenHorizontalStartIcon": "IconLayoutRows",
"BugPlayIcon": "IconBug",
"FolderArchiveIcon": "IconFolder",
"FolderCog2": "IconFolderCog",
"TerminalSquareIcon": "IconTerminal2",
"HardDriveDownloadIcon":"IconDeviceSdCard",
"DownloadCloudIcon": "IconCloudDownload",
"HardDrive": "IconDeviceSdCard",
"HardDriveIcon": "IconDeviceSdCard",
"MemoryStickIcon": "IconDeviceSdCard",
"MicrochipIcon": "IconCpu",
"BrickWallIcon": "IconWall",
"ViewIcon": "IconEye",
"BotMessageSquareIcon": "IconRobot",
"BotIcon": "IconRobot",
"TextCursorInputIcon": "IconCursorText",
"TextSelectionIcon": "IconTextCaption",
"TextSearchIcon": "IconZoomCode",
"TextIcon": "IconTextCaption",
"TypeIcon": "IconTypography",
"CaseSensitiveIcon": "IconLetterCase",
"WholeWordIcon": "IconAlphabetLatin",
"WrapTextIcon": "IconTextWrap",
"SendHorizontalIcon": "IconSend",
"NotebookText": "IconNotebook",
"NotebookPenIcon": "IconNotebook",
"BookMarkedIcon": "IconBookmark",
"BookTextIcon": "IconBook",
"KeyRoundIcon": "IconKey",
"PartyPopperIcon": "IconConfetti",
"PaintRollerIcon": "IconBrush",
"PackageCheckIcon": "IconPackage",
"PackageXIcon": "IconPackageOff",
"YoutubeIcon": "IconBrandYoutube",
"GithubIcon": "IconBrandGithub",
"Info": "IconInfoCircle",
"InfoIcon": "IconInfoCircle",
"CopyMinusIcon": "IconCopy",
"CopySlashIcon": "IconCopyOff",
"SearchCheck": "IconZoomCheck",
"ChevronDownCircleIcon": "IconCircleChevronDown",
"ChevronRightCircleIcon": "IconCircleChevronRight",
"MoreHorizontal": "IconDots",
"MoreHorizontalIcon": "IconDots",
"MoreVerticalIcon": "IconDotsVertical",
"RefreshCcw": "IconRefresh",
"RefreshCcwIcon": "IconRefresh",
"RefreshCwIcon": "IconRefresh",
"RotateCcwIcon": "IconRotate",
"RotateCwIcon": "IconRotateClockwise",
"AtSignIcon": "IconAt",
"SigmaIcon": "IconSum",
"BaselineIcon": "IconBaselineDensitySmall",
"AlignCenterVerticalIcon": "IconAlignCenter",
"AlignStartVerticalIcon": "IconAlignLeft",
"AlignEndVerticalIcon": "IconAlignRight",
"AlignHorizontalSpaceAroundIcon": "IconAlignJustified",
"AlignJustifyIcon": "IconAlignJustified",
"ZapIcon": "IconBolt",
"ZapOffIcon": "IconBoltOff",
"ListFilterIcon": "IconFilter",
"ListFilterPlusIcon": "IconFilterPlus",
"CircleHelp": "IconHelpCircle",
"CalendarClockIcon": "IconCalendarTime",
"ClipboardPasteIcon": "IconClipboardText",
"GraduationCapIcon": "IconSchool",
"ExpandIcon": "IconArrowsMaximize",
"LayersIcon": "IconStack",
"LayoutTemplateIcon": "IconLayoutBoard",
"ListOrderedIcon": "IconListNumbers",
"GroupIcon": "IconUsersGroup",
"GlobeIcon": "IconWorld",
"WorkflowIcon": "IconSitemap",
"Table2Icon": "IconTable",
"MessagesSquareIcon": "IconMessages",
"MenuIcon": "IconMenu2",
"MonitorIcon": "IconDeviceDesktop",
"PlusCircleIcon": "IconCirclePlus",
"PlusSquareIcon": "IconSquarePlus",
"FilePenIcon": "IconFilePencil",
"FilePlus2Icon": "IconFilePlus",
"FileImageIcon": "IconPhoto",
"FileJsonIcon": "IconJson",
"FileVideoIcon": "IconVideo",
"ImageIcon": "IconPhoto",
"PanelLeftIcon": "IconLayoutSidebar",
"PanelRightIcon": "IconLayoutSidebarRight",
"PowerOffIcon": "IconPower",
"PowerSquareIcon": "IconPower",
"SaveIcon": "IconDeviceFloppy",
"ShuffleIcon": "IconArrowsShuffle",
"GridIcon": "IconGrid3x3",
"CombineIcon": "IconStackPush",
"ArrowDownToLineIcon": "IconArrowBarToDown",
"ArrowRightFromLineIcon": "IconArrowBarRight",
"ArrowRightToLineIcon": "IconArrowBarToRight",
"ArrowRightSquareIcon": "IconSquareArrowRight",
"ArrowUpDownIcon": "IconArrowsUpDown",
"ArrowUpToLineIcon": "IconArrowBarToUp",
"SquareStack": "IconStack",
"WrenchIcon": "IconTool",
"SquareIcon": "IconSquare",
"CornerLeftUp": "IconCornerLeftUp",
"GripHorizontal": "IconGripHorizontal",
"GripHorizontalIcon": "IconGripHorizontal",
"GripVerticalIcon": "IconGripVertical",
"CrosshairIcon": "IconCrosshair",
"CommandIcon": "IconCommand",
"PinIcon": "IconPin",
"PinOffIcon": "IconPinnedOff",
"DiamondPlusIcon": "IconDiamondPlus",
"BinaryIcon": "IconBinary",
"BoxIcon": "IconBox",
"BracesIcon": "IconBraces",
"BracketsIcon": "IconBrackets",
"BrainIcon": "IconBrain",
"ActivityIcon": "IconActivity",
"Cog": "IconSettings",
"CogIcon": "IconSettings",
# Names that don't exist as-is in @tabler/icons-react v3.41.
"BookOpenIcon": "IconBook",
"BookPlusIcon": "IconBook",
"ChevronsUpDown": "IconSelector",
"ChevronsUpDownIcon": "IconSelector",
"ChevronsDownUpIcon": "IconSelector",
"CurlyBracesIcon": "IconBraces",
"DiamondPlusIcon": "IconDiamond",
"EllipsisIcon": "IconDots",
"FlaskConicalIcon": "IconFlask",
"FolderTreeIcon": "IconFolders",
"OrbitIcon": "IconCircleDashed",
"ScrollIcon": "IconArticle",
"ScrollTextIcon": "IconArticle",
}
_FALLBACK = "IconQuestionMark"
def load_tabler_exports() -> set[str] | None:
"""Return the set of Tabler icon names if the authoritative list is on
disk (generated by probing @tabler/icons-react in a container). If it's
missing, return None — generator emits without validation and build-time
errors surface the unknowns."""
cand = Path(__file__).resolve().parent / "tabler-icons.txt"
if not cand.is_file():
cand = Path("/tmp/tabler-icons.txt")
if not cand.is_file():
return None
return {line.strip() for line in cand.read_text().splitlines() if line.strip()}
_TABLER: set[str] | None = load_tabler_exports()
def transform(name: str) -> str:
"""Map a Lucide export name to its Tabler equivalent, falling back to
IconQuestionMark when the intended target doesn't actually exist in the
installed @tabler/icons-react version."""
if name in SPECIAL:
target = SPECIAL[name]
else:
base = name[:-4] if name.endswith("Icon") else name
target = "Icon" + base
if _TABLER is not None and target not in _TABLER:
return _FALLBACK
return target
def extract(src_root: Path) -> set[str]:
"""Collect every name marimo imports or re-exports from lucide-react."""
names: set[str] = set()
# Covers both `import { X, Y as Z } from "lucide-react"` and
# `export { X as A } from "lucide-react"` (0.23.1 added re-exports).
pattern = re.compile(
r'(?:import|export)\s*(?:type\s*)?\{([^}]+)\}\s*from\s*["\']lucide-react["\']'
)
for root, _, files in os.walk(src_root):
for fn in files:
if not fn.endswith((".ts", ".tsx")):
continue
text = (Path(root) / fn).read_text()
for match in pattern.finditer(text):
for token in match.group(1).split(","):
token = token.strip().removeprefix("type ").strip()
token = token.split(" as ")[0].strip()
if token and re.match(r"^[A-Z]", token):
names.add(token)
return names
def render(names: set[str]) -> str:
TYPE_ONLY = {"LucideIcon", "LucideProps"}
sorted_names = sorted(n for n in names if n not in TYPE_ONLY)
# De-dup tabler names (many lucide aliases resolve to the same icon).
tabler_set = sorted({transform(n) for n in sorted_names})
header = [
"// Auto-generated: Tabler Icons as a drop-in for lucide-react.",
"// Build-time replacement; do not edit by hand — see gen-lucide-shim.py",
"import type { ComponentType, SVGProps } from 'react';",
"import {",
]
header.extend(f" {name}," for name in tabler_set)
header.extend([
" type IconProps,",
"} from '@tabler/icons-react';",
"",
"// Prop shapes mirror lucide's public types so Omit<LucideProps, 'ref'>",
"// and similar patterns in upstream marimo code keep typechecking.",
"export type LucideProps = SVGProps<SVGSVGElement> & {",
" size?: number | string;",
" absoluteStrokeWidth?: boolean;",
"};",
"export type LucideIcon = ComponentType<LucideProps & IconProps>;",
"",
])
body = [
f"export const {n}: LucideIcon = {transform(n)} as LucideIcon;"
for n in sorted_names
]
return "\n".join(header + body) + "\n"
def main() -> None:
default = Path(__file__).resolve().parents[2] / "src"
src = Path(sys.argv[1]) if len(sys.argv) > 1 else default
if not src.is_dir():
raise SystemExit(f"marimo source not found: {src}")
names = extract(src / "frontend" / "src")
out = Path(__file__).resolve().parents[1] / "frontend" / "src" / "lucide-shim.tsx"
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(render(names))
print(f"wrote {out} ({len(names)} names)")
if __name__ == "__main__":
main()

6149
infra/marimo/theme/scripts/tabler-icons.txt Normal file
View File

File diff suppressed because it is too large Load Diff

34
infra/traefik/dynamic/services.yml
View File

@@ -1,24 +1,24 @@
{{- $domain := env "DOMAIN" | default "fhirworx.io" -}}
{{- $reef := dict
"dashboard" (dict "port" "80" "theme" true "extra_hosts" (list $domain) "mw" "secure-headers")
"docs" (dict "port" "80" "theme" true "mw" "gitea-sso,secure-headers")
"gitea" (dict "port" "3000" "subdomain" "git" "theme" true "mw" "secure-headers")
"woodpecker-server" (dict "port" "8000" "theme" true "subdomain" "ci" "mw" "gitea-sso,secure-headers")
"notebooks" (dict "port" "2718" "theme" true "mw" "gitea-sso,secure-headers")
"zotero" (dict "port" "8080" "theme" true "mw" "gitea-sso,secure-headers")
"docs" (dict "port" "80" "theme" true "mw" "git-sso,secure-headers")
"git" (dict "port" "3000" "theme" false "mw" "secure-headers")
"woodpecker-server" (dict "port" "8000" "theme" true "subdomain" "ci" "mw" "git-sso,secure-headers")
"notebooks" (dict "port" "2718" "theme" true "mw" "git-sso,secure-headers")
"zotero" (dict "port" "8080" "theme" true "mw" "git-sso,secure-headers")
"webdav" (dict "port" "8080" "theme" false "mw" "secure-headers")
"api" (dict "port" "8000" "theme" false "mw" "secure-headers")
"nessie" (dict "port" "19120" "theme" false "mw" "gitea-sso,infra-headers")
"trino" (dict "port" "8080" "theme" true "mw" "gitea-sso,infra-headers")
"polaris" (dict "port" "8181" "theme" false "mw" "gitea-sso,infra-headers")
"grafana" (dict "port" "3000" "theme" true "mw" "gitea-sso,secure-headers")
"prometheus" (dict "port" "9090" "theme" true "mw" "gitea-sso,infra-headers")
"jaeger" (dict "port" "16686" "theme" true "mw" "gitea-sso,infra-headers")
"loki" (dict "port" "3100" "theme" false "mw" "gitea-sso,infra-headers")
"nessie" (dict "port" "19120" "theme" false "mw" "git-sso,infra-headers")
"trino" (dict "port" "8080" "theme" true "mw" "git-sso,infra-headers")
"polaris" (dict "port" "8181" "theme" false "mw" "git-sso,infra-headers")
"grafana" (dict "port" "3000" "theme" true "mw" "git-sso,secure-headers")
"prometheus" (dict "port" "9090" "theme" true "mw" "git-sso,infra-headers")
"jaeger" (dict "port" "16686" "theme" true "mw" "git-sso,infra-headers")
"loki" (dict "port" "3100" "theme" false "mw" "git-sso,infra-headers")
-}}
{{- $multi := dict
"rustfs-api" (dict "container" "rustfs" "port" "9000" "subdomain" "s3" "theme" false "mw" "gitea-sso,infra-headers")
"rustfs-console" (dict "container" "rustfs" "port" "9001" "subdomain" "s3console" "theme" true "mw" "gitea-sso,infra-headers")
"rustfs-api" (dict "container" "rustfs" "port" "9000" "subdomain" "s3" "theme" false "mw" "git-sso,infra-headers")
"rustfs-console" (dict "container" "rustfs" "port" "9001" "subdomain" "s3console" "theme" true "mw" "git-sso,infra-headers")
-}}
http:
middlewares:
@@ -29,7 +29,7 @@ http:
contentTypeNosniff: true
# SSO: auth-handler (nginx) wraps oauth2-proxy to convert 401 → 302.
# Same logic as corwins.media: auth_request + error_page 401 = @signin.
gitea-sso:
git-sso:
forwardAuth:
address: "http://auth-handler:4181"
trustForwardHeader: true
@@ -129,7 +129,7 @@ http:
- web
middlewares:
- inject-fhirworx
- gitea-sso
- git-sso
- infra-headers
traefik-dashboard-tls:
rule: "Host(`traefik.{{ $domain }}`)"
@@ -139,7 +139,7 @@ http:
tls: {}
middlewares:
- inject-fhirworx
- gitea-sso
- git-sso
- infra-headers
services:
{{- range $name, $svc := $reef }}

1
notebooks/_template.py
View File

@@ -15,6 +15,7 @@ def _():
def _():
import altair as alt
import polars as pl
from conf import connect
connect.theme()

6
notebooks/acodb_explorer.py
View File

@@ -254,7 +254,11 @@ def _(PALETTE, alt, mo, q):
.encode(
x=alt.X("patients:Q", title="Patients"),
y=alt.Y("condition:N", sort="-x", title=None),
color=alt.Color("patients:Q", scale=alt.Scale(range=[PALETTE[1] + "33", PALETTE[1]]), legend=None),
color=alt.Color(
"patients:Q",
scale=alt.Scale(range=[PALETTE[1] + "33", PALETTE[1]]),
legend=None,
),
tooltip=["condition:N", "patients:Q"],
)
.properties(

42
notebooks/api_explorer.py
View File

@@ -44,10 +44,12 @@ def _(client, mo):
_rows = []
for _svc in health["services"]:
_icon = "\u2705" if _svc["status"] == "ok" else "\u26a0\ufe0f"
_rows.append(f"| {_icon} | {_svc['name']} | {_svc['status']} | {_svc.get('detail', '')} |")
_rows.append(
f"| {_icon} | {_svc['name']} | {_svc['status']} | {_svc.get('detail', '')} |"
)
mo.md(f"""
**Status:** {health['status']} | **Version:** {health['version']}
**Status:** {health["status"]} | **Version:** {health["version"]}
| | Service | Status | Detail |
|---|---------|--------|--------|
@@ -93,9 +95,9 @@ def _(client, mo, pipelines):
_outputs_list = "\n".join(f"- `{o}`" for o in detail.get("outputs", []))
mo.md(f"""
### Pipeline Detail: `{detail['name']}`
### Pipeline Detail: `{detail["name"]}`
**Steps:** {detail['steps']} | **Inputs:** {len(detail.get('inputs', []))} | **Outputs:** {len(detail.get('outputs', []))}
**Steps:** {detail["steps"]} | **Inputs:** {len(detail.get("inputs", []))} | **Outputs:** {len(detail.get("outputs", []))}
**Inputs:**
{_inputs_list}
@@ -110,7 +112,9 @@ def _(client, mo, pipelines):
def _(client, mo):
_r = client.get("/pipelines/nonexistent_xyz")
pipeline_404 = _r.status_code == 404
mo.md(f"**GET /pipelines/nonexistent_xyz** \u2192 `{_r.status_code}` {'PASS' if pipeline_404 else 'FAIL'}")
mo.md(
f"**GET /pipelines/nonexistent_xyz** \u2192 `{_r.status_code}` {'PASS' if pipeline_404 else 'FAIL'}"
)
return (pipeline_404,)
@@ -150,9 +154,9 @@ def _(client, lineage, mo):
_r = client.get(f"/lineage/{_test_table}")
_tl = _r.json()
_content = f"""
### Table Lineage: `{_tl['table']}`
### Table Lineage: `{_tl["table"]}`
**Inputs:** {', '.join(f'`{i}`' for i in _tl['inputs'])}
**Inputs:** {", ".join(f"`{i}`" for i in _tl["inputs"])}
"""
else:
_content = "No table with dependencies found"
@@ -170,7 +174,7 @@ def _(client, mo):
mo.md(f"""
### Mermaid Export
**Format:** {mermaid_data['format']} | **Lines:** {len(_lines)}
**Format:** {mermaid_data["format"]} | **Lines:** {len(_lines)}
```mermaid
{chr(10).join(_lines[:15])}
@@ -189,7 +193,7 @@ def _(client, mo):
mo.md(f"""
### DOT Export
**Format:** {dot_data['format']} | **Lines:** {len(_lines)}
**Format:** {dot_data["format"]} | **Lines:** {len(_lines)}
```dot
{chr(10).join(_lines[:10])}
@@ -217,7 +221,9 @@ def _(client, mo):
_item_rows = []
for _item in bib_items[:10]:
_item_rows.append(f"| {_item['key']} | {_item['title'][:50]} | {_item['item_type']} |")
_item_rows.append(
f"| {_item['key']} | {_item['title'][:50]} | {_item['item_type']} |"
)
_tag_rows = []
for _tag in bib_tags[:10]:
@@ -253,7 +259,9 @@ def _(mo):
def _(client, mo):
_r = client.get("/schema/nonexistent_table")
missing_ok = _r.status_code == 404
mo.md(f"**GET /schema/nonexistent_table** \u2192 `{_r.status_code}` {'PASS' if missing_ok else 'FAIL'}")
mo.md(
f"**GET /schema/nonexistent_table** \u2192 `{_r.status_code}` {'PASS' if missing_ok else 'FAIL'}"
)
return (missing_ok,)
@@ -276,8 +284,8 @@ def _(client, mo):
mo.md(f"""
| Test | Status | Result |
|------|--------|--------|
| Run requires auth | `{_r1.status_code}` | {'PASS' if auth_required else 'FAIL'} |
| Wrong secret rejected | `{_r2.status_code}` | {'PASS' if wrong_secret else 'FAIL'} |
| Run requires auth | `{_r1.status_code}` | {"PASS" if auth_required else "FAIL"} |
| Wrong secret rejected | `{_r2.status_code}` | {"PASS" if wrong_secret else "FAIL"} |
""")
return auth_required, wrong_secret
@@ -305,10 +313,14 @@ def _(
tests = {
"GET /health": health["status"] in ("ok", "degraded"),
"GET /pipelines": len(pipelines) > 0,
"GET /pipelines/{name}": client.get(f"/pipelines/{pipelines[0]['name']}").status_code == 200,
"GET /pipelines/{name}": client.get(
f"/pipelines/{pipelines[0]['name']}"
).status_code
== 200,
"GET /pipelines/404": pipeline_404,
"GET /lineage": len(lineage["tables"]) > 0,
"GET /lineage/export/mermaid": client.get("/lineage/export/mermaid").status_code == 200,
"GET /lineage/export/mermaid": client.get("/lineage/export/mermaid").status_code
== 200,
"GET /lineage/export/dot": client.get("/lineage/export/dot").status_code == 200,
"GET /bib/items": client.get("/bib/items").status_code == 200,
"GET /bib/tags": client.get("/bib/tags").status_code == 200,

188
notebooks/bib_explorer.py
View File

@@ -29,11 +29,11 @@ def _():
@app.cell(hide_code=True)
def _(mo):
from bib.client import COLLECTIONS
from bib.format import format_bibliography, format_citation
from bib.item import Download, Item, Manual, Regulation, Rule, Source
from bib.store import Store
from bib.tag import Tag, filter_tags
from conf import connect, path as _conf_path
from bib.format import format_citation
from bib.item import Download, Manual, Regulation, Rule, Source
from bib.tag import Tag
from conf import connect
from conf import path as _conf_path
store = connect.bib()
ZOTERO_DB = _conf_path("db.zotero")
@@ -98,7 +98,9 @@ def _(ZOTERO_DB, mo, store):
_count = store._con().execute("SELECT COUNT(*) FROM items").fetchone()[0]
_show_migrate = _count == 0 and ZOTERO_DB.exists()
migrate_btn = mo.ui.run_button(label="Migrate from Zotero") if _show_migrate else None
migrate_btn = (
mo.ui.run_button(label="Migrate from Zotero") if _show_migrate else None
)
migrate_btn
return (migrate_btn,)
@@ -127,7 +129,9 @@ def _(ZOTERO_DB, migrate_btn, mo, store):
if _zc["parentCollectionID"]:
_pk = _col_id_to_key.get(_zc["parentCollectionID"])
if _pk:
_pr = bcon.execute("SELECT id FROM collections WHERE key = ?", (_pk,)).fetchone()
_pr = bcon.execute(
"SELECT id FROM collections WHERE key = ?", (_pk,)
).fetchone()
if _pr:
_pid = _pr["id"]
bcon.execute(
@@ -172,23 +176,38 @@ def _(ZOTERO_DB, migrate_btn, mo, store):
_it = "source"
# Title
_title = _fields.get("nameOfAct", "") if _it in ("rule", "regulation") else _fields.get("title", "")
_date = _fields.get("dateEnacted", "") if _it in ("rule", "regulation") else _fields.get("date", "")
_title = (
_fields.get("nameOfAct", "")
if _it in ("rule", "regulation")
else _fields.get("title", "")
)
_date = (
_fields.get("dateEnacted", "")
if _it in ("rule", "regulation")
else _fields.get("date", "")
)
bcon.execute(
"""INSERT OR IGNORE INTO items
(key, item_type, title, url, date_published, access_date, abstract, institution, extra, extra_json)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(
_zi["key"], _it, _title,
_fields.get("url", ""), _date,
_fields.get("accessDate", ""), _fields.get("abstractNote", ""),
_zi["key"],
_it,
_title,
_fields.get("url", ""),
_date,
_fields.get("accessDate", ""),
_fields.get("abstractNote", ""),
_fields.get("institution", "") or _fields.get("publisher", ""),
_fields.get("extra", ""), "{}",
_fields.get("extra", ""),
"{}",
),
)
_bib_row = bcon.execute("SELECT id FROM items WHERE key = ?", (_zi["key"],)).fetchone()
_bib_row = bcon.execute(
"SELECT id FROM items WHERE key = ?", (_zi["key"],)
).fetchone()
if _bib_row is None:
continue
_bid = _bib_row["id"]
@@ -200,14 +219,19 @@ def _(ZOTERO_DB, migrate_btn, mo, store):
(_zi["itemID"],),
).fetchall():
_tid = store._ensure_tag(_tr["name"])
bcon.execute("INSERT OR IGNORE INTO item_tags (item_id, tag_id) VALUES (?, ?)", (_bid, _tid))
bcon.execute(
"INSERT OR IGNORE INTO item_tags (item_id, tag_id) VALUES (?, ?)",
(_bid, _tid),
)
# Collection membership
for _cr in zcon.execute(
"SELECT c.key FROM collectionItems ci JOIN collections c ON ci.collectionID = c.collectionID WHERE ci.itemID = ?",
(_zi["itemID"],),
).fetchall():
_cid_row = bcon.execute("SELECT id FROM collections WHERE key = ?", (_cr["key"],)).fetchone()
_cid_row = bcon.execute(
"SELECT id FROM collections WHERE key = ?", (_cr["key"],)
).fetchone()
if _cid_row:
bcon.execute(
"INSERT OR IGNORE INTO collection_items (collection_id, item_id) VALUES (?, ?)",
@@ -242,10 +266,14 @@ def _(mo, store):
_stats = {}
_stats["Items"] = _con.execute("SELECT COUNT(*) FROM items").fetchone()[0]
_stats["Collections"] = _con.execute("SELECT COUNT(*) FROM collections").fetchone()[0]
_stats["Collections"] = _con.execute("SELECT COUNT(*) FROM collections").fetchone()[
0
]
_stats["Tags"] = _con.execute("SELECT COUNT(*) FROM tags").fetchone()[0]
_stats["Creators"] = _con.execute("SELECT COUNT(*) FROM creators").fetchone()[0]
_stats["Attachments"] = _con.execute("SELECT COUNT(*) FROM attachments").fetchone()[0]
_stats["Attachments"] = _con.execute("SELECT COUNT(*) FROM attachments").fetchone()[
0
]
_stats["Notes"] = _con.execute("SELECT COUNT(*) FROM notes").fetchone()[0]
_stat_table = "\n".join(f"| {k} | {v:,} |" for k, v in _stats.items())
@@ -269,13 +297,21 @@ def _(mo):
def _(mo, store):
import polars as pl
_types_rows = store._con().execute(
"""SELECT item_type, COUNT(*) as count
_types_rows = (
store._con()
.execute(
"""SELECT item_type, COUNT(*) as count
FROM items GROUP BY item_type ORDER BY count DESC"""
).fetchall()
_types_df = pl.DataFrame([dict(r) for r in _types_rows]) if _types_rows else pl.DataFrame()
)
.fetchall()
)
_types_df = (
pl.DataFrame([dict(r) for r in _types_rows]) if _types_rows else pl.DataFrame()
)
mo.ui.table(_types_df, label="Item Types") if _types_df.height > 0 else mo.md("No items yet.")
mo.ui.table(_types_df, label="Item Types") if _types_df.height > 0 else mo.md(
"No items yet."
)
return (pl,)
@@ -307,10 +343,18 @@ def _(mo, store):
_stack = [(n, 0) for n in sorted(_roots, key=lambda x: x["name"], reverse=True)]
while _stack:
_node, _depth = _stack.pop()
_tree_lines.append(f"{' ' * _depth}- **{_node['name']}** ({_node['item_count']} items)")
for _ch in sorted(_children.get(_node["key"], []), key=lambda x: x["name"], reverse=True):
_tree_lines.append(
f"{' ' * _depth}- **{_node['name']}** ({_node['item_count']} items)"
)
for _ch in sorted(
_children.get(_node["key"], []), key=lambda x: x["name"], reverse=True
):
_stack.append((_ch, _depth + 1))
mo.md("\n".join(_tree_lines) if _tree_lines else "No collections. Ensure collections above.")
mo.md(
"\n".join(_tree_lines)
if _tree_lines
else "No collections. Ensure collections above."
)
return
@@ -341,19 +385,20 @@ def _(mo):
@app.cell(hide_code=True)
def _(mo, store):
_type_opts = [""] + sorted({
r[0] for r in store._con().execute(
"SELECT DISTINCT item_type FROM items"
).fetchall()
})
_tag_opts = [""] + [
t["name"] for t in store.list_tags()
]
_col_opts = [""] + [
f"{c['name']}|{c['key']}" for c in store.list_collections()
]
_type_opts = [""] + sorted(
{
r[0]
for r in store._con()
.execute("SELECT DISTINCT item_type FROM items")
.fetchall()
}
)
_tag_opts = [""] + [t["name"] for t in store.list_tags()]
_col_opts = [""] + [f"{c['name']}|{c['key']}" for c in store.list_collections()]
search_input = mo.ui.text(placeholder="Search titles and abstracts...", label="Search", full_width=True)
search_input = mo.ui.text(
placeholder="Search titles and abstracts...", label="Search", full_width=True
)
type_filter = mo.ui.dropdown(options=_type_opts, value="", label="Type")
tag_filter = mo.ui.dropdown(options=_tag_opts, value="", label="Tag")
col_filter = mo.ui.dropdown(options=_col_opts, value="", label="Collection")
@@ -415,6 +460,7 @@ def _(items_table, mo):
if items_table is not None:
_val = items_table.value
import polars as _pl
if isinstance(_val, _pl.DataFrame) and _val.height > 0:
_default_key = _val["key"][0]
elif isinstance(_val, list) and len(_val) > 0:
@@ -470,7 +516,9 @@ def _(format_citation, key_input, mo, store):
# Abstract
if _item.abstract:
_sections.append(f"\n**Abstract**: {_item.abstract[:300]}{'...' if len(_item.abstract) > 300 else ''}")
_sections.append(
f"\n**Abstract**: {_item.abstract[:300]}{'...' if len(_item.abstract) > 300 else ''}"
)
# Citation preview
_cite_apa = format_citation(_item, style="apa")
@@ -481,22 +529,32 @@ def _(format_citation, key_input, mo, store):
_sections.append(f"\n**Bluebook Citation**:\n> {_cite_bb}")
# Attachments
_atts = store._con().execute(
"""SELECT key, filename, content_type FROM attachments
_atts = (
store._con()
.execute(
"""SELECT key, filename, content_type FROM attachments
WHERE item_id = (SELECT id FROM items WHERE key = ?)""",
(_key,),
).fetchall()
(_key,),
)
.fetchall()
)
if _atts:
_sections.append("\n**Attachments**")
for _a in _atts:
_sections.append(f"- `{_a['filename']}` ({_a['content_type'] or 'unknown'})")
_sections.append(
f"- `{_a['filename']}` ({_a['content_type'] or 'unknown'})"
)
# Notes
_notes = store._con().execute(
"""SELECT title, content FROM notes
_notes = (
store._con()
.execute(
"""SELECT title, content FROM notes
WHERE item_id = (SELECT id FROM items WHERE key = ?)""",
(_key,),
).fetchall()
(_key,),
)
.fetchall()
)
if _notes:
_sections.append("\n**Notes**")
for _n in _notes:
@@ -527,14 +585,20 @@ def _(mo):
)
new_title = mo.ui.text(placeholder="Title", label="Title", full_width=True)
new_url = mo.ui.text(placeholder="URL", label="URL", full_width=True)
new_tags_input = mo.ui.text(placeholder="module:pfs, year:2026", label="Tags (comma-separated)", full_width=True)
new_tags_input = mo.ui.text(
placeholder="module:pfs, year:2026",
label="Tags (comma-separated)",
full_width=True,
)
create_btn = mo.ui.run_button(label="Create Item")
mo.vstack([
mo.hstack([new_type, new_title], widths=[1, 3]),
new_url,
mo.hstack([new_tags_input, create_btn], widths=[3, 1]),
])
mo.vstack(
[
mo.hstack([new_type, new_title], widths=[1, 3]),
new_url,
mo.hstack([new_tags_input, create_btn], widths=[3, 1]),
]
)
return create_btn, new_tags_input, new_title, new_type, new_url
@@ -646,7 +710,7 @@ def _(
_msg = f"""
Saved as `{_key}`: **{_item.title}**
Tags: {', '.join(f'`{t}`' for t in _item.tags)}
Tags: {", ".join(f"`{t}`" for t in _item.tags)}
> {_cite}
"""
@@ -670,11 +734,14 @@ def _(mo):
@app.cell(hide_code=True)
def _(mo, store):
_bib_tag_opts = [""] + [t["name"] for t in store.list_tags()]
_bib_type_opts = [""] + sorted({
r[0] for r in store._con().execute(
"SELECT DISTINCT item_type FROM items"
).fetchall()
})
_bib_type_opts = [""] + sorted(
{
r[0]
for r in store._con()
.execute("SELECT DISTINCT item_type FROM items")
.fetchall()
}
)
bib_tag = mo.ui.dropdown(options=_bib_tag_opts, value="", label="Filter by tag")
bib_type = mo.ui.dropdown(options=_bib_type_opts, value="", label="Filter by type")
@@ -730,6 +797,7 @@ def _(mo):
@app.cell(hide_code=True)
def _(mo):
import os
import s3fs
_endpoint = os.environ.get("S3_ENDPOINT", "http://rustfs:9000")

556
notebooks/cms_quality_measures.py
View File

@@ -1,6 +1,6 @@
import marimo
__generated_with = "0.20.2"
__generated_with = "0.21.1"
app = marimo.App(width="full")
@@ -41,6 +41,7 @@ def _():
import textwrap
import polars as pl
from conf import connect
con = connect.duckdb()
@@ -1037,11 +1038,13 @@ def _(con, ex, inspect, mo, pl):
else:
nearest = min(avail, key=lambda y: abs(y - spec_year))
_df = _df.filter(pl.col("performance_year") == nearest)
_fallback_log.append({
"table": table_ref.split(".")[-1],
"requested_year": spec_year,
"actual_year": nearest,
})
_fallback_log.append(
{
"table": table_ref.split(".")[-1],
"requested_year": spec_year,
"actual_year": nearest,
}
)
_year_cache[_key] = _df
return _df
@@ -1074,33 +1077,66 @@ def _(con, ex, inspect, mo, pl):
_LOOKBACK_BEGIN = _date(2017, 1, 1)
_LOOKBACK_END = _date(2017, 12, 31)
_uamcc_pp = pl.DataFrame({
"measure_id": ["UAMCC"], "measure_name": ["All-Cause Unplanned Admissions for MCC"],
"nqf_id": ["2888"], "performance_year": [_PERF_YEAR],
"performance_period_begin": [_PERF_BEGIN], "performance_period_end": [_PERF_END],
"lookback_period_begin": [_LOOKBACK_BEGIN], "lookback_period_end": [_LOOKBACK_END],
})
_acr_pp = pl.DataFrame({
"measure_id": ["ACR"], "measure_name": ["Risk-Standardized All-Condition Readmission"],
"nqf_id": ["1789"], "performance_year": [_PERF_YEAR],
"performance_period_begin": [_PERF_BEGIN], "performance_period_end": [_PERF_END],
})
_hwr_pp = pl.DataFrame({
"measure_id": ["HWR"], "measure_name": ["Hospital-Wide 30-Day All-Cause Unplanned Readmission"],
"performance_year": [_PERF_YEAR],
"performance_period_begin": [_PERF_BEGIN], "performance_period_end": [_PERF_END],
})
_uamcc_pp = pl.DataFrame(
{
"measure_id": ["UAMCC"],
"measure_name": ["All-Cause Unplanned Admissions for MCC"],
"nqf_id": ["2888"],
"performance_year": [_PERF_YEAR],
"performance_period_begin": [_PERF_BEGIN],
"performance_period_end": [_PERF_END],
"lookback_period_begin": [_LOOKBACK_BEGIN],
"lookback_period_end": [_LOOKBACK_END],
}
)
_acr_pp = pl.DataFrame(
{
"measure_id": ["ACR"],
"measure_name": ["Risk-Standardized All-Condition Readmission"],
"nqf_id": ["1789"],
"performance_year": [_PERF_YEAR],
"performance_period_begin": [_PERF_BEGIN],
"performance_period_end": [_PERF_END],
}
)
_hwr_pp = pl.DataFrame(
{
"measure_id": ["HWR"],
"measure_name": ["Hospital-Wide 30-Day All-Cause Unplanned Readmission"],
"performance_year": [_PERF_YEAR],
"performance_period_begin": [_PERF_BEGIN],
"performance_period_end": [_PERF_END],
}
)
_pipelines = {
"UAMCC": {
"steps": [
("cms_quality_measures._uamcc_performance_period", ex.uamcc_performance_period),
(
"cms_quality_measures._uamcc_performance_period",
ex.uamcc_performance_period,
),
("cms_quality_measures._uamcc_int_mcc_cohort", ex.uamcc_int_mcc_cohort),
("cms_quality_measures._uamcc_int_denominator", ex.uamcc_int_denominator),
("cms_quality_measures._uamcc_int_denominator_exclusion", ex.uamcc_int_denominator_exclusion),
("cms_quality_measures._uamcc_int_planned_admission", ex.uamcc_int_planned_admission),
("cms_quality_measures._uamcc_int_outcome_exclusion", ex.uamcc_int_outcome_exclusion),
("cms_quality_measures._uamcc_int_person_time", ex.uamcc_int_person_time),
(
"cms_quality_measures._uamcc_int_denominator",
ex.uamcc_int_denominator,
),
(
"cms_quality_measures._uamcc_int_denominator_exclusion",
ex.uamcc_int_denominator_exclusion,
),
(
"cms_quality_measures._uamcc_int_planned_admission",
ex.uamcc_int_planned_admission,
),
(
"cms_quality_measures._uamcc_int_outcome_exclusion",
ex.uamcc_int_outcome_exclusion,
),
(
"cms_quality_measures._uamcc_int_person_time",
ex.uamcc_int_person_time,
),
("cms_quality_measures._uamcc_int_numerator", ex.uamcc_int_numerator),
("cms_quality_measures.uamcc_summary", ex.uamcc_summary),
],
@@ -1110,10 +1146,22 @@ def _(con, ex, inspect, mo, pl):
},
"ACR": {
"steps": [
("cms_quality_measures._acr_performance_period", ex.acr_performance_period),
("cms_quality_measures._acr_int_index_admission", ex.acr_int_index_admission),
("cms_quality_measures._acr_int_specialty_cohort", ex.acr_int_specialty_cohort),
("cms_quality_measures._acr_int_planned_readmission", ex.acr_int_planned_readmission),
(
"cms_quality_measures._acr_performance_period",
ex.acr_performance_period,
),
(
"cms_quality_measures._acr_int_index_admission",
ex.acr_int_index_admission,
),
(
"cms_quality_measures._acr_int_specialty_cohort",
ex.acr_int_specialty_cohort,
),
(
"cms_quality_measures._acr_int_planned_readmission",
ex.acr_int_planned_readmission,
),
("cms_quality_measures.acr_summary", ex.acr_summary),
],
"pp_key": "cms_quality_measures._acr_performance_period",
@@ -1122,9 +1170,15 @@ def _(con, ex, inspect, mo, pl):
},
"HWR": {
"steps": [
("cms_quality_measures._hwr_performance_period", ex.hwr_performance_period),
(
"cms_quality_measures._hwr_performance_period",
ex.hwr_performance_period,
),
("cms_quality_measures._hwr_int_denominator", ex.hwr_int_denominator),
("cms_quality_measures._hwr_int_planned_readmission", ex.hwr_int_planned_readmission),
(
"cms_quality_measures._hwr_int_planned_readmission",
ex.hwr_int_planned_readmission,
),
("cms_quality_measures.hwr_summary", ex.hwr_summary),
],
"pp_key": "cms_quality_measures._hwr_performance_period",
@@ -1168,20 +1222,24 @@ def _(con, ex, inspect, mo, pl):
try:
_result = _run_step_for_year(_fn, _cache, _sy)
_cache[_step_name] = _result
_step_detail_rows.append({
"measure": _mname,
"spec_year": _sy,
"step": _step_name.split(".")[-1],
"rows": len(_result),
})
_step_detail_rows.append(
{
"measure": _mname,
"spec_year": _sy,
"step": _step_name.split(".")[-1],
"rows": len(_result),
}
)
except Exception as _exc:
_error = str(_exc)
_step_detail_rows.append({
"measure": _mname,
"spec_year": _sy,
"step": _step_name.split(".")[-1],
"rows": -1,
})
_step_detail_rows.append(
{
"measure": _mname,
"spec_year": _sy,
"step": _step_name.split(".")[-1],
"rows": -1,
}
)
break
_summary = _cache.get(_mcfg["summary_key"])
@@ -1191,9 +1249,13 @@ def _(con, ex, inspect, mo, pl):
_row[c] = _summary[c][0]
_sensitivity_rows.append(_row)
else:
_sensitivity_rows.append({
"measure": _mname, "spec_year": _sy, "error": _error or "no summary",
})
_sensitivity_rows.append(
{
"measure": _mname,
"spec_year": _sy,
"error": _error or "no summary",
}
)
if _sensitivity_rows:
_sens_df = pl.DataFrame(_sensitivity_rows)
@@ -1203,8 +1265,13 @@ def _(con, ex, inspect, mo, pl):
# ── Build a concise comparison view ─────────────────────────────────
_display_cols = ["measure", "spec_year"]
_optional = [
"denominator_count", "observed_admissions", "total_person_years",
"observed_rate_per_100", "observed_readmissions", "observed_rate", "error",
"denominator_count",
"observed_admissions",
"total_person_years",
"observed_rate_per_100",
"observed_readmissions",
"observed_rate",
"error",
]
for _c in _optional:
if _c in _sens_df.columns:
@@ -1237,28 +1304,30 @@ def _(con, ex, inspect, mo, pl):
for _m, _piv in _step_pivots.items():
_items[f"{_m} — row counts per step"] = mo.ui.table(_piv)
mo.vstack([
mo.md(f"""## Spec Year Sensitivity Analysis
mo.vstack(
[
mo.md(f"""## Spec Year Sensitivity Analysis
How do year-over-year changes in CMS value set specifications affect measure
results on the **same population** (Synthea CY{_PERF_YEAR})?
How do year-over-year changes in CMS value set specifications affect measure
results on the **same population** (Synthea CY{_PERF_YEAR})?
Each measure is re-run using value sets from each available performance year spec,
with nearest-year fallback for tables missing a specific year.
Each measure is re-run using value sets from each available performance year spec,
with nearest-year fallback for tables missing a specific year.
**Available spec years per measure:**
**Available spec years per measure:**
{_year_list_md}
{_fb_md}
> **Note:** Identical results across spec years likely mean that the ~5,600 unique
> diagnosis codes in the Synthea synthetic population don't overlap with the codes
> that CMS added or removed between spec years. With real-world claims data covering
> a broader code space, spec year changes would be more likely to produce observable
> differences in measure outcomes.
"""),
mo.ui.table(_sens_display, label="Summary by Spec Year"),
mo.accordion(_items),
])
{_year_list_md}
{_fb_md}
> **Note:** Identical results across spec years likely mean that the ~5,600 unique
> diagnosis codes in the Synthea synthetic population don't overlap with the codes
> that CMS added or removed between spec years. With real-world claims data covering
> a broader code space, spec year changes would be more likely to produce observable
> differences in measure outcomes.
"""),
mo.ui.table(_sens_display, label="Summary by Spec Year"),
mo.accordion(_items),
]
)
return
@@ -1269,37 +1338,172 @@ def _(con, mo, pl):
# Every value-set table, its code column, measure, and functional role.
_VS = [
("_uamcc_value_set_cohort", "icd_10_cm", "UAMCC", "MCC cohort inclusion", "ICD-10-CM"),
("_uamcc_value_set_exclusions", "category_or_code", "UAMCC", "Outcome exclusion", "CCS/ICD-10"),
("_uamcc_value_set_paa1", "ccs_procedure_category", "UAMCC", "PAA Rule 1 — always-planned procedure CCS", "CCS"),
("_uamcc_value_set_paa2", "ccs_diagnosis_category", "UAMCC", "PAA Rule 2 — always-planned diagnosis CCS", "CCS"),
("_uamcc_value_set_paa3", "category_or_code", "UAMCC", "PAA Rule 3 — potentially-planned procedure", "CCS/ICD-10-PCS"),
("_uamcc_value_set_paa4", "category_or_code", "UAMCC", "PAA Rule 3 gate — acute diagnosis", "CCS/ICD-10-CM"),
("_uamcc_value_set_ccs_icd10_cm", "icd_10_cm", "UAMCC", "CCS crosswalk (diagnosis)", "ICD-10-CM"),
("_uamcc_value_set_ccs_icd10_pcs", "icd_10_pcs", "UAMCC", "CCS crosswalk (procedure)", "ICD-10-PCS"),
("_acr_value_set_cohort_ccs", "ccs_category", "ACR", "Specialty cohort CCS", "CCS"),
("_acr_value_set_cohort_icd10", "icd_10_pcs", "ACR", "Specialty cohort ICD-10-PCS", "ICD-10-PCS"),
("_acr_value_set_exclusions", "ccs_diagnosis_category", "ACR", "Cohort exclusion CCS", "CCS"),
("_acr_value_set_paa1", "ccs_procedure_category", "ACR", "PAA Rule 1 — always-planned procedure CCS", "CCS"),
("_acr_value_set_paa2", "ccs_diagnosis_category", "ACR", "PAA Rule 2 — always-planned diagnosis CCS", "CCS"),
("_acr_value_set_paa3", "category_or_code", "ACR", "PAA Rule 3 — potentially-planned procedure", "CCS/ICD-10-PCS"),
("_acr_value_set_paa4", "category_or_code", "ACR", "PAA Rule 3 gate — acute diagnosis", "CCS/ICD-10-CM"),
("_hwr_value_set_specialty_cohort", "ccs_category", "HWR", "Specialty cohort CCS", "CCS"),
("_hwr_value_set_surg_gyn_cohort", "icd_10_pcs", "HWR", "Surgery/Gyn cohort ICD-10-PCS", "ICD-10-PCS"),
("_hwr_value_set_cohort_exclusions", "ccs_diagnosis_category", "HWR", "Cohort exclusion CCS", "CCS"),
("_hwr_value_set_paa1", "ccs_procedure_category", "HWR", "PAA Rule 1 — always-planned procedure CCS", "CCS"),
("_hwr_value_set_paa2", "ccs_diagnosis_category", "HWR", "PAA Rule 2 — always-planned diagnosis CCS", "CCS"),
("_hwr_value_set_paa3", "category_or_code", "HWR", "PAA Rule 3 — potentially-planned procedure", "CCS/ICD-10-PCS"),
("_hwr_value_set_paa4", "category_or_code", "HWR", "PAA Rule 3 gate — acute diagnosis", "CCS/ICD-10-CM"),
(
"_uamcc_value_set_cohort",
"icd_10_cm",
"UAMCC",
"MCC cohort inclusion",
"ICD-10-CM",
),
(
"_uamcc_value_set_exclusions",
"category_or_code",
"UAMCC",
"Outcome exclusion",
"CCS/ICD-10",
),
(
"_uamcc_value_set_paa1",
"ccs_procedure_category",
"UAMCC",
"PAA Rule 1 — always-planned procedure CCS",
"CCS",
),
(
"_uamcc_value_set_paa2",
"ccs_diagnosis_category",
"UAMCC",
"PAA Rule 2 — always-planned diagnosis CCS",
"CCS",
),
(
"_uamcc_value_set_paa3",
"category_or_code",
"UAMCC",
"PAA Rule 3 — potentially-planned procedure",
"CCS/ICD-10-PCS",
),
(
"_uamcc_value_set_paa4",
"category_or_code",
"UAMCC",
"PAA Rule 3 gate — acute diagnosis",
"CCS/ICD-10-CM",
),
(
"_uamcc_value_set_ccs_icd10_cm",
"icd_10_cm",
"UAMCC",
"CCS crosswalk (diagnosis)",
"ICD-10-CM",
),
(
"_uamcc_value_set_ccs_icd10_pcs",
"icd_10_pcs",
"UAMCC",
"CCS crosswalk (procedure)",
"ICD-10-PCS",
),
(
"_acr_value_set_cohort_ccs",
"ccs_category",
"ACR",
"Specialty cohort CCS",
"CCS",
),
(
"_acr_value_set_cohort_icd10",
"icd_10_pcs",
"ACR",
"Specialty cohort ICD-10-PCS",
"ICD-10-PCS",
),
(
"_acr_value_set_exclusions",
"ccs_diagnosis_category",
"ACR",
"Cohort exclusion CCS",
"CCS",
),
(
"_acr_value_set_paa1",
"ccs_procedure_category",
"ACR",
"PAA Rule 1 — always-planned procedure CCS",
"CCS",
),
(
"_acr_value_set_paa2",
"ccs_diagnosis_category",
"ACR",
"PAA Rule 2 — always-planned diagnosis CCS",
"CCS",
),
(
"_acr_value_set_paa3",
"category_or_code",
"ACR",
"PAA Rule 3 — potentially-planned procedure",
"CCS/ICD-10-PCS",
),
(
"_acr_value_set_paa4",
"category_or_code",
"ACR",
"PAA Rule 3 gate — acute diagnosis",
"CCS/ICD-10-CM",
),
(
"_hwr_value_set_specialty_cohort",
"ccs_category",
"HWR",
"Specialty cohort CCS",
"CCS",
),
(
"_hwr_value_set_surg_gyn_cohort",
"icd_10_pcs",
"HWR",
"Surgery/Gyn cohort ICD-10-PCS",
"ICD-10-PCS",
),
(
"_hwr_value_set_cohort_exclusions",
"ccs_diagnosis_category",
"HWR",
"Cohort exclusion CCS",
"CCS",
),
(
"_hwr_value_set_paa1",
"ccs_procedure_category",
"HWR",
"PAA Rule 1 — always-planned procedure CCS",
"CCS",
),
(
"_hwr_value_set_paa2",
"ccs_diagnosis_category",
"HWR",
"PAA Rule 2 — always-planned diagnosis CCS",
"CCS",
),
(
"_hwr_value_set_paa3",
"category_or_code",
"HWR",
"PAA Rule 3 — potentially-planned procedure",
"CCS/ICD-10-PCS",
),
(
"_hwr_value_set_paa4",
"category_or_code",
"HWR",
"PAA Rule 3 gate — acute diagnosis",
"CCS/ICD-10-CM",
),
]
# Tables where `code_type` mixes CCS categories with redundant ICD-10
# detail expansions. The measure logic operates at CCS level — ICD-10
# detail rows are reference-only and should be diffed separately.
_HAS_CODE_TYPE = {
"_uamcc_value_set_paa3", "_uamcc_value_set_paa4",
"_acr_value_set_paa3", "_acr_value_set_paa4",
"_hwr_value_set_paa3", "_hwr_value_set_paa4",
"_uamcc_value_set_paa3",
"_uamcc_value_set_paa4",
"_acr_value_set_paa3",
"_acr_value_set_paa4",
"_hwr_value_set_paa3",
"_hwr_value_set_paa4",
"_uamcc_value_set_exclusions",
}
@@ -1307,9 +1511,12 @@ def _(con, mo, pl):
"""Return {year: {norm_code: orig_code}} dicts."""
_by_year = {}
for _y in _years:
_where = f" AND code_type = '{_code_type_filter}'" if _code_type_filter else ""
_where = (
f" AND code_type = '{_code_type_filter}'" if _code_type_filter else ""
)
_raw = [
r[0] for r in con.execute(
r[0]
for r in con.execute(
f'SELECT DISTINCT "{_key_col}" FROM {_q}'
f" WHERE performance_year = {_y}"
f' AND "{_key_col}" IS NOT NULL{_where}'
@@ -1326,17 +1533,29 @@ def _(con, mo, pl):
_sa = set(_by_year[_ya])
_sb = set(_by_year[_yb])
for _c in sorted(_sb - _sa):
_rows.append({
"measure": _measure, "role": _role, "code_type": _code_type,
"code": _by_year[_yb][_c], "change": "added",
"transition": f"{_ya} -> {_yb}", "table": _tbl,
})
_rows.append(
{
"measure": _measure,
"role": _role,
"code_type": _code_type,
"code": _by_year[_yb][_c],
"change": "added",
"transition": f"{_ya} -> {_yb}",
"table": _tbl,
}
)
for _c in sorted(_sa - _sb):
_rows.append({
"measure": _measure, "role": _role, "code_type": _code_type,
"code": _by_year[_ya][_c], "change": "removed",
"transition": f"{_ya} -> {_yb}", "table": _tbl,
})
_rows.append(
{
"measure": _measure,
"role": _role,
"code_type": _code_type,
"code": _by_year[_ya][_c],
"change": "removed",
"transition": f"{_ya} -> {_yb}",
"table": _tbl,
}
)
return _rows
_code_rows = []
@@ -1344,7 +1563,8 @@ def _(con, mo, pl):
_q = f'cms_quality_measures."{_tbl}"'
try:
_years = sorted(
r[0] for r in con.execute(
r[0]
for r in con.execute(
f"SELECT DISTINCT performance_year FROM {_q} ORDER BY 1"
).fetchall()
)
@@ -1356,25 +1576,47 @@ def _(con, mo, pl):
if _tbl in _HAS_CODE_TYPE:
# Diff CCS-level entries (functionally meaningful)
_ccs_sets = _build_code_sets(_q, _key_col, _years, "CCS")
_code_rows.extend(_diff_years(
_ccs_sets, _years, _measure, _role + " (CCS — operative)", "CCS", _tbl,
))
_code_rows.extend(
_diff_years(
_ccs_sets,
_years,
_measure,
_role + " (CCS — operative)",
"CCS",
_tbl,
)
)
# Diff ICD-10 detail entries separately (reference-only)
for _icd_type in ("ICD-10-CM", "ICD-10-PCS"):
_icd_sets = _build_code_sets(_q, _key_col, _years, _icd_type)
if any(len(v) > 0 for v in _icd_sets.values()):
_code_rows.extend(_diff_years(
_icd_sets, _years, _measure,
_role + f" ({_icd_type} — reference detail)", _icd_type, _tbl,
))
_code_rows.extend(
_diff_years(
_icd_sets,
_years,
_measure,
_role + f" ({_icd_type} — reference detail)",
_icd_type,
_tbl,
)
)
else:
_by_year = _build_code_sets(_q, _key_col, _years)
_code_rows.extend(_diff_years(
_by_year, _years, _measure, _role, _code_type, _tbl,
))
_code_rows.extend(
_diff_years(
_by_year,
_years,
_measure,
_role,
_code_type,
_tbl,
)
)
_codes_df = pl.DataFrame(_code_rows) if _code_rows else pl.DataFrame(
{"note": ["No code changes detected"]}
_codes_df = (
pl.DataFrame(_code_rows)
if _code_rows
else pl.DataFrame({"note": ["No code changes detected"]})
)
# Summary by measure × role × direction
@@ -1386,47 +1628,53 @@ def _(con, mo, pl):
# Impact classification: which changes could shift measure results?
_impact_md = """
| Change Type | Potential Impact | What to Query |
|-------------|-----------------|---------------|
| **Cohort inclusion** codes added | More patients enter the denominator | `WHERE dx_code IN ({codes}) AND encounter_type = 'acute inpatient'` |
| **Cohort inclusion** codes removed | Fewer patients in denominator | Same query — patients with these codes drop out |
| **Exclusion** codes added | More encounters excluded from numerator/denominator | `WHERE dx_ccs IN ({codes})` on your index admissions |
| **Exclusion** codes removed | Fewer exclusions → larger effective denominator | Same query — previously excluded patients now included |
| **PAA Rule 1/2/3** codes added | More admissions classified as *planned* → lower unplanned rate | `WHERE procedure_ccs IN ({codes})` or `WHERE dx_ccs IN ({codes})` on readmissions |
| **PAA Rule 3 gate (acute dx)** codes added | More procedures remain *unplanned* (acute dx negates Rule 3) → higher unplanned rate | `WHERE dx_ccs IN ({codes})` on readmissions with potentially-planned procedures |
| **CCS crosswalk** codes added/remapped | Diagnosis-to-CCS mapping changes cascade into all CCS-based logic above | `WHERE dx_code IN ({codes})` — check if CCS category assignment changed |
"""
| Change Type | Potential Impact | What to Query |
|-------------|-----------------|---------------|
| **Cohort inclusion** codes added | More patients enter the denominator | `WHERE dx_code IN ({codes}) AND encounter_type = 'acute inpatient'` |
| **Cohort inclusion** codes removed | Fewer patients in denominator | Same query — patients with these codes drop out |
| **Exclusion** codes added | More encounters excluded from numerator/denominator | `WHERE dx_ccs IN ({codes})` on your index admissions |
| **Exclusion** codes removed | Fewer exclusions → larger effective denominator | Same query — previously excluded patients now included |
| **PAA Rule 1/2/3** codes added | More admissions classified as *planned* → lower unplanned rate | `WHERE procedure_ccs IN ({codes})` or `WHERE dx_ccs IN ({codes})` on readmissions |
| **PAA Rule 3 gate (acute dx)** codes added | More procedures remain *unplanned* (acute dx negates Rule 3) → higher unplanned rate | `WHERE dx_ccs IN ({codes})` on readmissions with potentially-planned procedures |
| **CCS crosswalk** codes added/remapped | Diagnosis-to-CCS mapping changes cascade into all CCS-based logic above | `WHERE dx_code IN ({codes})` — check if CCS category assignment changed |
"""
mo.vstack([
mo.md("""## Research Strategy — Spec Year Code Changes
mo.vstack(
[
mo.md(
"""## Research Strategy — Spec Year Code Changes
To measure the real-world impact of spec year changes, query the **specific codes
that changed** against a target population. The tables below enumerate every code
added or removed between consecutive spec years, tagged by measure, functional role,
and code type.
To measure the real-world impact of spec year changes, query the **specific codes
that changed** against a target population. The tables below enumerate every code
added or removed between consecutive spec years, tagged by measure, functional role,
and code type.
### How spec changes propagate through the measures
### How spec changes propagate through the measures
""" + _impact_md + """
"""
+ _impact_md
+ """
### Step-by-step research protocol
### Step-by-step research protocol
1. **Export the code change table** below (CSV download via table widget)
2. **Filter to your measure of interest** (UAMCC, ACR, or HWR)
3. **Query your claims population** for encounters matching the changed codes:
- For ICD-10-CM changes: join on `principal_diagnosis_code` or `condition.normalized_code`
- For ICD-10-PCS changes: join on `procedure.normalized_code` or `hcpcs_code`
- For CCS changes: first map your ICD codes through the CCS crosswalk, then match
4. **Count affected encounters** — the overlap between changed codes and your population
determines whether the spec change would shift the measure result
5. **Re-run the pipeline** with each spec year's value sets (using the sensitivity
analysis cell above) on your real data to quantify the actual difference
"""),
mo.md(f"### Change Summary — {len(_codes_df)} total code changes"),
mo.ui.table(_summary, label="Changes by Measure / Role / Direction"),
mo.md("### Full Code Change Inventory"),
mo.ui.table(_codes_df, label="All Changed Codes (exportable)"),
])
1. **Export the code change table** below (CSV download via table widget)
2. **Filter to your measure of interest** (UAMCC, ACR, or HWR)
3. **Query your claims population** for encounters matching the changed codes:
- For ICD-10-CM changes: join on `principal_diagnosis_code` or `condition.normalized_code`
- For ICD-10-PCS changes: join on `procedure.normalized_code` or `hcpcs_code`
- For CCS changes: first map your ICD codes through the CCS crosswalk, then match
4. **Count affected encounters** — the overlap between changed codes and your population
determines whether the spec change would shift the measure result
5. **Re-run the pipeline** with each spec year's value sets (using the sensitivity
analysis cell above) on your real data to quantify the actual difference
"""
),
mo.md(f"### Change Summary — {len(_codes_df)} total code changes"),
mo.ui.table(_summary, label="Changes by Measure / Role / Direction"),
mo.md("### Full Code Change Inventory"),
mo.ui.table(_codes_df, label="All Changed Codes (exportable)"),
]
)
return

87
notebooks/gpu_test.py
View File

@@ -4,23 +4,23 @@ __generated_with = "0.19.7"
app = marimo.App(width="medium")
with app.setup:
import marimo as mo
import subprocess
import platform
import os
import platform
import subprocess
import sys
import time
import marimo as mo
import numpy as np
import polars as pl
import pandas as pd
import polars as pl
@app.cell(hide_code=True)
def gpu_diagnostics():
try:
smi_result = subprocess.run(
["nvidia-smi"],
capture_output=True, text=True, timeout=10
["nvidia-smi"], capture_output=True, text=True, timeout=10
)
nvidia_smi_output = smi_result.stdout
gpu_detected = smi_result.returncode == 0
@@ -51,13 +51,15 @@ def polars_gpu_benchmark():
bench_rng = np.random.default_rng(42)
bench_gen_start = time.perf_counter()
bench_df = pl.DataFrame({
"id": np.arange(benchmark_rows),
"group": bench_rng.choice(["A", "B", "C", "D", "E"], size=benchmark_rows),
"value_1": bench_rng.standard_normal(benchmark_rows),
"value_2": bench_rng.uniform(0, 1000, size=benchmark_rows),
"value_3": bench_rng.integers(0, 100, size=benchmark_rows),
})
bench_df = pl.DataFrame(
{
"id": np.arange(benchmark_rows),
"group": bench_rng.choice(["A", "B", "C", "D", "E"], size=benchmark_rows),
"value_1": bench_rng.standard_normal(benchmark_rows),
"value_2": bench_rng.uniform(0, 1000, size=benchmark_rows),
"value_3": bench_rng.integers(0, 100, size=benchmark_rows),
}
)
bench_gen_elapsed = time.perf_counter() - bench_gen_start
# Pre-create lazy frame to exclude setup from timing
@@ -77,8 +79,7 @@ def polars_gpu_benchmark():
# --- GPU collect ---
bench_gpu_agg_start = time.perf_counter()
bench_gpu_agg_result = (
bench_lazy
.group_by("group")
bench_lazy.group_by("group")
.agg(*bench_agg_expr)
.sort("group")
.collect(engine="gpu")
@@ -88,15 +89,15 @@ def polars_gpu_benchmark():
# --- CPU collect ---
bench_cpu_agg_start = time.perf_counter()
bench_cpu_agg_result = (
bench_lazy
.group_by("group")
.agg(*bench_agg_expr)
.sort("group")
.collect()
bench_lazy.group_by("group").agg(*bench_agg_expr).sort("group").collect()
)
bench_cpu_agg_elapsed = time.perf_counter() - bench_cpu_agg_start
bench_agg_speedup = bench_cpu_agg_elapsed / bench_gpu_agg_elapsed if bench_gpu_agg_elapsed > 0 else float("inf")
bench_agg_speedup = (
bench_cpu_agg_elapsed / bench_gpu_agg_elapsed
if bench_gpu_agg_elapsed > 0
else float("inf")
)
# GPU-supported window functions only (no rank which causes fallback)
bench_window_expr = [
@@ -107,24 +108,22 @@ def polars_gpu_benchmark():
# --- GPU window ---
bench_gpu_window_start = time.perf_counter()
bench_gpu_window_result = (
bench_lazy
.with_columns(*bench_window_expr)
.head(5)
.collect(engine="gpu")
bench_lazy.with_columns(*bench_window_expr).head(5).collect(engine="gpu")
)
bench_gpu_window_elapsed = time.perf_counter() - bench_gpu_window_start
# --- CPU window ---
bench_cpu_window_start = time.perf_counter()
bench_cpu_window_result = (
bench_lazy
.with_columns(*bench_window_expr)
.head(5)
.collect()
bench_lazy.with_columns(*bench_window_expr).head(5).collect()
)
bench_cpu_window_elapsed = time.perf_counter() - bench_cpu_window_start
bench_window_speedup = bench_cpu_window_elapsed / bench_gpu_window_elapsed if bench_gpu_window_elapsed > 0 else float("inf")
bench_window_speedup = (
bench_cpu_window_elapsed / bench_gpu_window_elapsed
if bench_gpu_window_elapsed > 0
else float("inf")
)
mo.md(f"""
# Polars GPU vs CPU — {benchmark_rows:,} rows
@@ -136,10 +135,12 @@ def polars_gpu_benchmark():
| Window functions | `{bench_gpu_window_elapsed:.3f}s` | `{bench_cpu_window_elapsed:.3f}s` | **{bench_window_speedup:.1f}x** |
""")
mo.hstack([
mo.ui.table(bench_gpu_agg_result, label="GPU Aggregation Results"),
mo.ui.table(bench_gpu_window_result, label="GPU Window Functions (head 5)"),
])
mo.hstack(
[
mo.ui.table(bench_gpu_agg_result, label="GPU Aggregation Results"),
mo.ui.table(bench_gpu_window_result, label="GPU Window Functions (head 5)"),
]
)
return
@@ -155,7 +156,9 @@ def pandas_vs_polars_gpu():
# --- Pandas (CPU only) - time only the compute, not DataFrame creation ---
cmp_pandas_df = pd.DataFrame({"category": cmp_categories, "amount": cmp_amounts})
cmp_pandas_start = time.perf_counter()
cmp_pandas_agg = cmp_pandas_df.groupby("category")["amount"].agg(["mean", "std", "sum"])
cmp_pandas_agg = cmp_pandas_df.groupby("category")["amount"].agg(
["mean", "std", "sum"]
)
cmp_pandas_elapsed = time.perf_counter() - cmp_pandas_start
# --- Polars GPU - time only the compute ---
@@ -164,8 +167,7 @@ def pandas_vs_polars_gpu():
cmp_gpu_start = time.perf_counter()
cmp_gpu_agg = (
cmp_lazy
.group_by("category")
cmp_lazy.group_by("category")
.agg(
pl.col("amount").mean().alias("mean"),
pl.col("amount").std().alias("std"),
@@ -179,8 +181,7 @@ def pandas_vs_polars_gpu():
# --- Polars CPU ---
cmp_cpu_start = time.perf_counter()
cmp_cpu_agg = (
cmp_lazy
.group_by("category")
cmp_lazy.group_by("category")
.agg(
pl.col("amount").mean().alias("mean"),
pl.col("amount").std().alias("std"),
@@ -191,8 +192,12 @@ def pandas_vs_polars_gpu():
)
cmp_cpu_elapsed = time.perf_counter() - cmp_cpu_start
cmp_gpu_vs_pandas = cmp_pandas_elapsed / cmp_gpu_elapsed if cmp_gpu_elapsed > 0 else float("inf")
cmp_cpu_vs_pandas = cmp_pandas_elapsed / cmp_cpu_elapsed if cmp_cpu_elapsed > 0 else float("inf")
cmp_gpu_vs_pandas = (
cmp_pandas_elapsed / cmp_gpu_elapsed if cmp_gpu_elapsed > 0 else float("inf")
)
cmp_cpu_vs_pandas = (
cmp_pandas_elapsed / cmp_cpu_elapsed if cmp_cpu_elapsed > 0 else float("inf")
)
mo.md(f"""
# Three-Way Comparison — {cmp_rows:,} rows

19
notebooks/nessie_tutorial.py
View File

@@ -7,6 +7,7 @@ app = marimo.App(width="medium")
@app.cell(hide_code=True)
def _():
import marimo as mo
mo.md("""
# Nessie Tutorial - Git for Your Data Lake
@@ -19,9 +20,10 @@ def _():
@app.cell(hide_code=True)
def _():
import requests
import json
import requests
from conf import cfg
NESSIE_API = f"{cfg.services.nessie}/api/v2"
@@ -98,11 +100,9 @@ def _(NESSIE_API, json, main_ref, requests):
resp = requests.post(
f"{NESSIE_API}/trees?name={new_branch}&type=branch",
headers={"Content-Type": "application/json"},
data=json.dumps({
"type": "BRANCH",
"name": main_ref["name"],
"hash": main_ref["hash"]
})
data=json.dumps(
{"type": "BRANCH", "name": main_ref["name"], "hash": main_ref["hash"]}
),
)
if resp.status_code == 200:
@@ -249,7 +249,9 @@ def _(mo):
@app.cell(hide_code=True)
def _(query):
count = query("SELECT count(*) FROM iceberg.tutorial.events FOR VERSION AS OF 'main'")
count = query(
"SELECT count(*) FROM iceberg.tutorial.events FOR VERSION AS OF 'main'"
)
print(f"Events on main: {count[0][0]}")
return
@@ -271,8 +273,7 @@ def _(NESSIE_API, requests):
print(f"Branch {name} not found")
return
resp = requests.delete(
f"{NESSIE_API}/trees/{name}",
headers={"Expected-Hash": _hash}
f"{NESSIE_API}/trees/{name}", headers={"Expected-Hash": _hash}
)
if resp.status_code == 204:
print(f"Deleted: {name}")

29
notebooks/pfs_calcs.py
View File

@@ -25,6 +25,7 @@ def _(mo):
@app.cell(hide_code=True)
def _():
import polars as pl
from conf import connect
con = connect.duckdb()
@@ -51,10 +52,7 @@ def _(con, mo):
"""
).fetchall()
_loc_options = {
f"{name} ({loc})": f"{mac}|{loc}"
for mac, loc, name in _localities
}
_loc_options = {f"{name} ({loc})": f"{mac}|{loc}" for mac, loc, name in _localities}
_first_key = next(iter(_loc_options))
year_picker = mo.ui.dropdown(
@@ -139,7 +137,7 @@ def _(hcpcs_input, locality_picker, pl, q, year_picker):
_cf = RULES[_year].conversion_factor
rvu_df = q(f"""
SELECT *, {_cf} as conv_factor, '{_locality}' as locality, '{_mac}' as mac
SELECT *, '{_locality}' as locality, '{_mac}' as mac
FROM pfs.rvu
WHERE year = {_year} AND hcpcs = '{_hcpcs}'
""")
@@ -149,14 +147,21 @@ def _(hcpcs_input, locality_picker, pl, q, year_picker):
WHERE year = {_year} AND mac = '{_mac}' AND locality = '{_locality}'
""")
calc_nf = payment(rvu_df, gpci_df, facility=False).select(
"hcpcs", "mod", "work_rvu", "non_fac_pe_rvu", "mp_rvu",
"work_gpci", "pe_gpci", "mp_gpci", "conv_factor",
calc_nf = payment(rvu_df, gpci_df, cf=_cf, facility=False).select(
"hcpcs",
"mod",
"work_rvu",
"non_fac_pe_rvu",
"mp_rvu",
"work_gpci",
"pe_gpci",
"mp_gpci",
pl.col("payment_amount").round(2).alias("non_fac_payment"),
)
calc_f = payment(rvu_df, gpci_df, facility=True).select(
"hcpcs", "mod",
calc_f = payment(rvu_df, gpci_df, cf=_cf, facility=True).select(
"hcpcs",
"mod",
pl.col("payment_amount").round(2).alias("fac_payment"),
)
@@ -253,7 +258,9 @@ def _(carrier_result, mo, pl, calc_result):
indicates a carrier-priced or status-indicator edge case.
"""
else:
_comparison = "*Select a valid year/locality/code combination to see comparison.*"
_comparison = (
"*Select a valid year/locality/code combination to see comparison.*"
)
mo.md(_comparison)
return

24
notebooks/pfs_reconciliation.py
View File

@@ -13,15 +13,13 @@ def _():
@app.cell(hide_code=True)
def _(mo):
mo.md(
"""
mo.md("""
# PFS Reconciliation
Runs `rec.pricers.pfs.PfsPricer` against `pfs.carrier_locality`
for a chosen year and reports the per-row delta. Goal is perfect
1:1 concordance. Tracks **homelab/stack#340**.
"""
)
""")
return
@@ -58,7 +56,7 @@ def _(con, mo, pricer):
return tolerance, year_picker
@app.cell
@app.cell(hide_code=True)
def _(con, mo, pricer, reconcile, tolerance, year_picker):
_year = int(year_picker.value) if year_picker.value else 0
if _year == 0:
@@ -72,20 +70,18 @@ def _(con, mo, pricer, reconcile, tolerance, year_picker):
@app.cell(hide_code=True)
def _(mo):
mo.md(
"""
mo.md("""
## Delta table
Every row in the outer join, sorted by the largest absolute delta.
``is_exact`` uses the tolerance above; ``is_near`` is always a 1¢
window. Null ``fee_gt`` means the row is calculated-only; null
``fee_calc`` means ground-truth-only.
"""
)
""")
return
@app.cell
@app.cell(hide_code=True)
def _(result):
_deltas = result.deltas if result is not None else None
_deltas
@@ -94,18 +90,16 @@ def _(result):
@app.cell(hide_code=True)
def _(mo):
mo.md(
"""
mo.md("""
## Warnings
Non-fatal issues surfaced by the engine — duplicate join keys,
missing columns, rule lookup failures, etc.
"""
)
""")
return
@app.cell
@app.cell(hide_code=True)
def _(mo, result):
if result is None or not result.warnings:
mo.md("*(none)*")

70
notebooks/polaris_tutorial.py
View File

@@ -7,6 +7,7 @@ app = marimo.App(width="medium")
@app.cell(hide_code=True)
def _():
import marimo as mo
mo.md("""
# Apache Polaris Tutorial - Iceberg Catalog with Governance
@@ -20,10 +21,11 @@ def _():
@app.cell(hide_code=True)
def _():
import requests
import json
import os
import requests
from conf import cfg
POLARIS_API = cfg.services.polaris
@@ -51,8 +53,8 @@ def _(POLARIS_API, CLIENT_ID, CLIENT_SECRET, requests):
"grant_type": "client_credentials",
"client_id": CLIENT_ID,
"client_secret": CLIENT_SECRET,
"scope": "PRINCIPAL_ROLE:ALL"
}
"scope": "PRINCIPAL_ROLE:ALL",
},
)
if token_resp.status_code == 200:
@@ -81,8 +83,7 @@ def _(mo):
def _(POLARIS_API, access_token, requests):
headers = {"Authorization": f"Bearer {access_token}"}
catalogs_resp = requests.get(
f"{POLARIS_API}/api/management/v1/catalogs",
headers=headers
f"{POLARIS_API}/api/management/v1/catalogs", headers=headers
)
if catalogs_resp.status_code == 200:
@@ -114,22 +115,24 @@ def _(POLARIS_API, headers, json, requests):
create_resp = requests.post(
f"{POLARIS_API}/api/management/v1/catalogs",
headers={**headers, "Content-Type": "application/json"},
data=json.dumps({
"name": catalog_name,
"type": "INTERNAL",
"properties": {
"default-base-location": f"s3://polaris/{catalog_name}/"
},
"storageConfigInfo": {
"storageType": "S3",
"allowedLocations": ["s3://polaris/", "s3://lakehouse/"],
"s3": {
"region": "us-east-1",
"endpoint": cfg.services.rustfs,
"pathStyleAccess": True
}
data=json.dumps(
{
"name": catalog_name,
"type": "INTERNAL",
"properties": {
"default-base-location": f"s3://polaris/{catalog_name}/"
},
"storageConfigInfo": {
"storageType": "S3",
"allowedLocations": ["s3://polaris/", "s3://lakehouse/"],
"s3": {
"region": "us-east-1",
"endpoint": cfg.services.rustfs,
"pathStyleAccess": True,
},
},
}
})
),
)
if create_resp.status_code == 200:
@@ -153,8 +156,7 @@ def _(mo):
@app.cell(hide_code=True)
def _(POLARIS_API, catalog_name, headers, requests):
cat_resp = requests.get(
f"{POLARIS_API}/api/management/v1/catalogs/{catalog_name}",
headers=headers
f"{POLARIS_API}/api/management/v1/catalogs/{catalog_name}", headers=headers
)
if cat_resp.status_code == 200:
@@ -180,8 +182,7 @@ def _(mo):
@app.cell(hide_code=True)
def _(POLARIS_API, headers, requests):
principals_resp = requests.get(
f"{POLARIS_API}/api/management/v1/principals",
headers=headers
f"{POLARIS_API}/api/management/v1/principals", headers=headers
)
if principals_resp.status_code == 200:
@@ -207,8 +208,7 @@ def _(mo):
@app.cell(hide_code=True)
def _(POLARIS_API, headers, requests):
roles_resp = requests.get(
f"{POLARIS_API}/api/management/v1/principal-roles",
headers=headers
f"{POLARIS_API}/api/management/v1/principal-roles", headers=headers
)
if roles_resp.status_code == 200:
@@ -239,7 +239,7 @@ def _(POLARIS_API, catalog_name, headers, requests):
config_resp = requests.get(
f"{POLARIS_API}/api/catalog/v1/config",
headers=headers,
params={"warehouse": catalog_name}
params={"warehouse": catalog_name},
)
if config_resp.status_code == 200:
@@ -266,8 +266,7 @@ def _(mo):
@app.cell(hide_code=True)
def _(POLARIS_API, catalog_name, headers, requests):
ns_resp = requests.get(
f"{POLARIS_API}/api/catalog/v1/{catalog_name}/namespaces",
headers=headers
f"{POLARIS_API}/api/catalog/v1/{catalog_name}/namespaces", headers=headers
)
if ns_resp.status_code == 200:
@@ -298,12 +297,12 @@ def _(POLARIS_API, catalog_name, headers, json, requests):
create_ns_resp = requests.post(
f"{POLARIS_API}/api/catalog/v1/{catalog_name}/namespaces",
headers={**headers, "Content-Type": "application/json"},
data=json.dumps({
"namespace": [ns_name],
"properties": {
"description": "Tutorial namespace"
data=json.dumps(
{
"namespace": [ns_name],
"properties": {"description": "Tutorial namespace"},
}
})
),
)
if create_ns_resp.status_code == 200:
@@ -329,8 +328,7 @@ def _(mo):
def _(POLARIS_API, headers, requests):
def delete_catalog(name):
resp = requests.delete(
f"{POLARIS_API}/api/management/v1/catalogs/{name}",
headers=headers
f"{POLARIS_API}/api/management/v1/catalogs/{name}", headers=headers
)
if resp.status_code == 204:
print(f"Deleted: {name}")

2
notebooks/sample.py
View File

@@ -2,7 +2,7 @@
import marimo
__generated_with = "0.19.9"
__generated_with = "0.21.1"
app = marimo.App()

102
notebooks/skin_sub_budget_neutrality.py
View File

@@ -37,6 +37,7 @@ def _(mo):
def _():
import altair as alt
import polars as pl
from conf import connect
from pfs.rules import RULES
@@ -102,7 +103,11 @@ def _(SKIN_CODES, alt, mo, q):
y=alt.Y("skin_rvu:Q", title="Skin Sub Total NF RVUs (8 codes)"),
tooltip=["year", "skin_rvu", "total_rvu", "skin_pct", "skin_delta"],
)
.properties(title="Skin Sub Application Codes — Total NF RVUs by Year", width=700, height=300)
.properties(
title="Skin Sub Application Codes — Total NF RVUs by Year",
width=700,
height=300,
)
)
mo.vstack([share_chart, pool_share])
@@ -153,7 +158,14 @@ def _(alt, q):
x=alt.X("year:O", title="Year"),
y=alt.Y("non_fac_pe_rvu:Q", title="Non-Facility PE RVU"),
color=alt.Color("label:N", title="Code"),
tooltip=["year", "hcpcs", "label", "non_fac_pe_rvu", "work_rvu", "total_nf_rvu"],
tooltip=[
"year",
"hcpcs",
"label",
"non_fac_pe_rvu",
"work_rvu",
"total_nf_rvu",
],
)
.properties(title="Practice Expense RVU Trajectory", width=700, height=350)
)
@@ -293,27 +305,42 @@ def _(SKIN_CODES, alt, mo, q):
alt.Chart(category_impact.to_pandas())
.mark_bar()
.encode(
x=alt.X("implied_pe_loss:Q", title=f"Implied PE RVU Loss (from {delta_val:+.2f} skin sub PE delta)"),
x=alt.X(
"implied_pe_loss:Q",
title=f"Implied PE RVU Loss (from {delta_val:+.2f} skin sub PE delta)",
),
y=alt.Y("category:N", title="", sort="-x"),
color=alt.Color("pe_share_pct:Q", title="PE Pool Share %",
scale=alt.Scale(scheme="reds")),
tooltip=["category", "codes", "category_pe", "pe_share_pct", "implied_pe_loss"],
color=alt.Color(
"pe_share_pct:Q",
title="PE Pool Share %",
scale=alt.Scale(scheme="reds"),
),
tooltip=[
"category",
"codes",
"category_pe",
"pe_share_pct",
"implied_pe_loss",
],
)
.properties(
title=f"Budget Neutrality Burden by Service Category (CY{delta_year})",
width=700, height=350,
width=700,
height=350,
)
)
mo.vstack([
mo.md(f"""
mo.vstack(
[
mo.md(f"""
**CY{delta_year}:** Skin sub application codes gained **{delta_val:+.2f} PE RVUs**.
Under budget neutrality, this is redistributed across ~{category_impact.select('codes').sum().item():,} other codes
Under budget neutrality, this is redistributed across ~{category_impact.select("codes").sum().item():,} other codes
proportional to their PE share.
"""),
impact_chart,
category_impact,
])
impact_chart,
category_impact,
]
)
return
@@ -339,10 +366,12 @@ def _(mo):
@app.cell(hide_code=True)
def _(RULES, alt, pl, q):
cf_data = pl.DataFrame({
"year": list(RULES.keys()),
"conversion_factor": [r.conversion_factor for r in RULES.values()],
})
cf_data = pl.DataFrame(
{
"year": list(RULES.keys()),
"conversion_factor": [r.conversion_factor for r in RULES.values()],
}
)
pool_growth = q("""
SELECT year, round(sum(work_rvu + non_fac_pe_rvu + mp_rvu), 0) as total_rvu
@@ -357,8 +386,11 @@ def _(RULES, alt, pl, q):
.mark_line(point=True, color="#1f77b4")
.encode(
x=alt.X("year:O", title="Year"),
y=alt.Y("conversion_factor:Q", title="Conversion Factor ($)",
scale=alt.Scale(zero=False)),
y=alt.Y(
"conversion_factor:Q",
title="Conversion Factor ($)",
scale=alt.Scale(zero=False),
),
tooltip=["year", "conversion_factor", "total_rvu"],
)
)
@@ -368,15 +400,20 @@ def _(RULES, alt, pl, q):
.mark_line(point=True, color="#d62728", strokeDash=[4, 4])
.encode(
x=alt.X("year:O"),
y=alt.Y("total_rvu:Q", title="Total Unweighted RVU Pool",
scale=alt.Scale(zero=False)),
y=alt.Y(
"total_rvu:Q",
title="Total Unweighted RVU Pool",
scale=alt.Scale(zero=False),
),
)
)
cf_chart = (
alt.layer(cf_line, rvu_line)
.resolve_scale(y="independent")
.properties(title="Conversion Factor vs. RVU Pool Growth", width=700, height=350)
.properties(
title="Conversion Factor vs. RVU Pool Growth", width=700, height=350
)
)
cf_chart
return
@@ -449,16 +486,18 @@ def _(RULES, SKIN_CODES, mo, q):
ORDER BY dollar_loss
""")
mo.vstack([
mo.md(f"""
mo.vstack(
[
mo.md(f"""
**CY{latest_year}** parameters:
- Skin sub PE delta: **{skin_delta:+.2f} RVUs**
- Other-code PE pool: **{total_pe:,.1f} RVUs**
- Implied tax rate: **{tax_rate*100:.4f}%** of each code's PE
- Implied tax rate: **{tax_rate * 100:.4f}%** of each code's PE
- CF: **${cf}**
"""),
common_codes,
])
common_codes,
]
)
return
@@ -502,8 +541,10 @@ def _(SKIN_CODES, alt, q):
.mark_area(opacity=0.3, color="#d62728")
.encode(
x=alt.X("year:O", title="Year"),
y=alt.Y("cumulative_pe_growth:Q",
title="Cumulative Skin Sub PE Growth (RVUs above 2015 baseline)"),
y=alt.Y(
"cumulative_pe_growth:Q",
title="Cumulative Skin Sub PE Growth (RVUs above 2015 baseline)",
),
tooltip=["year", "skin_pe", "cumulative_pe_growth", "total_pe"],
)
) + (
@@ -517,7 +558,8 @@ def _(SKIN_CODES, alt, q):
cum_chart_final = cum_chart.properties(
title="Cumulative PE RVU Growth — Skin Sub Application Codes vs. 2015 Baseline",
width=700, height=300,
width=700,
height=300,
)
cum_chart_final
return

202
notebooks/skin_sub_cost_sharing.py
View File

@@ -38,6 +38,7 @@ def _(mo):
def _():
import altair as alt
import polars as pl
from conf import connect
from pfs.rules import RULES
@@ -50,9 +51,18 @@ def _():
# Part B deductible history (published by CMS annually)
DEDUCTIBLES = {
2015: 147.00, 2016: 166.00, 2017: 183.00, 2018: 183.00,
2019: 185.00, 2020: 198.00, 2021: 203.00, 2022: 233.00,
2023: 226.00, 2024: 240.00, 2025: 257.00, 2026: 257.00,
2015: 147.00,
2016: 166.00,
2017: 183.00,
2018: 183.00,
2019: 185.00,
2020: 198.00,
2021: 203.00,
2022: 233.00,
2023: 226.00,
2024: 240.00,
2025: 257.00,
2026: 257.00,
}
return DEDUCTIBLES, RULES, SKIN_CODES, alt, con, pl, q
@@ -75,8 +85,11 @@ def _(mo):
@app.cell(hide_code=True)
def _(alt, q):
regions = [
"MANHATTAN", "REST OF FLORIDA", "REST OF TEXAS",
"REST OF CALIFORNIA", "SOUTH CAROLINA",
"MANHATTAN",
"REST OF FLORIDA",
"REST OF TEXAS",
"REST OF CALIFORNIA",
"SOUTH CAROLINA",
]
region_list = ", ".join(f"'{r}'" for r in regions)
@@ -100,12 +113,18 @@ def _(alt, q):
x=alt.X("year:O", title="Year"),
y=alt.Y("bene_coinsurance:Q", title="Beneficiary Coinsurance ($)"),
color=alt.Color("locality_name:N", title="Region"),
tooltip=["year", "locality_name", "non_fac_fee",
"bene_coinsurance", "non_fac_limiting_charge"],
tooltip=[
"year",
"locality_name",
"non_fac_fee",
"bene_coinsurance",
"non_fac_limiting_charge",
],
)
.properties(
title="15271 Application — Beneficiary 20% Coinsurance by Region",
width=700, height=350,
width=700,
height=350,
)
)
app_chart
@@ -151,21 +170,27 @@ def _(alt, pl, q):
alt.Chart(product_coins.to_pandas())
.mark_line()
.encode(
x=alt.X("quarter:O", title="Quarter",
axis=alt.Axis(labelAngle=-45, labelFontSize=8)),
x=alt.X(
"quarter:O",
title="Quarter",
axis=alt.Axis(labelAngle=-45, labelFontSize=8),
),
y=alt.Y("bene_per_unit:Q", title="Beneficiary Coinsurance per cm² ($)"),
color=alt.Color("short_description:N", title="Product"),
tooltip=["quarter", "hcpcs_code", "short_description",
"payment_limit", "bene_per_unit"],
tooltip=[
"quarter",
"hcpcs_code",
"short_description",
"payment_limit",
"bene_per_unit",
],
)
)
product_chart = (
(product_lines + flat_coins)
.properties(
title="Product Coinsurance per Unit (20% of ASP + 6%)",
width=700, height=400,
)
product_chart = (product_lines + flat_coins).properties(
title="Product Coinsurance per Unit (20% of ASP + 6%)",
width=700,
height=400,
)
product_chart
return
@@ -234,21 +259,24 @@ def _(alt, pl, q):
.mark_line(point=True)
.encode(
x=alt.X("year:O", title="Year"),
y=alt.Y("total_bene_cost:Q",
title="Beneficiary Total Cost Sharing ($)"),
y=alt.Y("total_bene_cost:Q", title="Beneficiary Total Cost Sharing ($)"),
color=alt.Color("short_description:N", title="Product"),
tooltip=["year", "hcpcs_code", "short_description",
"asp_per_unit", "app_coinsurance",
"product_coinsurance_25cm", "total_bene_cost"],
tooltip=[
"year",
"hcpcs_code",
"short_description",
"asp_per_unit",
"app_coinsurance",
"product_coinsurance_25cm",
"total_bene_cost",
],
)
)
episode_chart = (
(episode_lines + flat_line)
.properties(
title="Total Beneficiary Cost Sharing — 25cm² Wound Episode",
width=700, height=400,
)
episode_chart = (episode_lines + flat_line).properties(
title="Total Beneficiary Cost Sharing — 25cm² Wound Episode",
width=700,
height=400,
)
episode_chart
return
@@ -302,12 +330,19 @@ def _(alt, q):
alt.value("#2ca02c"),
alt.value("#1f77b4"),
),
tooltip=["year", "products", "avg_copay", "min_copay",
"max_copay", "copay_spread"],
tooltip=[
"year",
"products",
"avg_copay",
"min_copay",
"max_copay",
"copay_spread",
],
)
.properties(
title="OPPS Minimum Unadjusted Copayment — Skin Substitutes",
width=700, height=300,
width=700,
height=300,
)
)
@@ -357,19 +392,28 @@ def _(alt, q):
alt.Chart(bene_impact.to_pandas())
.mark_bar()
.encode(
x=alt.X("bene_delta_per_unit:Q",
title="Change in Beneficiary Coinsurance per Unit ($)"),
y=alt.Y("short_description:N", title="", sort="x",
axis=alt.Axis(labelLimit=300)),
color=alt.Color("bene_impact:N",
scale=alt.Scale(
domain=["bene saves", "bene pays more", "neutral"],
range=["#2ca02c", "#d62728", "#7f7f7f"],
),
title="Impact"),
tooltip=["hcpcs_code", "short_description",
"current_bene_per_unit", "flat_bene_per_unit",
"bene_delta_per_unit"],
x=alt.X(
"bene_delta_per_unit:Q",
title="Change in Beneficiary Coinsurance per Unit ($)",
),
y=alt.Y(
"short_description:N", title="", sort="x", axis=alt.Axis(labelLimit=300)
),
color=alt.Color(
"bene_impact:N",
scale=alt.Scale(
domain=["bene saves", "bene pays more", "neutral"],
range=["#2ca02c", "#d62728", "#7f7f7f"],
),
title="Impact",
),
tooltip=[
"hcpcs_code",
"short_description",
"current_bene_per_unit",
"flat_bene_per_unit",
"bene_delta_per_unit",
],
)
.properties(
title="CY2026 Beneficiary Coinsurance Change per Unit",
@@ -405,27 +449,40 @@ def _(pl, q):
WHERE year = 2025 AND hcpcs = '15271'
""").item()
scenarios = pl.DataFrame({
"scenario": [
"Small wound (10cm²) — Q4101 Apligraf",
"Medium wound (25cm²) — Q4186 EpiFix",
"Large wound (50cm²) — Q4132 GrafixCore",
],
"units": [10, 25, 50],
"asp_per_unit": [30.23, 151.17, 106.70],
"product_name": ["Apligraf", "EpiFix", "GrafixCore"],
}).with_columns(
# Pre-2026: ASP + 6%
(pl.col("asp_per_unit") * pl.col("units") * 0.20).round(2).alias("pre_product_coins"),
pl.lit(avg_fee * 0.20).round(2).alias("pre_app_coins"),
# Post-2026: flat $127.28
(pl.lit(127.28) * pl.col("units") * 0.20).round(2).alias("post_product_coins"),
pl.lit(avg_fee * 0.20).round(2).alias("post_app_coins"),
).with_columns(
(pl.col("pre_product_coins") + pl.col("pre_app_coins")).alias("pre_total"),
(pl.col("post_product_coins") + pl.col("post_app_coins")).alias("post_total"),
).with_columns(
(pl.col("post_total") - pl.col("pre_total")).round(2).alias("delta"),
scenarios = (
pl.DataFrame(
{
"scenario": [
"Small wound (10cm²) — Q4101 Apligraf",
"Medium wound (25cm²) — Q4186 EpiFix",
"Large wound (50cm²) — Q4132 GrafixCore",
],
"units": [10, 25, 50],
"asp_per_unit": [30.23, 151.17, 106.70],
"product_name": ["Apligraf", "EpiFix", "GrafixCore"],
}
)
.with_columns(
# Pre-2026: ASP + 6%
(pl.col("asp_per_unit") * pl.col("units") * 0.20)
.round(2)
.alias("pre_product_coins"),
pl.lit(avg_fee * 0.20).round(2).alias("pre_app_coins"),
# Post-2026: flat $127.28
(pl.lit(127.28) * pl.col("units") * 0.20)
.round(2)
.alias("post_product_coins"),
pl.lit(avg_fee * 0.20).round(2).alias("post_app_coins"),
)
.with_columns(
(pl.col("pre_product_coins") + pl.col("pre_app_coins")).alias("pre_total"),
(pl.col("post_product_coins") + pl.col("post_app_coins")).alias(
"post_total"
),
)
.with_columns(
(pl.col("post_total") - pl.col("pre_total")).round(2).alias("delta"),
)
)
scenarios
return
@@ -449,10 +506,12 @@ def _(mo):
@app.cell(hide_code=True)
def _(DEDUCTIBLES, alt, pl):
deductible_df = pl.DataFrame({
"year": list(DEDUCTIBLES.keys()),
"deductible": list(DEDUCTIBLES.values()),
})
deductible_df = pl.DataFrame(
{
"year": list(DEDUCTIBLES.keys()),
"deductible": list(DEDUCTIBLES.values()),
}
)
ded_chart = (
alt.Chart(deductible_df.to_pandas())
@@ -464,7 +523,8 @@ def _(DEDUCTIBLES, alt, pl):
)
.properties(
title="Medicare Part B Annual Deductible",
width=700, height=250,
width=700,
height=250,
)
)
ded_chart

134
notebooks/skin_sub_pricing.py
View File

@@ -35,6 +35,7 @@ def _(mo):
def _():
import altair as alt
import polars as pl
from conf import connect
con = connect.duckdb()
@@ -87,8 +88,15 @@ def _(alt, q):
x=alt.X("year:O", title="Year"),
y=alt.Y("non_fac_total:Q", title="Total Non-Facility RVUs"),
color=alt.Color("description:N", title="Code"),
tooltip=["year", "hcpcs", "description",
"work_rvu", "non_fac_pe_rvu", "mp_rvu", "non_fac_total"],
tooltip=[
"year",
"hcpcs",
"description",
"work_rvu",
"non_fac_pe_rvu",
"mp_rvu",
"non_fac_total",
],
)
.properties(title="Application Code RVUs (Non-Facility)", width=700, height=400)
)
@@ -153,12 +161,19 @@ def _(alt, q, region_picker):
x=alt.X("year:O", title="Year"),
y=alt.Y("non_fac_fee:Q", title="Non-Facility Fee ($)"),
color=alt.Color("description:N", title="Code"),
tooltip=["year", "hcpcs", "description", "non_fac_fee",
"fac_fee", "locality_name"],
tooltip=[
"year",
"hcpcs",
"description",
"non_fac_fee",
"fac_fee",
"locality_name",
],
)
.properties(
title=f"Application Code Fees — {_region}",
width=700, height=400,
width=700,
height=400,
)
)
carrier_chart
@@ -179,9 +194,16 @@ def _(mo):
@app.cell(hide_code=True)
def _(alt, q):
regions = [
"MANHATTAN", "ALASKA*", "REST OF FLORIDA", "REST OF TEXAS",
"REST OF CALIFORNIA", "CHICAGO", "DETROIT", "ATLANTA",
"HAWAII, GUAM", "SOUTH CAROLINA",
"MANHATTAN",
"ALASKA*",
"REST OF FLORIDA",
"REST OF TEXAS",
"REST OF CALIFORNIA",
"CHICAGO",
"DETROIT",
"ATLANTA",
"HAWAII, GUAM",
"SOUTH CAROLINA",
]
region_list = ", ".join(f"'{r}'" for r in regions)
@@ -206,7 +228,8 @@ def _(alt, q):
)
.properties(
title="15271 Non-Facility Fee — Regional Comparison",
width=700, height=400,
width=700,
height=400,
)
)
regional_chart
@@ -259,22 +282,31 @@ def _(alt, pl, q):
alt.Chart(asp_ts.to_pandas())
.mark_line()
.encode(
x=alt.X("quarter:O", title="Quarter",
axis=alt.Axis(labelAngle=-45, labelFontSize=8)),
y=alt.Y("payment_limit:Q", title="Payment Limit (ASP + 6%) $",
scale=alt.Scale(domainMax=800)),
x=alt.X(
"quarter:O",
title="Quarter",
axis=alt.Axis(labelAngle=-45, labelFontSize=8),
),
y=alt.Y(
"payment_limit:Q",
title="Payment Limit (ASP + 6%) $",
scale=alt.Scale(domainMax=800),
),
color=alt.Color("short_description:N", title="Product"),
tooltip=["quarter", "hcpcs_code", "short_description",
"payment_limit", "asp_per_unit"],
tooltip=[
"quarter",
"hcpcs_code",
"short_description",
"payment_limit",
"asp_per_unit",
],
)
)
asp_chart = (
(asp_lines + flat_rate_rule)
.properties(
title="ASP Quarterly Payment Limits — Top 15 Products",
width=700, height=450,
)
asp_chart = (asp_lines + flat_rate_rule).properties(
title="ASP Quarterly Payment Limits — Top 15 Products",
width=700,
height=450,
)
asp_chart
return
@@ -356,11 +388,9 @@ def _(alt, pl, q, region_picker):
""")
# Join to get total episode cost
combined = (
episode_data.join(app_fee, on="year", how="left")
.with_columns(
(pl.col("product_cost_25cm2") + pl.col("application_fee").fill_null(0))
.alias("total_episode_cost")
combined = episode_data.join(app_fee, on="year", how="left").with_columns(
(pl.col("product_cost_25cm2") + pl.col("application_fee").fill_null(0)).alias(
"total_episode_cost"
)
)
@@ -377,18 +407,21 @@ def _(alt, pl, q, region_picker):
x=alt.X("year:O", title="Year"),
y=alt.Y("total_episode_cost:Q", title="Total Episode Cost ($)"),
color=alt.Color("short_description:N", title="Product"),
tooltip=["year", "hcpcs_code", "short_description",
"product_cost_25cm2", "application_fee",
"total_episode_cost"],
tooltip=[
"year",
"hcpcs_code",
"short_description",
"product_cost_25cm2",
"application_fee",
"total_episode_cost",
],
)
)
episode_chart = (
(episode_lines + flat_line)
.properties(
title=f"Total Episode Cost (25cm² wound) — {_region}",
width=700, height=400,
)
episode_chart = (episode_lines + flat_line).properties(
title=f"Total Episode Cost (25cm² wound) — {_region}",
width=700,
height=400,
)
episode_chart
return
@@ -428,16 +461,27 @@ def _(alt, q):
.mark_bar()
.encode(
x=alt.X("delta:Q", title="Payment Limit − $127.28 Flat Rate"),
y=alt.Y("short_description:N", title="", sort="-x",
axis=alt.Axis(labelLimit=300)),
color=alt.Color("impact:N",
scale=alt.Scale(
domain=["loses", "gains", "neutral"],
range=["#d62728", "#2ca02c", "#7f7f7f"],
),
title="Impact"),
tooltip=["hcpcs_code", "short_description",
"payment_limit", "flat_rate", "delta"],
y=alt.Y(
"short_description:N",
title="",
sort="-x",
axis=alt.Axis(labelLimit=300),
),
color=alt.Color(
"impact:N",
scale=alt.Scale(
domain=["loses", "gains", "neutral"],
range=["#d62728", "#2ca02c", "#7f7f7f"],
),
title="Impact",
),
tooltip=[
"hcpcs_code",
"short_description",
"payment_limit",
"flat_rate",
"delta",
],
)
.properties(title="CY2026 Flat-Rate Impact by Product", width=700)
)

114
notebooks/skin_subs_explorer.py
View File

@@ -31,6 +31,7 @@ def _(mo):
def _():
import altair as alt
import polars as pl
from conf import connect
con = connect.duckdb()
@@ -85,7 +86,9 @@ def _(alt, asp, mo, pl, product_select):
filtered.select(["hcpcs_code", "product_name"])
.unique()
.with_columns(
(pl.col("hcpcs_code") + " " + pl.col("product_name").fill_null("")).alias("label")
(pl.col("hcpcs_code") + " " + pl.col("product_name").fill_null("")).alias(
"label"
)
)
)
chart_data = filtered.join(labels, on="hcpcs_code").to_pandas()
@@ -97,8 +100,15 @@ def _(alt, asp, mo, pl, product_select):
x=alt.X("quarter:N", title="Quarter", axis=alt.Axis(labelAngle=-45)),
y=alt.Y("payment_limit:Q", title="Payment Limit ($/cm²)"),
color=alt.Color("label:N", title="Product"),
tooltip=["quarter", "hcpcs_code", "product_name", "manufacturer",
"asp_per_unit", "payment_limit", "qoq_pct_change"],
tooltip=[
"quarter",
"hcpcs_code",
"product_name",
"manufacturer",
"asp_per_unit",
"payment_limit",
"qoq_pct_change",
],
)
.properties(width=700, height=400, title="ASP Payment Limit Over Time")
)
@@ -140,9 +150,17 @@ def _(alt, con, mo, pl):
.encode(
x=alt.X("total_revenue:Q", title="Total Revenue ($)"),
y=alt.Y("manufacturer:N", sort="-x", title=None),
color=alt.Color("avg_asp:Q", scale=alt.Scale(scheme="reds"), title="Avg ASP ($/cm²)"),
tooltip=["manufacturer", "product_count", "active_products",
"total_revenue", "avg_asp", "categories"],
color=alt.Color(
"avg_asp:Q", scale=alt.Scale(scheme="reds"), title="Avg ASP ($/cm²)"
),
tooltip=[
"manufacturer",
"product_count",
"active_products",
"total_revenue",
"avg_asp",
"categories",
],
)
.properties(width=600, height=350, title="Top Manufacturers by Revenue")
)
@@ -165,7 +183,9 @@ def _(alt, con, mo, pl):
.encode(
x=alt.X("products:Q", title="Product Count"),
y=alt.Y("category:N", sort="-x", title=None),
color=alt.Color("avg_asp:Q", scale=alt.Scale(scheme="blues"), title="Avg ASP"),
color=alt.Color(
"avg_asp:Q", scale=alt.Scale(scheme="blues"), title="Avg ASP"
),
tooltip=["category", "products", "avg_asp", "claims"],
)
.properties(width=600, height=300, title="Products by Category")
@@ -197,14 +217,23 @@ def _(alt, con, mo, pl):
x=alt.X("paid_per_bene:Q", title="Paid per Beneficiary ($)"),
y=alt.Y("state:N", sort="-x", title=None),
color=alt.Color("mac_jurisdiction:N", title="MAC Jurisdiction"),
tooltip=["state", "mac_jurisdiction", "unique_benes", "total_paid",
"paid_per_bene", "pct_office", "pct_podiatry"],
tooltip=[
"state",
"mac_jurisdiction",
"unique_benes",
"total_paid",
"paid_per_bene",
"pct_office",
"pct_podiatry",
],
)
.properties(width=600, height=400, title="Spend per Beneficiary by State")
)
mac = pl.from_pandas(
con.execute("SELECT * FROM skin_subs.mac_summary ORDER BY total_paid DESC").fetchdf()
con.execute(
"SELECT * FROM skin_subs.mac_summary ORDER BY total_paid DESC"
).fetchdf()
)
mac_chart = (
@@ -213,10 +242,20 @@ def _(alt, con, mo, pl):
.encode(
x=alt.X("total_paid:Q", title="Total Paid ($)"),
y=alt.Y("mac_jurisdiction:N", sort="-x", title=None),
color=alt.Color("avg_pct_podiatry:Q", scale=alt.Scale(scheme="oranges"),
title="% Podiatry"),
tooltip=["mac_jurisdiction", "states", "claim_lines", "total_paid",
"avg_paid_per_bene", "avg_pct_office", "avg_pct_podiatry"],
color=alt.Color(
"avg_pct_podiatry:Q",
scale=alt.Scale(scheme="oranges"),
title="% Podiatry",
),
tooltip=[
"mac_jurisdiction",
"states",
"claim_lines",
"total_paid",
"avg_paid_per_bene",
"avg_pct_office",
"avg_pct_podiatry",
],
)
.properties(width=600, height=250, title="Total Paid by MAC Jurisdiction")
)
@@ -278,16 +317,27 @@ def _(alt, con, mo, pl):
.encode(
x=alt.X("volume_zscore:Q", title="Volume Z-Score (vs specialty peers)"),
y=alt.Y("intensity_zscore:Q", title="Intensity Z-Score ($/patient)"),
color=alt.Color("risk_tier:N",
scale=alt.Scale(
domain=["critical", "high", "moderate", "low"],
range=["#d62728", "#ff7f0e", "#bcbd22", "#2ca02c"]),
title="Risk Tier"),
color=alt.Color(
"risk_tier:N",
scale=alt.Scale(
domain=["critical", "high", "moderate", "low"],
range=["#d62728", "#ff7f0e", "#bcbd22", "#2ca02c"],
),
title="Risk Tier",
),
size=alt.Size("total_paid:Q", title="Total Paid"),
shape=alt.Shape("primary_setting:N", title="Setting"),
tooltip=["rendering_npi", "provider_specialty", "state",
"primary_setting", "total_paid", "paid_per_patient",
"composite_risk", "risk_tier", "fraud_pattern_match"],
tooltip=[
"rendering_npi",
"provider_specialty",
"state",
"primary_setting",
"total_paid",
"paid_per_patient",
"composite_risk",
"risk_tier",
"fraud_pattern_match",
],
)
.properties(width=650, height=450, title="Provider Risk: Volume vs Intensity")
)
@@ -300,7 +350,9 @@ def _(alt, con, mo, pl):
tier_table = mo.ui.table(tier.to_pandas())
mo.vstack([mo.ui.altair_chart(scatter), mo.md("### Risk Tier Distribution"), tier_table])
mo.vstack(
[mo.ui.altair_chart(scatter), mo.md("### Risk Tier Distribution"), tier_table]
)
return risk, scatter, tier, tier_table
@@ -329,13 +381,15 @@ def _(con, mo, pl):
""").fetchdf()
)
mo.vstack([
mo.md("### Evidence Base by Source"),
mo.ui.table(ev.to_pandas()),
mo.md("### Study Characteristics by Type"),
mo.ui.table(study.to_pandas()),
mo.md(f"**Total evidence items:** {ev['items'].sum():,}"),
])
mo.vstack(
[
mo.md("### Evidence Base by Source"),
mo.ui.table(ev.to_pandas()),
mo.md("### Study Characteristics by Type"),
mo.ui.table(study.to_pandas()),
mo.md(f"**Total evidence items:** {ev['items'].sum():,}"),
]
)
return ev, study

82
notebooks/zotero_tutorial.py
View File

@@ -7,6 +7,7 @@ app = marimo.App(width="medium")
@app.cell(hide_code=True)
def _():
import marimo as mo
mo.md("""
# Zotero Library Explorer
@@ -26,9 +27,12 @@ def _():
@app.cell(hide_code=True)
def _(mo):
import sqlite3
import polars as pl
from pyzotero import zotero
from conf import connect, path as _conf_path
from conf import connect
from conf import path as _conf_path
ZOTERO_DB = _conf_path("db.zotero")
@@ -47,9 +51,12 @@ def _(connect, sqlite3):
db.row_factory = sqlite3.Row
# Quick health check
tables = [r[0] for r in db.execute(
"SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"
).fetchall()]
tables = [
r[0]
for r in db.execute(
"SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"
).fetchall()
]
print(f"Connected — {len(tables)} tables found")
print(f"Tables: {', '.join(tables[:15])}{'...' if len(tables) > 15 else ''}")
return (db,)
@@ -114,17 +121,25 @@ def _(db, mo, pl):
def _build_tree(df, parent_id=None, depth=0):
_rows = df.filter(
pl.col("parentCollectionID") == parent_id if parent_id else pl.col("parentCollectionID").is_null()
pl.col("parentCollectionID") == parent_id
if parent_id
else pl.col("parentCollectionID").is_null()
)
_lines = []
for _row in _rows.iter_rows(named=True):
_indent = " " * depth
_lines.append(f"{_indent}- **{_row['collectionName']}** ({_row['itemCount']} items)")
_lines.append(
f"{_indent}- **{_row['collectionName']}** ({_row['itemCount']} items)"
)
_lines.extend(_build_tree(df, _row["collectionID"], depth + 1))
return _lines
_tree = _build_tree(collections_df) if collections_df.height > 0 else []
mo.md("\n".join(_tree) if _tree else "No collections found. Add some in the Zotero desktop app.")
mo.md(
"\n".join(_tree)
if _tree
else "No collections found. Add some in the Zotero desktop app."
)
return
@@ -155,7 +170,9 @@ def _(db, mo, pl):
connection=db,
)
mo.ui.table(item_types_df, label="Item Types") if item_types_df.height > 0 else mo.md("No library items found yet.")
mo.ui.table(
item_types_df, label="Item Types"
) if item_types_df.height > 0 else mo.md("No library items found yet.")
return
@@ -195,7 +212,9 @@ def _(db, mo, pl):
connection=db,
)
mo.ui.table(recent_df, label="Recent Items") if recent_df.height > 0 else mo.md("No items found. Add references through the Zotero desktop app (VNC).")
mo.ui.table(recent_df, label="Recent Items") if recent_df.height > 0 else mo.md(
"No items found. Add references through the Zotero desktop app (VNC)."
)
return (recent_df,)
@@ -211,7 +230,9 @@ def _(mo):
@app.cell(hide_code=True)
def _(mo):
search_input = mo.ui.text(placeholder="Enter search terms...", label="Search", full_width=True)
search_input = mo.ui.text(
placeholder="Enter search terms...", label="Search", full_width=True
)
search_input
return (search_input,)
@@ -257,7 +278,9 @@ def _(db, mo, pl, search_input):
execute_options={"parameters": {"q": _query}},
)
mo.ui.table(_search_results, label=f"Results for '{_query}'") if _search_results.height > 0 else mo.md(f"No results for **{_query}**")
mo.ui.table(
_search_results, label=f"Results for '{_query}'"
) if _search_results.height > 0 else mo.md(f"No results for **{_query}**")
return
@@ -351,7 +374,9 @@ def _(db, item_id_input, mo):
if _creators:
_sections.append("\n**Creators**\n")
for _c in _creators:
_sections.append(f"- {_c['firstName']} {_c['lastName']} ({_c['creatorType']})")
_sections.append(
f"- {_c['firstName']} {_c['lastName']} ({_c['creatorType']})"
)
if _tags:
_sections.append("\n**Tags**\n")
@@ -392,7 +417,9 @@ def _(db, mo, pl):
connection=db,
)
mo.ui.table(tags_df, label="Tags") if tags_df.height > 0 else mo.md("No tags found.")
mo.ui.table(tags_df, label="Tags") if tags_df.height > 0 else mo.md(
"No tags found."
)
return
@@ -426,7 +453,9 @@ def _(db, mo, pl):
connection=db,
)
mo.ui.table(creators_df, label="Top Creators") if creators_df.height > 0 else mo.md("No creators found.")
mo.ui.table(creators_df, label="Top Creators") if creators_df.height > 0 else mo.md(
"No creators found."
)
return
@@ -501,9 +530,15 @@ def _(mo):
@app.cell(hide_code=True)
def _(mo):
api_key_input = mo.ui.text(placeholder="Zotero API key", label="API Key", kind="password", full_width=True)
library_id_input = mo.ui.text(placeholder="Library ID (numeric)", label="Library ID", full_width=True)
lib_type_input = mo.ui.dropdown(options=["user", "group"], value="user", label="Library Type")
api_key_input = mo.ui.text(
placeholder="Zotero API key", label="API Key", kind="password", full_width=True
)
library_id_input = mo.ui.text(
placeholder="Library ID (numeric)", label="Library ID", full_width=True
)
lib_type_input = mo.ui.dropdown(
options=["user", "group"], value="user", label="Library Type"
)
mo.hstack([library_id_input, lib_type_input, api_key_input], widths=[1, 1, 2])
return api_key_input, lib_type_input, library_id_input
@@ -515,7 +550,9 @@ def _(api_key_input, lib_type_input, library_id_input, mo, zotero):
lib_type = lib_type_input.value
if not api_key or not lib_id:
mo.md("Enter your Zotero API key and library ID above to enable Web API access.")
mo.md(
"Enter your Zotero API key and library ID above to enable Web API access."
)
mo.stop(True)
zot = zotero.Zotero(lib_id, lib_type, api_key)
@@ -534,9 +571,9 @@ def _(api_key_input, lib_type_input, library_id_input, mo, zotero):
mo.md(f"""
Connected to Zotero Web API
- **User**: {_key_info.get('userID', 'N/A')}
- **Key name**: {_key_info.get('key', 'N/A')[:8]}...
- **Access**: {'read/write' if _key_info.get('access', {}).get('user', {}).get('library') else 'read-only'}
- **User**: {_key_info.get("userID", "N/A")}
- **Key name**: {_key_info.get("key", "N/A")[:8]}...
- **Access**: {"read/write" if _key_info.get("access", {}).get("user", {}).get("library") else "read-only"}
""")
return (zot,)
@@ -552,7 +589,8 @@ def _(mo, zot):
_title = _data.get("title", "(untitled)")
_item_type = _data.get("itemType", "?")
_authors = ", ".join(
_cr.get("lastName", _cr.get("name", "?")) for _cr in _data.get("creators", [])
_cr.get("lastName", _cr.get("name", "?"))
for _cr in _data.get("creators", [])
)
print(f"[{_item_type}] {_title}")
if _authors:

17
pyproject.toml
View File

@@ -52,9 +52,15 @@ cli = [
"stack[aco]",
"stack[api]",
"stack[bib]",
"stack[mail]",
"typer>=0.24.1",
"uvicorn>=0.41.0",
]
mail = [
"httpx>=0.28.1",
"pydo>=0.29.0",
"resend>=2.0.0",
]
cms = [
"narwhals>=2.17.0",
"pydantic>=2.0.0",
@@ -84,6 +90,16 @@ rex = [
"pyarrow>=23.0.0",
"fsspec>=2024.1.0",
]
prisma = [
"stack[conf]",
"stack[bib]",
"anthropic>=0.40.0",
"httpx[socks]>=0.28.1",
"pyyaml>=6.0.0",
"pydo>=0.29.0",
"pdfminer.six>=20221105",
"resend>=2.0.0",
]
perf = [
"stack[conf]",
"opentelemetry-api>=1.25.0",
@@ -148,6 +164,7 @@ dev = [
"obstore>=0.9.2",
"s3fs>=2026.2.0",
"jinja2>=3.1.0",
"fastexcel>=0.19.0",
]
[build-system]

219
src/bib/email_ingest.py Normal file
View File

@@ -0,0 +1,219 @@
"""IMAP → bib ingest. Pulls mail from a mailbox and upserts each message
as a ``Source`` item with attachments, suitable for downstream Zotero
sync.
Dedup uses two layers:
* IMAP ``\\Seen`` flag — set on each successful ingest so re-runs only
pull UNSEEN messages.
* ``store.upsert`` keys items by ``url`` and we encode each message's
``Message-ID`` as ``email:<message-id>`` — re-importing the same
message updates instead of duplicating, even if Seen state was lost.
Attachments are extracted via ``email.message.iter_attachments``
(Python 3.6+) and pushed through ``store.attach_file`` so they end up
in the bib storage tree alongside the parent Source row.
CLI: ``stack bib ingest-mail [--user <addr>] [--limit N]``.
"""
from __future__ import annotations
import email
import email.utils
import imaplib
import logging
import re
import ssl
from dataclasses import dataclass
from email.policy import default as default_policy
from pathlib import Path
from typing import TYPE_CHECKING
from bib.item import Source
if TYPE_CHECKING:
from bib.store import Store
log = logging.getLogger(__name__)
@dataclass
class Mailbox:
host: str
port: int
username: str
password: str
folder: str = "INBOX"
def ingest(
store: "Store",
mailbox: Mailbox,
*,
scratch_root: Path = Path(".state/email-ingest"),
limit: int | None = None,
) -> dict[str, int]:
"""Pull UNSEEN messages from *mailbox*, upsert each as a Source.
Returns counts of {seen, ingested, errors, attached}. ``seen``
is the count of UIDs the IMAP server returned as UNSEEN at the
start of the run; the others are subsets.
"""
stats = {"seen": 0, "ingested": 0, "errors": 0, "attached": 0}
ctx = ssl.create_default_context()
conn = imaplib.IMAP4_SSL(mailbox.host, mailbox.port, ssl_context=ctx)
try:
conn.login(mailbox.username, mailbox.password)
conn.select(mailbox.folder)
typ, data = conn.uid("SEARCH", None, "UNSEEN")
if typ != "OK":
log.warning("UNSEEN search failed: %s %s", typ, data)
return stats
uids = (data[0] or b"").split()
if limit:
uids = uids[: int(limit)]
stats["seen"] = len(uids)
for uid in uids:
try:
typ, msg_data = conn.uid("FETCH", uid, "(RFC822)")
if typ != "OK" or not msg_data or not msg_data[0]:
stats["errors"] += 1
continue
raw = msg_data[0][1]
msg = email.message_from_bytes(raw, policy=default_policy)
attached = _ingest_message(store, mailbox, msg, scratch_root)
stats["ingested"] += 1
stats["attached"] += attached
conn.uid("STORE", uid, "+FLAGS", r"(\Seen)")
except Exception as e: # noqa: BLE001
log.warning("ingest uid=%s failed: %s", uid.decode(), e)
stats["errors"] += 1
finally:
try:
conn.close()
except Exception: # noqa: BLE001
pass
try:
conn.logout()
except Exception: # noqa: BLE001
pass
return stats
# ── Per-message ────────────────────────────────────────────────
def _ingest_message(
store: "Store",
mailbox: Mailbox,
msg: email.message.EmailMessage,
scratch_root: Path,
) -> int:
"""Upsert one message + its attachments. Returns attachment count."""
raw_mid = (msg.get("Message-ID") or "").strip().strip("<>")
if not raw_mid:
# Synthesize from headers — same input always yields same key.
raw_mid = "synth:" + _slug(
f"{msg.get('Date', '')}|{msg.get('From', '')}|{msg.get('Subject', '')[:60]}"
)
url = f"email:{raw_mid}"
sender = (msg.get("From") or "").strip()
subject = (msg.get("Subject") or "(no subject)").strip()
sender_domain = _sender_domain(sender)
body = _extract_body(msg)
item = Source(title=subject[:255], url=url)
item.doc_type = "Email"
item.institution = sender_domain or "Email"
item.date_published = _date_iso(msg.get("Date", ""))
item.abstract = (body or "").strip()[:4000]
# Tags. mailbox tag uses the local-part so multi-mailbox setups can
# filter at the store level (mailbox:cmsupdates, mailbox:postmaster).
item.add_tag("source:email")
mailbox_local = mailbox.username.split("@", 1)[0]
item.add_tag(f"mailbox:{_slug(mailbox_local)}")
if sender_domain:
item.add_tag(f"sender:{_slug(sender_domain)}")
list_id = (msg.get("List-ID") or msg.get("List-Id") or "").strip().strip("<>")
if list_id:
item.add_tag(f"list:{_slug(list_id)}")
if item.date_published:
item.add_tag(f"year:{item.date_published[:4]}")
key = store.upsert(item)
# Attachments → temp dir → attach_file
attached = 0
msg_scratch = scratch_root / _slug(raw_mid)[:60]
for part in msg.iter_attachments():
try:
filename = (part.get_filename() or "").strip()
if not filename:
# Use a content-type-derived filename so multi-attachment
# messages without filenames still distinct.
ext = (part.get_content_subtype() or "bin").split(";")[0]
filename = f"part-{attached + 1}.{ext}"
payload = part.get_payload(decode=True)
if not payload:
continue
msg_scratch.mkdir(parents=True, exist_ok=True)
dest = msg_scratch / _safe_filename(filename)
dest.write_bytes(payload)
store.attach_file(key, dest, title=filename)
attached += 1
except Exception as e: # noqa: BLE001
log.warning("attachment '%s' for %s: %s", part.get_filename(), raw_mid, e)
return attached
# ── Helpers ────────────────────────────────────────────────────
def _extract_body(msg: email.message.EmailMessage) -> str:
"""Prefer text/plain; fall back to HTML→text."""
for part in msg.walk():
if part.get_content_type() == "text/plain":
try:
return part.get_content()
except (KeyError, LookupError):
continue
for part in msg.walk():
if part.get_content_type() == "text/html":
try:
html = part.get_content()
# Crude tag-strip; real HTML→text would pull bs4, not
# worth the dep for body previews.
return re.sub(r"<[^>]+>", " ", html)
except (KeyError, LookupError):
continue
return ""
def _sender_domain(sender: str) -> str:
m = re.search(r"<([^@<>]+)@([^>]+)>", sender)
if m:
return m.group(2)
m = re.search(r"@([^\s]+)", sender)
return m.group(1) if m else ""
def _slug(s: str) -> str:
return re.sub(r"[^a-z0-9]+", "-", (s or "").lower()).strip("-")[:60]
def _safe_filename(s: str) -> str:
return re.sub(r"[^A-Za-z0-9._-]+", "_", s)[:200] or "attachment"
def _date_iso(date_str: str) -> str:
if not date_str:
return ""
try:
dt = email.utils.parsedate_to_datetime(date_str)
return dt.date().isoformat()
except Exception: # noqa: BLE001
return ""

193
src/bib/federalregister.py Normal file
View File

@@ -0,0 +1,193 @@
"""Federal Register API client — rule discovery.
Thin wrapper over ``federalregister.gov/api/v1/documents``. Use it to
enumerate every rule matching a pattern (e.g. all PFS proposed + final
rules since 2017), then hand the document numbers off to
:func:`bib.translate.federal_register` or to
:mod:`bib.regulations_gov` for comment retrieval.
API docs: https://www.federalregister.gov/reader-aids/developer-resources/rest-api-documentation
No auth required. Rate limit: ~60 req/min. We batch with ``per_page=100``
which is the max the API returns.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass
from typing import Iterator
import httpx
log = logging.getLogger(__name__)
_BASE = "https://www.federalregister.gov/api/v1"
# PFS (CY NNNN Physician Fee Schedule, Medicare Part B) proposed and
# final rules. Title wording shifted over the years:
#
# 2017–2019 — "Revisions to Payment Policies Under the Physician Fee Schedule"
# 2020+ — "CY <YYYY> Payment Policies Under the Physician Fee Schedule"
#
# Both forms share "Payment Policies Under the Physician Fee Schedule".
# Use that substring to match every era without sweeping in unrelated
# rules that only mention the phrase "physician fee schedule" in passing.
PFS_TITLE_PATTERN = "Payment Policies Under the Physician Fee Schedule"
@dataclass
class FRDocument:
"""Subset of a Federal Register document we care about for rule
discovery + comment retrieval."""
document_number: str
title: str
type: str # "Rule" (final) | "Proposed Rule"
publication_date: str # YYYY-MM-DD
dockets: list[str] # e.g. ["CMS-1676-P"]
volume: str = ""
start_page: str = ""
regulation_id_numbers: list[str] | None = None
html_url: str = ""
# ── Search ──────────────────────────────────────────────────────
def search(
*,
term: str = "",
agencies: list[str] | None = None,
types: list[str] | None = None,
since: str | None = None,
until: str | None = None,
per_page: int = 100,
client: httpx.Client | None = None,
) -> Iterator[FRDocument]:
"""Yield every FR document matching the given filters, paginating
through every page.
Parameters mirror the API's ``conditions[…]`` keys. ``types`` values
use the API's enum: ``"RULE"`` (final rules) and ``"PRORULE"``
(proposed rules).
"""
own_client = client is None
client = client or httpx.Client(timeout=30)
try:
params: list[tuple[str, str]] = [
("per_page", str(per_page)),
("order", "oldest"),
*_fields(),
]
if term:
params.append(("conditions[term]", term))
for a in agencies or []:
params.append(("conditions[agencies][]", a))
for t in types or []:
params.append(("conditions[type][]", t))
if since:
params.append(("conditions[publication_date][gte]", since))
if until:
params.append(("conditions[publication_date][lte]", until))
page = 1
while True:
paged = params + [("page", str(page))]
r = client.get(f"{_BASE}/documents", params=paged)
r.raise_for_status()
data = r.json()
for result in data.get("results", []):
yield _to_doc(result)
if page >= data.get("total_pages", 1):
break
page += 1
finally:
if own_client:
client.close()
def split_docket_ids(raw: list[str]) -> list[str]:
"""Clean the FR API's docket_ids list into individual CMS-XXXX-Y ids.
The API sometimes returns a single-element list whose only entry is
a human-written aggregate like::
["CMS-1693-F, CMS-1693-IFC, CMS-5522-F3, and CMS-1701-F"]
Split on commas + "and" to recover the atoms. Dedup preserving order.
"""
import re
out: list[str] = []
seen: set[str] = set()
for entry in raw:
for atom in re.split(r",\s*|\s+and\s+", entry):
atom = atom.strip()
if atom and atom not in seen:
seen.add(atom)
out.append(atom)
return out
def pfs_rules(
*,
since: str = "2017-01-01",
until: str | None = None,
client: httpx.Client | None = None,
) -> list[FRDocument]:
"""Every PFS proposed/final rule since *since* (inclusive).
Title-filtered so we get the true annual PFS rules, not every CMS
rule that mentions PFS in passing.
"""
found: list[FRDocument] = []
seen: set[str] = set()
for doc in search(
term="physician fee schedule",
agencies=["centers-for-medicare-medicaid-services"],
types=["RULE", "PRORULE"],
since=since,
until=until,
client=client,
):
if PFS_TITLE_PATTERN.lower() not in doc.title.lower():
continue
if doc.document_number in seen:
continue
seen.add(doc.document_number)
found.append(doc)
return found
# ── Internals ───────────────────────────────────────────────────
def _fields() -> list[tuple[str, str]]:
"""Fields we need from the API — smaller payloads, fewer timeouts."""
names = [
"document_number",
"title",
"type",
"publication_date",
"docket_ids",
"volume",
"start_page",
"regulation_id_numbers",
"html_url",
]
return [("fields[]", n) for n in names]
def _to_doc(raw: dict) -> FRDocument:
return FRDocument(
document_number=raw.get("document_number", ""),
title=raw.get("title", ""),
type=raw.get("type", ""),
publication_date=raw.get("publication_date", ""),
dockets=list(raw.get("docket_ids") or []),
volume=str(raw.get("volume") or ""),
start_page=str(raw.get("start_page") or ""),
regulation_id_numbers=list(raw.get("regulation_id_numbers") or []),
html_url=raw.get("html_url", ""),
)

446
src/bib/iom.py Normal file
View File

@@ -0,0 +1,446 @@
"""CMS Internet-Only Manual (IOM) index crawler.
Walks the public CMS IOM index page, enumerates every manual, and for
each one discovers its chapter PDFs from the corresponding landing
page. Produces :class:`bib.item.Manual` records and idempotently
upserts them via the bibliography store.
The scraper is HTML-regex based — CMS's IOM pages have very stable
structure (every chapter sits inside a ``<ul class="field__items">``
with an anchor pointing to ``…/manuals/downloads/<prefix><pub>c<NN>.pdf``)
— so the heavier HTML parsers pull their weight elsewhere.
Usage::
from bib import connect
from bib.iom import ingest_all
import httpx
store = connect()
with httpx.Client() as client:
summary = ingest_all(store, client)
# {"100-02": 18, "100-04": 38, ...}
Or from the CLI: ``uv run stack bib ingest-iom``.
"""
from __future__ import annotations
import hashlib
import html
import json
import logging
import re
from dataclasses import dataclass
from pathlib import Path
from typing import TYPE_CHECKING, Iterable
from bib.item import Download
from bib.tag import Tag
from bib.translate import cms_manual
if TYPE_CHECKING:
import httpx
from bib.store import Store
log = logging.getLogger(__name__)
INDEX_URL = "https://www.cms.gov/medicare/regulations-guidance/manuals/internet-only-manuals-ioms"
BASE = "https://www.cms.gov"
# CMS publishes upcoming transmittal schedules here. We poll it to detect
# IOM changes between crawls — any checksum change is a signal to
# re-ingest affected manuals.
FUTURE_PDF_URL = "https://www.cms.gov/regulations-and-guidance/guidance/manuals/downloads/futurepdf.pdf"
# Publication-number → filename prefix used by CMS download URLs.
# Discovered by inspecting each IOM's landing page. Not authoritative
# — the scraper doesn't depend on this map (it trusts the anchor hrefs
# directly) but it makes post-hoc URL parsing possible elsewhere.
PUB_PREFIXES: dict[str, str] = {
"100-01": "ge101",
"100-02": "bp102",
"100-03": "ncd103",
"100-04": "clm104",
"100-05": "msp105",
"100-06": "fin106",
"100-07": "som107",
"100-08": "pim83", # CMS uses pim83c, not pim108
"100-09": "qio109",
"100-10": "bp102", # Hospice shares file stem in practice; varies
"100-11": "hha111",
# Newer pubs (100-12..100-25) vary in prefix; we rely on anchor
# hrefs for those.
}
@dataclass(frozen=True)
class IOMEntry:
"""One row on the IOM index page."""
pub: str # e.g. "100-02"
title: str # e.g. "Medicare Benefit Policy Manual"
landing_url: str
@dataclass(frozen=True)
class ChapterEntry:
"""One chapter PDF discovered on an IOM landing page."""
pub: str
manual: str # display name from index page
chapter: str # "1", "6A", …
title: str # "Inpatient Hospital Services Covered Under Part A"
url: str
# ── Scraping ──────────────────────────────────────────────────────
_INDEX_ROW = re.compile(
r'<a\s+href="(?P<href>/regulations-and-guidance/guidance/manuals/'
r'internet-only-manuals-ioms-items/cms\d+)"[^>]*>\s*'
r"(?P<pub>100(?:-\d+)?)\s*</a>"
r".*?Title</label>\s*(?P<title>[^<]+?)\s*</div>",
re.DOTALL | re.IGNORECASE,
)
def fetch_index(client: httpx.Client) -> list[IOMEntry]:
"""Scrape the IOM index table."""
r = client.get(INDEX_URL, follow_redirects=True, timeout=30)
r.raise_for_status()
seen: set[str] = set()
out: list[IOMEntry] = []
for m in _INDEX_ROW.finditer(r.text):
pub = m.group("pub").strip()
if pub in seen:
continue
seen.add(pub)
title = html.unescape(re.sub(r"\s+", " ", m.group("title"))).strip()
out.append(
IOMEntry(
pub=pub,
title=title,
landing_url=BASE + m.group("href"),
)
)
return out
_CHAPTER_LINK = re.compile(
r'<a[^>]*href="(?P<href>/regulations-and-guidance/guidance/manuals/'
r'downloads/[^"]+\.pdf)"[^>]*>\s*(?P<label>[^<]+?)\s*</a>',
re.IGNORECASE,
)
_CHAPTER_HEAD = re.compile(
r"""^(?:Chapter|Ch\.)\s+
(?P<num>\d+[A-Za-z]?) # chapter number, optional trailing letter
(?:\s*[-–—:.,]\s*(?P<rest>.*))?
$""",
re.IGNORECASE | re.VERBOSE,
)
# Some manuals (e.g. NCD 100-03) split a single chapter across multiple
# "Part N" PDFs. Capture the part so we can disambiguate the dedup key.
_CHAPTER_PART = re.compile(r"\bPart\s+(?P<part>\d+)\b", re.IGNORECASE)
_SUPPLEMENT_HINTS = re.compile(
r"crosswalk|addendum|appendix|transmittal|revision\s+history|old\s+version"
r"|table\s+of\s+contents|archive|transition|change\s+request|redline",
re.IGNORECASE,
)
# "Pub 100-18 - Medicare Prescription Drug Benefit Manual" — single-file
# manuals that don't split into chapters. We ingest them as one record
# with chapter="" so the whole-PDF content is still discoverable.
_WHOLE_PUB = re.compile(
r"^(?:Pub\s*100[_-]?\d+|Pub100_\d+)\b\s*[-–—:.]?\s*(?P<rest>.*)$",
re.IGNORECASE,
)
def fetch_chapters(client: httpx.Client, entry: IOMEntry) -> list[ChapterEntry]:
"""Extract the chapter list from one IOM's landing page.
Skips crosswalks, addenda, transmittals — the chapter PDFs are the
source-of-truth for content; supplements get their own issues if
they're needed later. Single-file manuals (no per-chapter PDFs) are
captured as one ChapterEntry with chapter="".
"""
r = client.get(entry.landing_url, follow_redirects=True, timeout=30)
r.raise_for_status()
chapters: list[ChapterEntry] = []
whole: list[ChapterEntry] = []
seen: set[str] = set()
for m in _CHAPTER_LINK.finditer(r.text):
label = html.unescape(re.sub(r"\s+", " ", m.group("label"))).strip()
if _SUPPLEMENT_HINTS.search(label):
continue
url = BASE + m.group("href")
head = _CHAPTER_HEAD.match(label)
if head:
num = head.group("num")
rest = (head.group("rest") or "").strip()
part_match = _CHAPTER_PART.search(rest)
chapter_id = f"{num}P{part_match.group('part')}" if part_match else num
key = chapter_id.upper()
if key in seen:
continue
seen.add(key)
chapters.append(
ChapterEntry(
pub=entry.pub,
manual=entry.title,
chapter=chapter_id,
title=rest or label,
url=url,
)
)
continue
whole_m = _WHOLE_PUB.match(label)
if whole_m:
whole.append(
ChapterEntry(
pub=entry.pub,
manual=entry.title,
chapter="",
title=whole_m.group("rest").strip() or entry.title,
url=url,
)
)
# Chapters take precedence; fall back to a single whole-manual
# record when the landing page doesn't expose per-chapter PDFs.
return chapters if chapters else whole
# ── Orchestration ────────────────────────────────────────────────
def _short_manual_name(full_title: str) -> str:
"""Drop "Medicare" prefix and "Manual" suffix for the stored name."""
s = re.sub(r"^Medicare\s+", "", full_title.strip())
s = re.sub(r"\s+Manual\s*$", "", s)
return s.strip()
def ingest_entry(
store: Store,
client: httpx.Client,
entry: IOMEntry,
*,
prune: bool = True,
) -> list[str]:
"""Ingest a single IOM. Returns the store keys of upserted items.
With ``prune=True`` (default), any prior Manual records for this
pub whose URLs don't appear in the freshly-discovered set are
deleted. That removes legacy dupes caused by CMS renaming chapter
filenames (e.g. ``clm104c18pdf.pdf`` → ``clm104c18.pdf``).
"""
keys: list[str] = []
short = _short_manual_name(entry.title)
fresh = fetch_chapters(client, entry)
fresh_urls = {c.url for c in fresh}
for ch in fresh:
title = (
f"{entry.title} — Chapter {ch.chapter}: {ch.title}"
if ch.chapter
else f"{entry.title}: {ch.title}"
)
item = cms_manual(ch.url, manual=short, chapter=ch.chapter)
item.title = title
item.pub_number = entry.pub
item.add_tag(Tag.source("iom").label)
item.add_tag(f"pub:{entry.pub}")
keys.append(store.upsert(item))
if prune:
_prune_stale(store, entry.pub, fresh_urls)
return keys
def _prune_stale(store: Store, pub: str, live_urls: set[str]) -> int:
"""Delete Manual rows for this pub whose URL isn't in the live set.
Uses the store's sqlite connection directly because the public API
doesn't expose a "list by pub_number" helper.
"""
con = store._con() # noqa: SLF001 — internal access by design here
rows = con.execute(
"""
SELECT key, url FROM items
WHERE item_type='manual'
AND json_extract(extra_json, '$.pub_number') = ?
""",
(pub,),
).fetchall()
stale = [r["key"] for r in rows if r["url"] and r["url"] not in live_urls]
for key in stale:
store.delete(key)
if stale:
log.info("pruned %d stale record(s) for pub %s", len(stale), pub)
return len(stale)
def ingest_all(
store: Store,
client: httpx.Client,
*,
pubs: Iterable[str] | None = None,
prune: bool = True,
) -> dict[str, int]:
"""Walk the IOM index and upsert every chapter (or just the filter).
Parameters
----------
pubs : iterable of pub numbers (e.g. ``["100-02", "100-08"]``) to
limit ingestion to a subset. Defaults to every IOM on the index.
Returns
-------
dict
Mapping of pub number → count of chapters upserted.
"""
wanted = set(pubs) if pubs else None
summary: dict[str, int] = {}
for entry in fetch_index(client):
if wanted is not None and entry.pub not in wanted:
continue
try:
keys = ingest_entry(store, client, entry, prune=prune)
except Exception as e: # noqa: BLE001 — one manual failing shouldn't stop the others
log.warning("IOM %s (%s) failed: %s", entry.pub, entry.title, e)
summary[entry.pub] = 0
continue
log.info("IOM %s (%s): %d chapters", entry.pub, entry.title, len(keys))
summary[entry.pub] = len(keys)
return summary
# ── Attachments ──────────────────────────────────────────────────
def _sha256(path: Path) -> str:
h = hashlib.sha256()
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
h.update(chunk)
return h.hexdigest()
def _existing_attachment_hash(store: Store, item_key: str) -> str | None:
"""SHA-256 of an item's first already-attached file, or None."""
con = store._con() # noqa: SLF001
row = con.execute(
"""SELECT a.storage_path FROM attachments a
JOIN items i ON a.item_id = i.id
WHERE i.key = ? ORDER BY a.id LIMIT 1""",
(item_key,),
).fetchone()
if not row or not row["storage_path"]:
return None
p = Path(row["storage_path"])
return _sha256(p) if p.is_file() else None
def download_attachments(
store: Store,
client: httpx.Client,
*,
pubs: Iterable[str] | None = None,
force: bool = False,
tmp_dir: Path | None = None,
) -> dict[str, int]:
"""Download every Manual's PDF and attach it via Store.attach_file.
Idempotent: the remote file's SHA-256 is compared against the
currently-attached copy, and the download is skipped if unchanged.
Returns ``{pub: count_attached}``.
"""
wanted = set(pubs) if pubs else None
tmp = tmp_dir or (Path(store._db_path).parent / "tmp-iom") # noqa: SLF001
tmp.mkdir(parents=True, exist_ok=True)
results: dict[str, int] = {}
items = store.list_items(item_type="manual")
for item in items:
ej = json.loads(item.to_row().get("extra_json", "{}"))
pub = ej.get("pub_number", "")
if wanted is not None and pub not in wanted:
continue
url = item.url
if not url:
continue
filename = url.rsplit("/", 1)[-1]
tmp_path = tmp / filename
try:
r = client.get(url, follow_redirects=True, timeout=60)
r.raise_for_status()
tmp_path.write_bytes(r.content)
except Exception as e: # noqa: BLE001
log.warning("download failed: %s — %s", url, e)
continue
remote_hash = _sha256(tmp_path)
if not force and _existing_attachment_hash(store, item.key) == remote_hash:
tmp_path.unlink(missing_ok=True)
continue
store.attach_file(item.key, tmp_path, title=item.title)
tmp_path.unlink(missing_ok=True)
results[pub] = results.get(pub, 0) + 1
log.info("attached: %s (%s)", filename, pub or "?")
return results
# ── Future-updates watcher ──────────────────────────────────────
_FUTURE_ITEM_KEY_TAG = "iom:futurepdf"
def _futurepdf_anchor(store: Store) -> str:
"""Return the key of the anchor Download item tracking futurepdf.pdf.
Creates one on first call. We use a tag rather than a stable key so
the item can live alongside the normal collection hierarchy.
"""
existing = store.list_items(tag=_FUTURE_ITEM_KEY_TAG)
if existing:
return existing[0].key
item = Download(
title="CMS IOM — Future Transmittal Schedule (futurepdf.pdf)",
url=FUTURE_PDF_URL,
)
item.add_tag(_FUTURE_ITEM_KEY_TAG)
item.add_tag(Tag.source("iom").label)
item.add_tag("pub:futurepdf")
return store.upsert(item)
def check_future_updates(
store: Store,
client: httpx.Client,
*,
tmp_dir: Path | None = None,
) -> tuple[bool, str]:
"""Poll CMS futurepdf.pdf; attach if changed; return (changed, sha256).
Pass this into a cron/daemon. When ``changed`` is True, run
``ingest_all`` + ``download_attachments`` to pick up the new chapters
CMS is advertising on its schedule.
"""
anchor_key = _futurepdf_anchor(store)
tmp = tmp_dir or (Path(store._db_path).parent / "tmp-iom") # noqa: SLF001
tmp.mkdir(parents=True, exist_ok=True)
path = tmp / "futurepdf.pdf"
r = client.get(FUTURE_PDF_URL, follow_redirects=True, timeout=60)
r.raise_for_status()
path.write_bytes(r.content)
new_hash = _sha256(path)
prior = _existing_attachment_hash(store, anchor_key)
changed = prior != new_hash
if changed:
store.attach_file(anchor_key, path, title=f"futurepdf.pdf @ {new_hash[:8]}")
log.info("futurepdf.pdf CHANGED — prior=%s new=%s", prior, new_hash)
path.unlink(missing_ok=True)
return changed, new_hash

309
src/bib/oig.py Normal file
View File

@@ -0,0 +1,309 @@
"""HHS OIG compliance guidance + fraud alert crawler.
Walks OIG's public CPG and alerts indexes, extracts every PDF/HTML
guidance document, and upserts them as :class:`bib.item.Source` records
tagged with ``agency:oig`` plus the specific guidance type.
Covered sources:
- ``/compliance/compliance-guidance/`` — sector-specific CPGs, supplemental
CPGs, Industry-Specific CPGs (ICPGs), drafts.
- ``/compliance/alerts/`` — Special Fraud Alerts, Advisory Bulletins,
Enforcement Alerts, Advisory Opinions that live on the alerts page.
Usage mirrors :mod:`bib.iom`::
from bib import connect
from bib.oig import ingest_all, download_attachments
import httpx
store = connect()
with httpx.Client() as client:
ingest_all(store, client)
download_attachments(store, client)
"""
from __future__ import annotations
import hashlib
import html
import logging
import re
from dataclasses import dataclass
from pathlib import Path
from typing import TYPE_CHECKING, Iterable
from bib.item import Source
from bib.tag import Tag
if TYPE_CHECKING:
import httpx
from bib.store import Store
log = logging.getLogger(__name__)
BASE = "https://oig.hhs.gov"
CPG_URL = f"{BASE}/compliance/compliance-guidance/"
ALERTS_URL = f"{BASE}/compliance/alerts/"
@dataclass(frozen=True)
class OIGDoc:
"""One scraped OIG guidance record."""
title: str # human-readable title from anchor text
url: str # absolute doc URL
guidance_type: str # cpg | sfa | sab | ea | enforcement | other
sector: str = "" # "hospitals", "dme", "nursing", …
# ── Heuristics ───────────────────────────────────────────────────
# Classifier: title → guidance_type. Order matters (most specific first).
_TYPE_RULES: list[tuple[re.Pattern[str], str]] = [
(re.compile(r"special\s+fraud\s+alert", re.I), "sfa"),
(re.compile(r"special\s+advisory\s+bulletin", re.I), "sab"),
(re.compile(r"advisory\s+bulletin", re.I), "sab"),
(re.compile(r"enforcement\s+alert", re.I), "ea"),
(re.compile(r"\b(?:compliance\s+program\s+guidance|cpg|icpg|gcpg)\b", re.I), "cpg"),
(re.compile(r"open\s+letter", re.I), "open_letter"),
]
# Sector tagger: crude keyword match on titles. Unknown → "".
_SECTOR_KEYWORDS: dict[str, str] = {
"hospital": "hospitals",
"nursing": "nursing",
"hospice": "hospice",
"ambulance": "ambulance",
"home health": "home_health",
"home care": "home_health",
"pharmaceutical": "pharmaceutical",
"clinical laborator": "labs",
"durable medical equipment": "dme",
"dme": "dme",
"physician": "physicians",
"medicare advantage": "medicare_advantage",
"medicare+choice": "medicare_advantage",
"third-party medical billing": "billing",
"pharmacy benefit": "pbm",
}
# Skip hints: labels that point at navigation, category pages, or
# non-document links.
_SKIP_HINTS = re.compile(
r"^(?:download|view|click\s+here|\s*-\s*|more\s+information|read\s+more)\s*$",
re.IGNORECASE,
)
def _classify_type(title: str, href: str) -> str:
for pat, kind in _TYPE_RULES:
if pat.search(title):
return kind
if "/special-fraud-alerts/" in href:
return "sfa"
if "/special-advisory-bulletins/" in href:
return "sab"
if "/compliance-guidance/" in href:
return "cpg"
return "other"
def _classify_sector(title: str) -> str:
low = title.lower()
for kw, tag in _SECTOR_KEYWORDS.items():
if kw in low:
return tag
return ""
# ── Scraping ─────────────────────────────────────────────────────
# OIG PDFs live under /documents/... (compliance-guidance, special-fraud-alerts,
# special-advisory-bulletins, root/…). HTML articles under /documents/... .html
# also count — we ingest them the same way.
_DOC_LINK = re.compile(
r'<a[^>]*href="(?P<href>/documents/[^"]+?\.(?:pdf|html?))"[^>]*>'
r"\s*(?P<label>[^<]+?)\s*</a>",
re.IGNORECASE,
)
def _fetch(client: httpx.Client, url: str) -> str:
r = client.get(url, follow_redirects=True, timeout=30)
r.raise_for_status()
return r.text
def _extract_docs(page_html: str) -> list[tuple[str, str]]:
"""Return [(href, clean_title), …] for every doc link on a page."""
out: list[tuple[str, str]] = []
seen_hrefs: set[str] = set()
for m in _DOC_LINK.finditer(page_html):
href = m.group("href")
if href in seen_hrefs:
continue
label = html.unescape(re.sub(r"\s+", " ", m.group("label"))).strip()
if not label or _SKIP_HINTS.match(label):
continue
seen_hrefs.add(href)
out.append((href, label))
return out
def fetch_cpgs(client: httpx.Client) -> list[OIGDoc]:
"""Scrape the OIG Compliance Program Guidance index."""
docs: list[OIGDoc] = []
for href, title in _extract_docs(_fetch(client, CPG_URL)):
docs.append(
OIGDoc(
title=title,
url=BASE + href,
guidance_type=_classify_type(title, href),
sector=_classify_sector(title),
)
)
return docs
def fetch_alerts(client: httpx.Client) -> list[OIGDoc]:
"""Scrape the OIG Special Fraud Alerts / Advisory Bulletins index."""
docs: list[OIGDoc] = []
for href, title in _extract_docs(_fetch(client, ALERTS_URL)):
docs.append(
OIGDoc(
title=title,
url=BASE + href,
guidance_type=_classify_type(title, href),
sector=_classify_sector(title),
)
)
return docs
# ── Ingestion ────────────────────────────────────────────────────
def _build_item(doc: OIGDoc) -> Source:
item = Source(title=doc.title, url=doc.url)
item.doc_type = {
"cpg": "Compliance Program Guidance",
"sfa": "Special Fraud Alert",
"sab": "Special Advisory Bulletin",
"ea": "Enforcement Alert",
"open_letter": "Open Letter",
"other": "OIG Guidance",
}.get(doc.guidance_type, "OIG Guidance")
item.institution = (
"U.S. Department of Health and Human Services, Office of Inspector General"
)
item.add_tag("agency:oig")
item.add_tag(f"guidance:{doc.guidance_type}")
if doc.sector:
item.add_tag(f"sector:{doc.sector}")
item.add_tag(Tag.source("oig").label)
return item
def ingest_all(
store: Store,
client: httpx.Client,
*,
kinds: Iterable[str] | None = None,
) -> dict[str, int]:
"""Crawl OIG CPG + alerts indexes and upsert every document.
Parameters
----------
kinds : iterable, optional
Restrict to a subset of sources. Accepted values: ``"cpg"``,
``"alerts"``. Defaults to both.
Returns
-------
dict
``{"cpg": n, "alerts": n}`` counts of upserted records.
"""
wanted = set(kinds) if kinds else {"cpg", "alerts"}
summary = {"cpg": 0, "alerts": 0}
if "cpg" in wanted:
for doc in fetch_cpgs(client):
store.upsert(_build_item(doc))
summary["cpg"] += 1
log.info("OIG CPG: %d docs", summary["cpg"])
if "alerts" in wanted:
for doc in fetch_alerts(client):
store.upsert(_build_item(doc))
summary["alerts"] += 1
log.info("OIG alerts: %d docs", summary["alerts"])
return summary
# ── Attachments (PDF/HTML) ──────────────────────────────────────
def _sha256(path: Path) -> str:
h = hashlib.sha256()
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
h.update(chunk)
return h.hexdigest()
def _existing_attachment_hash(store: Store, item_key: str) -> str | None:
con = store._con() # noqa: SLF001
row = con.execute(
"""SELECT a.storage_path FROM attachments a
JOIN items i ON a.item_id = i.id
WHERE i.key = ? ORDER BY a.id LIMIT 1""",
(item_key,),
).fetchone()
if not row or not row["storage_path"]:
return None
p = Path(row["storage_path"])
return _sha256(p) if p.is_file() else None
def download_attachments(
store: Store,
client: httpx.Client,
*,
force: bool = False,
) -> int:
"""Download and attach every OIG document we've ingested.
Idempotent on SHA-256. Returns count of new/changed attachments.
"""
tmp = Path(store._db_path).parent / "tmp-oig" # noqa: SLF001
tmp.mkdir(parents=True, exist_ok=True)
items = store.list_items(tag="agency:oig")
changed = 0
for item in items:
url = item.url
if not url:
continue
filename = url.rsplit("/", 1)[-1]
tmp_path = tmp / filename
try:
r = client.get(url, follow_redirects=True, timeout=60)
r.raise_for_status()
tmp_path.write_bytes(r.content)
except Exception as e: # noqa: BLE001
log.warning("download failed: %s — %s", url, e)
continue
remote_hash = _sha256(tmp_path)
if not force and _existing_attachment_hash(store, item.key) == remote_hash:
tmp_path.unlink(missing_ok=True)
continue
store.attach_file(item.key, tmp_path, title=item.title)
tmp_path.unlink(missing_ok=True)
changed += 1
log.info("attached: %s", filename)
return changed

547
src/bib/regulations_gov.py Normal file
View File

@@ -0,0 +1,547 @@
"""regulations.gov v4 API client — public comments + attachments.
Walks every comment posted to a docket and (optionally) fetches each
comment's attached files (PDF / DOCX). Results are upserted to the bib
store as ``Source`` records tagged ``source:regulations-gov``,
``doctype:comment``, ``docket:<id>``, and ``rule:<cms-id>``.
API docs: https://open.gsa.gov/api/regulationsgov/
Rate limit: 1000 req/hr, 50 req/min. Keep ``--sleep`` above 1.25s for
safety; bursty short-runs are fine.
Reference endpoints::
GET /v4/documents?filter[docketId]=CMS-1676-P
GET /v4/comments?filter[commentOnId]=<documentId>&page[size]=250
GET /v4/comments/{id}?include=attachments
"""
from __future__ import annotations
import hashlib
import logging
import os
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import TYPE_CHECKING, Iterator
import httpx
from bib.item import Source
from bib.tag import Tag
if TYPE_CHECKING:
from bib.store import Store
log = logging.getLogger(__name__)
_BASE = "https://api.regulations.gov/v4"
# ── Value objects ──────────────────────────────────────────────
@dataclass
class Comment:
"""One row returned by /v4/comments — flattened."""
id: str
title: str
posted_date: str
received_date: str
docket_id: str
comment_on_id: str # FR document this comment replies to
first_name: str = ""
last_name: str = ""
organization: str = ""
comment_text: str = "" # inline text; large comments go to attachments
attachment_count: int = 0
raw: dict = field(default_factory=dict)
@dataclass
class Attachment:
url: str
filename: str
content_type: str = ""
size: int | None = None
# ── Client ─────────────────────────────────────────────────────
class Client:
"""Thin httpx wrapper honoring the API key + burst rate limit."""
def __init__(
self,
api_key: str | None = None,
*,
sleep: float = 1.3, # ~46 req/min — under the 50/min burst cap
client: httpx.Client | None = None,
) -> None:
key = api_key or os.environ.get("REGULATIONS_GOV_API_KEY")
if not key:
raise RuntimeError(
"REGULATIONS_GOV_API_KEY not set — stack/.env should carry it."
)
self._key = key
self._sleep = sleep
self._owned = client is None
self._client = client or httpx.Client(
timeout=30,
headers={"X-Api-Key": key, "User-Agent": "stack-bib/1.0"},
)
def close(self) -> None:
if self._owned:
self._client.close()
def __enter__(self): # noqa: D401
return self
def __exit__(self, *exc) -> None:
self.close()
def _get(self, path: str, **params) -> dict:
time.sleep(self._sleep)
r = self._client.get(f"{_BASE}{path}", params=params)
# Back off hard on rate-limit 429s; retry once.
if r.status_code == 429:
log.warning("rate limited — sleeping 60s and retrying once")
time.sleep(60)
r = self._client.get(f"{_BASE}{path}", params=params)
r.raise_for_status()
return r.json()
# ── Discovery ──────────────────────────────────────────────
def resolve_docket(self, cms_rule_id: str) -> str | None:
"""Map a CMS rule ID (e.g. ``CMS-1676-P``) to its regulations.gov
docket (e.g. ``CMS-2017-0092``).
Federal Register's ``docket_ids`` field returns the CMS rule
identifier — not the reg.gov tracking number. Reg.gov's
``/documents`` search with the CMS rule ID as ``searchTerm``
surfaces the matching document, whose ``attributes.docketId``
is the real docket we need for comment pulls.
Returns None when no document matches — happens for rules that
were never opened for public comment (corrections, IFCs).
"""
data = self._get(
"/documents",
**{
"filter[searchTerm]": cms_rule_id,
"filter[agencyId]": "CMS",
"page[size]": 5,
},
)
for row in data.get("data", []):
docket = (row.get("attributes") or {}).get("docketId")
if docket:
return docket
return None
def find_documents_in_docket(self, docket_id: str) -> list[dict]:
"""Return the FR documents the API indexes under *docket_id*.
Each document's ``id`` (the API's internal ID, not the FR doc
number) is the ``commentOnId`` you need to list its comments.
"""
out: list[dict] = []
page = 1
while True:
data = self._get(
"/documents",
**{
"filter[docketId]": docket_id,
"page[size]": 250,
"page[number]": page,
},
)
out.extend(data.get("data", []))
meta = data.get("meta", {})
if page >= meta.get("totalPages", 1):
break
page += 1
return out
# ── Comment iteration ──────────────────────────────────────
def iter_comments(self, object_id: str) -> Iterator[Comment]:
"""Yield every comment against a single FR document.
The ``commentOnId`` filter on ``/comments`` takes a document's
**objectId**, not its public ``id`` — the field lives under
``attributes.objectId`` and looks like ``0900006482921ba1``.
Getting this wrong silently returns zero results.
Two pagination quirks we've been burned by:
1. The ``page[number]`` counter caps at 20 (≈5K items at page
size 250). Dockets with more comments need a date-based
cursor via ``filter[lastModifiedDate][ge]``.
2. That filter rejects ISO-8601 timestamps with ``T``/``Z`` —
returns 400. It wants ``YYYY-MM-DD HH:MM:SS`` (space-
separated, no timezone suffix). Normalize before sending.
"""
cursor: str | None = None
page = 1
while True:
params: dict[str, str] = {
"filter[commentOnId]": object_id,
"page[size]": 250,
"page[number]": page,
"sort": "lastModifiedDate,documentId",
}
if cursor:
params["filter[lastModifiedDate][ge]"] = _reg_date(cursor)
try:
data = self._get("/comments", **params)
except httpx.HTTPStatusError as e:
log.warning(
"iter_comments: %s on object %s page %d cursor %r — stopping this object",
e.response.status_code,
object_id,
page,
cursor,
)
break
rows = data.get("data", [])
if not rows:
break
for row in rows:
yield _parse_comment(row)
if len(rows) < 250:
break
meta = data.get("meta", {})
total_pages = meta.get("totalPages") or 1
if page < total_pages:
page += 1
continue
# Ran out of pages — advance the date cursor past the last
# row we've seen and restart pagination from page 1.
last_mod = rows[-1].get("attributes", {}).get("lastModifiedDate")
if not last_mod:
break
new_cursor = _reg_date(last_mod)
if new_cursor == cursor:
break
cursor = new_cursor
page = 1
def get_comment_detail(self, comment_id: str) -> dict:
"""Full comment including attachments relationship."""
return self._get(f"/comments/{comment_id}", **{"include": "attachments"})
# ── Attachments ────────────────────────────────────────────
def attachments_for(self, comment_id: str) -> list[Attachment]:
data = self.get_comment_detail(comment_id)
attachments: list[Attachment] = []
for inc in data.get("included", []) or []:
if inc.get("type") != "attachments":
continue
attrs = inc.get("attributes", {})
for f in attrs.get("fileFormats") or []:
attachments.append(
Attachment(
url=f.get("fileUrl", ""),
filename=_filename_from(f.get("fileUrl", "")),
content_type=f.get("format", ""),
size=f.get("size"),
)
)
return attachments
# Attachment CDN lives on downloads.regulations.gov and rejects the
# API-key header used by the v4 API host. A browser-shaped UA +
# Referer is what it expects; otherwise every GET returns 403 with
# an HTML error page.
_DL_HEADERS = {
"User-Agent": (
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Firefox/122.0"
),
"Accept": "application/pdf,application/octet-stream,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
"Referer": "https://www.regulations.gov/",
}
def download_attachment(
self,
url: str,
dest_dir: Path,
*,
overwrite: bool = False,
) -> Path | None:
dest_dir.mkdir(parents=True, exist_ok=True)
filename = _filename_from(url) or _hash(url) + ".bin"
dest = dest_dir / filename
if dest.is_file() and not overwrite:
return dest
time.sleep(self._sleep)
try:
with self._client.stream(
"GET",
url,
timeout=60,
follow_redirects=True,
headers=self._DL_HEADERS,
) as r:
if r.status_code != 200:
log.warning(
"attachment %s → HTTP %s",
url.rsplit("/", 1)[-1],
r.status_code,
)
return None
with open(dest, "wb") as f:
for chunk in r.iter_bytes(1 << 16):
f.write(chunk)
except httpx.HTTPError as e:
log.warning("attachment %s failed: %s", url, e)
return None
return dest
# ── Bib integration ────────────────────────────────────────────
def backfill_details(
store: Store,
client: Client,
*,
limit: int | None = None,
log_path: Path | None = None,
commit_every: int = 25,
scratch_root: Path = Path(".state/comments"),
) -> dict[str, int]:
"""Enrich every ``source:regulations-gov`` stub with body text,
organization, and any attached files.
Background: ``iter_comments`` pulls from ``/v4/comments`` which only
returns surface metadata — no body, no attachment URLs, no org
name. Full text + attachments live on the per-comment detail
endpoint, one round-trip per item. With 164K comments against
reg.gov's 1000-req/hr cap this runs for days; this helper is
structured so it can be restarted without losing progress.
Resume rule: we treat ``items.abstract`` as the "already enriched"
flag. Any reg-gov item with an empty abstract is retried on next
run. A comment whose detail returns 404 is tagged ``enriched:gone``
so we stop hammering it. Commits land every ``commit_every`` items
so a crash loses at most that many items of work.
"""
con = store._con() # noqa: SLF001
# Resume rule: treat ``enriched:ok`` as the completion marker.
# Using a tag (not just the abstract) lets us handle "see attached"
# comments with near-empty body AND still drives distinct retry
# states (``enriched:gone`` for 404s, untagged for transient fails).
rows = con.execute(
"""
SELECT i.id, i.key, i.url
FROM items i
WHERE i.id IN (
SELECT item_id FROM item_tags
WHERE tag_id IN (SELECT id FROM tags WHERE name='source:regulations-gov')
)
AND i.id NOT IN (
SELECT item_id FROM item_tags
WHERE tag_id IN (SELECT id FROM tags WHERE name IN ('enriched:ok','enriched:gone'))
)
ORDER BY i.id
"""
+ (f" LIMIT {int(limit)}" if limit else "")
).fetchall()
stats = {"enriched": 0, "attached": 0, "gone": 0, "errors": 0, "seen": len(rows)}
def _write_log(msg: str) -> None:
log.info(msg)
print(msg, flush=True)
if log_path:
log_path.parent.mkdir(parents=True, exist_ok=True)
with open(log_path, "a") as f:
f.write(msg + "\n")
_write_log(f"backfill start: {len(rows)} items pending")
for i, row in enumerate(rows, 1):
iid, key, url = row["id"], row["key"], row["url"]
cid = (url or "").rsplit("/", 1)[-1]
if not cid:
stats["errors"] += 1
continue
try:
data = client.get_comment_detail(cid)
except httpx.HTTPStatusError as e:
if e.response.status_code == 404:
try:
store.add_tag(key, "enriched:gone")
stats["gone"] += 1
except Exception as add_err: # noqa: BLE001
log.warning("add_tag enriched:gone failed for %s: %s", key, add_err)
stats["errors"] += 1
else:
log.warning("HTTP %s on %s — skipping", e.response.status_code, cid)
stats["errors"] += 1
continue
except httpx.HTTPError as e:
log.warning("transport error on %s: %s", cid, e)
stats["errors"] += 1
continue
attrs = (data.get("data") or {}).get("attributes") or {}
body = (attrs.get("comment") or "")[:4000]
org = (attrs.get("organization") or "").strip()
if body:
con.execute("UPDATE items SET abstract=? WHERE id=?", (body, iid))
if org:
try:
store.add_tag(key, f"org:{_slug(org)}")
except Exception as e: # noqa: BLE001
log.warning("add_tag org failed for %s: %s", key, e)
# Download any attached files into per-comment dir and register.
docket = _docket_from_comment_id(cid)
dest_dir = scratch_root / docket / cid
for inc in data.get("included") or []:
if inc.get("type") != "attachments":
continue
for f in inc.get("attributes", {}).get("fileFormats") or []:
att_url = f.get("fileUrl") or ""
if not att_url:
continue
path = client.download_attachment(att_url, dest_dir)
if not path:
continue
try:
store.attach_file(key, path, title=path.name)
stats["attached"] += 1
except Exception as e: # noqa: BLE001
# Usually a dup filename; fine to skip.
log.debug("attach_file skipped for %s/%s: %s", key, path.name, e)
try:
store.add_tag(key, "enriched:ok")
except Exception as e: # noqa: BLE001
log.warning("add_tag enriched:ok failed for %s: %s", key, e)
stats["enriched"] += 1
if i % commit_every == 0:
con.commit()
_write_log(
f" {i}/{len(rows)} "
f"enriched={stats['enriched']} "
f"attached={stats['attached']} "
f"gone={stats['gone']} errors={stats['errors']}"
)
con.commit()
_write_log(f"backfill done: {stats}")
return stats
def _docket_from_comment_id(cid: str) -> str:
"""``CMS-2025-0304-14107`` → ``CMS-2025-0304``; safe fallback to cid."""
parts = cid.split("-")
return "-".join(parts[:3]) if len(parts) >= 3 else cid
def upsert_comment(
store: Store,
comment: Comment,
*,
cms_id: str = "",
extra_tags: list[str] | None = None,
) -> str:
"""Upsert the comment as a Source item, return bib key."""
title = comment.title or f"Comment on {comment.comment_on_id}"
byline = _byline(comment)
if byline:
title = f"{byline}: {title[:120]}"
url = f"https://www.regulations.gov/comment/{comment.id}"
item = Source(title=title, url=url)
item.doc_type = "Public Comment"
item.institution = "U.S. Government — regulations.gov"
item.date_published = comment.posted_date or comment.received_date
item.abstract = (comment.comment_text or "")[:4000]
tags = [
Tag.source("regulations-gov").label,
"doctype:comment",
f"docket:{comment.docket_id}",
]
if cms_id:
tags.append(f"rule:{cms_id}")
if comment.posted_date:
tags.append(f"year:{comment.posted_date[:4]}")
if comment.organization:
tags.append(f"org:{_slug(comment.organization)}")
for t in extra_tags or []:
tags.append(t)
for t in tags:
item.add_tag(t)
return store.upsert(item)
# ── Internals ──────────────────────────────────────────────────
def _parse_comment(row: dict) -> Comment:
attrs = row.get("attributes", {})
return Comment(
id=row.get("id", ""),
title=attrs.get("title", ""),
posted_date=(attrs.get("postedDate") or "")[:10],
received_date=(attrs.get("receivedDate") or "")[:10],
docket_id=attrs.get("docketId", ""),
comment_on_id=attrs.get("commentOnId", ""),
first_name=attrs.get("firstName") or "",
last_name=attrs.get("lastName") or "",
organization=attrs.get("organization") or "",
comment_text=attrs.get("comment") or "",
attachment_count=(attrs.get("attachmentCount") or 0),
raw=row,
)
def _byline(c: Comment) -> str:
if c.organization:
return c.organization
name = " ".join(p for p in (c.first_name, c.last_name) if p).strip()
return name
def _slug(s: str) -> str:
import re
s = (s or "").lower()
s = re.sub(r"[^a-z0-9]+", "-", s)
return s.strip("-")[:40]
def _filename_from(url: str) -> str:
return url.rsplit("/", 1)[-1].split("?")[0] if url else ""
def _hash(s: str) -> str:
return hashlib.sha1(s.encode()).hexdigest()[:10] # noqa: S324
def _reg_date(iso_ts: str) -> str:
"""Convert an ISO-8601 timestamp (what the API returns in payloads)
into the ``YYYY-MM-DD HH:MM:SS`` form the filter operators require.
``2017-08-30T18:35:48Z`` → ``2017-08-30 18:35:48``. Idempotent: if
the input is already space-separated we leave it alone.
"""
if not iso_ts:
return iso_ts
return iso_ts.replace("T", " ").rstrip("Z").strip()

207
src/bib/sync.py
View File

@@ -11,26 +11,37 @@ Usage::
s = Store()
items = s.list_items(tag="source:spider")
stats = push_to_zotero(items)
stats = push_to_zotero(items, store=s)
print(stats)
# {'created': 5908, 'skipped': 12, 'tags': 23456, 'collections': 2}
# {'created': 5908, 'skipped': 12, 'tags': 23456, 'attachments': 240}
"""
from __future__ import annotations
import json
import shutil
from pathlib import Path
from typing import TYPE_CHECKING
from bib.item import Item
from zot.db import Db, is_valid_key, now_iso
from zot.db import TYPE_MAP as ZOT_TYPE_MAP
from zot.db import Db, generate_key, is_valid_key, now_iso
# bib item_type → Zotero itemTypeID
if TYPE_CHECKING:
from bib.store import Store
# bib item_type → Zotero itemType *name*. Numeric IDs come from
# zot.db.TYPE_MAP, which is the single source of truth.
_TYPE_NAMES: dict[str, str] = {
"rule": "statute",
"regulation": "statute",
"manual": "report",
"download": "webpage",
"source": "document",
"journal-article": "journalArticle",
}
_TYPE_MAP: dict[str, int] = {
"rule": 20, # statute
"regulation": 20, # statute
"manual": 15, # report
"download": 13, # webpage
"source": 34, # document
"journal-article": 4, # journalArticle
bib_type: ZOT_TYPE_MAP[zot_name] for bib_type, zot_name in _TYPE_NAMES.items()
}
@@ -207,7 +218,9 @@ def _item_to_zotero_fields(item: Item) -> dict[str, str]:
def push_to_zotero(
items: list[Item],
*,
store: Store | None = None,
zotero_db: str | None = None,
zotero_storage: str | None = None,
collection_key: str = "",
) -> dict[str, int]:
"""Push bib items into Zotero's SQLite database.
@@ -216,20 +229,30 @@ def push_to_zotero(
----------
items : list[Item]
Items to sync.
store : bib.Store, optional
Source bib store. When provided, each item's attached PDFs are
copied into Zotero's storage dir and registered as child
``attachment`` items. Without it, attachments are skipped.
zotero_db : str
Path to Zotero's SQLite database.
zotero_storage : str, optional
Zotero storage directory (``data/zotero/data/storage/``).
Defaults to ``<zotero_db>/../storage``.
collection_key : str
Optional Zotero collection key to add items to.
Returns
-------
dict
Counts: created, skipped, tags, collections, creators.
Counts: created, skipped, tags, collections, creators, attachments.
"""
if zotero_db is None:
from conf import path
zotero_db = str(path("db.zotero"))
storage_dir = (
Path(zotero_storage) if zotero_storage else Path(zotero_db).parent / "storage"
)
with Db(zotero_db) as db:
stats: dict[str, int] = {
@@ -237,8 +260,23 @@ def push_to_zotero(
"skipped": 0,
"tags": 0,
"collections": 0,
"attachments": 0,
}
# Resolve a tuple-path to a Zotero collection key, ensuring each
# level exists. Cached across items so we only touch the DB once
# per unique path per sync run.
path_cache: dict[tuple[str, ...], str] = {}
def _resolve_path(path: tuple[str, ...]) -> str:
if path in path_cache:
return path_cache[path]
parent = ""
for name in path:
parent = db.ensure_collection(name, parent_key=parent)
path_cache[path] = parent
return parent
ts = now_iso()
# Resolve target collection
@@ -247,11 +285,33 @@ def push_to_zotero(
collection_id = db.find_collection(collection_key)
for item in items:
# Skip if URL already exists in Zotero
# Skip (update-in-place) if URL already exists in Zotero
if item.url:
existing_id = db.find_item_by_url(item.url)
if existing_id is not None:
# Refresh fields from scratch — earlier syncs used
# a stale FIELD_MAP that landed values in wrongly-
# named columns (title→sessionTitle, seriesTitle→
# ISBN, etc.). replace_fields clears the item's
# entire itemData row set before rewriting with the
# corrected map.
db.replace_fields(existing_id, _item_to_zotero_fields(item))
db.sync_tags(existing_id, item.tags)
if store is not None:
stats["attachments"] += _sync_attachments(
db,
store,
item,
existing_id,
storage_dir,
)
# Backfill collection membership for items that were
# created before sync learned about the hierarchy.
path = _zotero_collection_path(item)
if path:
key = _resolve_path(tuple(path))
if db.add_to_collection(existing_id, collection_key=key):
stats["collections"] += 1
stats["skipped"] += 1
continue
@@ -287,13 +347,134 @@ def push_to_zotero(
db.add_to_collection(item_id, collection_key=collection_key)
stats["collections"] += 1
# Item's own collections
# Item's own collections (legacy: bib keys; usually no hits
# since bib/Zotero live in separate key spaces, but safe).
for col_key in item.collections:
if col_key != collection_key:
db.add_to_collection(item_id, collection_key=col_key)
# Derived collection path (Manual → Healthcare Data Platform
# / Manuals / <name>; OIG → …/OIG Guidance/<type>, etc.).
path = _zotero_collection_path(item)
if path:
key = _resolve_path(tuple(path))
if db.add_to_collection(item_id, collection_key=key):
stats["collections"] += 1
# Attachments (PDFs etc.) — copy from bib storage into Zotero
# storage and register as child items.
if store is not None:
stats["attachments"] += _sync_attachments(
db,
store,
item,
item_id,
storage_dir,
)
stats["created"] += 1
db.commit()
return stats
def _zotero_collection_path(item: Item) -> list[str]:
"""Derive the Zotero collection hierarchy an item should live under.
Returns a path of human-readable names (``ensure_collection`` will
create missing levels on the fly). An empty list means no routing —
the item lands at the top of "My Library" only.
"""
ej = json.loads(item.to_row().get("extra_json", "{}"))
if item.item_type == "manual":
manual = (ej.get("manual_name") or "").strip() or "General"
return ["Healthcare Data Platform", "Manuals", manual]
if "source:regulations-gov" in item.tags:
# Every rulemaking comment lands flat in Rules/Comments. The
# rule:<id> and year:<Y> tags on each item carry the
# disambiguating citation, so a single bucket is readable.
return ["Rules", "Comments"]
if "source:email" in item.tags:
# IMAP-ingested email lands under Inbox/<Mailbox-Title>.
# Each mailbox tag (mailbox:cmsupdates) becomes its own
# subfolder so multi-mailbox setups don't intermingle.
for t in item.tags:
if t.startswith("mailbox:"):
label = t.split(":", 1)[1].replace("-", " ").title()
return ["Inbox", label]
return ["Inbox"]
if "agency:oig" in item.tags:
sector = ""
guidance = ""
for t in item.tags:
if t.startswith("sector:"):
sector = t.split(":", 1)[1].replace("_", " ").title()
elif t.startswith("guidance:"):
guidance = t.split(":", 1)[1].upper()
base = ["Healthcare Data Platform", "OIG Guidance"]
# Prefer sector when known; fall back to guidance type.
if sector:
return [*base, sector]
if guidance:
return [*base, guidance]
return base
return []
def _sync_attachments(
db: Db,
store: Store,
bib_item: Item,
zot_parent_id: int,
zot_storage: Path,
) -> int:
"""Copy every bib attachment for ``bib_item`` into Zotero's storage
and create the matching ``itemAttachments`` row. Idempotent: an
existing same-filename child attachment is left alone."""
con = store._con() # noqa: SLF001
rows = con.execute(
"""SELECT a.filename, a.content_type, a.storage_path
FROM attachments a
JOIN items i ON a.item_id = i.id
WHERE i.key = ?""",
(bib_item.key,),
).fetchall()
if not rows:
return 0
# Existing child attachments keyed by path so we don't duplicate.
existing = {
r[0]
for r in db.con.execute(
"SELECT ia.path FROM itemAttachments ia WHERE ia.parentItemID = ?",
(zot_parent_id,),
).fetchall()
if r[0]
}
count = 0
for row in rows:
src = Path(row["storage_path"])
if not src.is_file():
continue
# Bib's `filename` column is a display label, not a filesystem
# name; the actual PDF filename is the basename of storage_path.
disk_name = src.name
zot_path = f"storage:{disk_name}"
if zot_path in existing:
continue
att_key = generate_key()
dest_dir = zot_storage / att_key
dest_dir.mkdir(parents=True, exist_ok=True)
shutil.copy2(src, dest_dir / disk_name)
db.add_attachment(
zot_parent_id,
link_mode=1, # Zotero.Attachments.LINK_MODE_IMPORTED_FILE
content_type=row["content_type"] or "application/pdf",
path=zot_path,
key=att_key,
)
count += 1
return count

2
src/bib/translate.py
View File

@@ -146,7 +146,7 @@ def federal_register(url: str) -> Rule:
cms_id=cms_id,
rule_type=rule_type,
date_published=data.get("publication_date", ""),
effective_date=data.get("effective_on", ""),
effective_date=data.get("effective_on") or "",
url=data.get("html_url", url),
abstract=data.get("abstract", ""),
)

6
src/cli/__init__.py
View File

@@ -20,10 +20,13 @@ from cli.generate import app as generate_app
from cli.health import health
from cli.lake import app as lake_app
from cli.load import app as load_app
from cli.mail import app as mail_app
from cli.perf import app as perf_app
from cli.prisma import app as prisma_app
from cli.rec import app as rec_app
from cli.run import run
from cli.validate import validate
from cli.zot import app as zot_app
app = typer.Typer(
name="stack",
@@ -47,6 +50,9 @@ app.add_typer(
name="rec",
help="Reconcile calculated payments against CMS ground-truth files.",
)
app.add_typer(zot_app, name="zot", help="Zotero database maintenance.")
app.add_typer(prisma_app, name="prisma", help="LLM-driven PRISMA screening workflow.")
app.add_typer(mail_app, name="mail", help="Maddy mail server (DO + Resend smarthost).")
def main() -> None:

558
src/cli/bib.py
View File

@@ -2,6 +2,8 @@
from __future__ import annotations
from pathlib import Path
import typer
app = typer.Typer(no_args_is_help=True)
@@ -71,3 +73,559 @@ def query(
if len(items) > 20:
typer.echo(f" ... and {len(items) - 20} more")
typer.echo(f"{len(items)} items found.")
_UA = "fhirworx-bib/0.1"
def _http_client():
import httpx
return httpx.Client(headers={"User-Agent": _UA})
@app.command(name="discover-pfs-rules")
def discover_pfs_rules(
since: str = typer.Option("2017-01-01", "--since"),
until: str = typer.Option("", "--until"),
dry_run: bool = typer.Option(False, "--dry-run"),
) -> None:
"""Walk the Federal Register API for every PFS rule since a date.
Upserts each as a bib ``Rule`` record tagged ``module:pfs``,
``source:federal-register``, ``rule:<cms-id>``. Cheap — the FR API
has no auth and no rate charge we can trigger at this scale
(~20 rules since 2017).
"""
from bib import connect
from bib.federalregister import pfs_rules
from bib.tag import Tag
from bib.translate import federal_register
store = connect() if not dry_run else None
for doc in pfs_rules(since=since, until=until or None):
typer.echo(
f" {doc.publication_date} {doc.type:14s} "
f"{doc.document_number:14s} {','.join(doc.dockets) or '(no docket)'}"
)
if dry_run or store is None:
continue
try:
rule = federal_register(
doc.html_url
or f"https://www.federalregister.gov/documents/{doc.document_number}"
)
except Exception as e: # noqa: BLE001
typer.echo(f" skipped: {e.__class__.__name__}: {e}")
continue
rule.add_tag(Tag.source("federal-register").label)
rule.add_tag("module:pfs")
for d in doc.dockets:
rule.add_tag(f"docket:{d}")
store.upsert(rule)
@app.command(name="fetch-docket-comments")
def fetch_docket_comments(
docket: str = typer.Argument(..., help="e.g. CMS-1676-P"),
cms_id: str = typer.Option(
"", "--cms-id", help="Associate comments with a specific CMS rule tag."
),
limit: int = typer.Option(
0, "--limit", "-n", help="Stop after N comments. 0 = all."
),
attachments: bool = typer.Option(
False,
"--attachments",
help="Also download each comment's PDF/DOCX attachments "
"(expensive — skim first without).",
),
sleep: float = typer.Option(
1.3, "--sleep", help="Seconds between API calls (rate budget)."
),
) -> None:
"""Walk every comment on one docket; upsert as Source items.
Policy: we iterate every FR document indexed under the docket and
pull their comments. For one PFS rule that usually means one parent
document with a few thousand to tens of thousands of comments.
"""
from pathlib import Path
from bib import connect
from bib.regulations_gov import Client, upsert_comment
store = connect()
scratch = Path(f".state/comments/{docket}")
with Client(sleep=sleep) as api:
docs = api.find_documents_in_docket(docket)
typer.echo(f" docket {docket}: {len(docs)} FR documents indexed")
total = 0
for fr_doc in docs:
attrs = fr_doc.get("attributes") or {}
fr_id = fr_doc["id"]
object_id = attrs.get("objectId")
if not object_id or not attrs.get("commentEndDate"):
continue
typer.echo(f" ↓ comments on {fr_id} (objectId={object_id})")
for c in api.iter_comments(object_id):
if limit and total >= limit:
return
key = upsert_comment(store, c, cms_id=cms_id)
if attachments and c.attachment_count:
for att in api.attachments_for(c.id):
path = api.download_attachment(att.url, scratch / c.id)
if path:
store.attach_file(key, path, title=att.filename)
total += 1
if total % 50 == 0:
typer.echo(f" processed {total} comments")
store._con().commit() # noqa: SLF001
typer.echo(f" total: {total} comments upserted")
@app.command(name="fetch-pfs-comments")
def fetch_pfs_comments(
since: str = typer.Option("2017-01-01", "--since"),
until: str = typer.Option("", "--until"),
attachments: bool = typer.Option(False, "--attachments"),
per_docket_limit: int = typer.Option(
0,
"--per-docket-limit",
help="Cap comments fetched per docket. 0 = unlimited.",
),
sleep: float = typer.Option(1.3, "--sleep"),
) -> None:
"""One-shot: discover every PFS rule since *since* and pull every
comment on each of their dockets.
Heavy run. A single PFS proposed rule can hold 5K–25K comments;
times ~20 rules and with attachments this runs for many hours
under the reg.gov rate budget. Use ``--per-docket-limit`` to
smoke-test first.
"""
from pathlib import Path
from bib import connect
from bib.federalregister import pfs_rules, split_docket_ids
from bib.regulations_gov import Client, upsert_comment
from bib.tag import Tag
from bib.translate import federal_register
store = connect()
rules = pfs_rules(since=since, until=until or None)
typer.echo(f"==> {len(rules)} PFS rules since {since}")
# Only proposed rules have public comment periods — skip finals +
# corrections to keep the work focused.
proposed = [r for r in rules if r.type == "Proposed Rule"]
typer.echo(f" {len(proposed)} proposed rules (the ones with comments)")
with Client(sleep=sleep) as api:
for doc in proposed:
cms_ids = split_docket_ids(doc.dockets)
try:
rule = federal_register(
doc.html_url
or f"https://www.federalregister.gov/documents/{doc.document_number}"
)
except Exception as e: # noqa: BLE001
typer.echo(f" skipped rule meta: {e}")
continue
rule.add_tag(Tag.source("federal-register").label)
rule.add_tag("module:pfs")
for cid in cms_ids:
rule.add_tag(f"cms-rule:{cid}")
store.upsert(rule)
# Resolve each CMS-XXXX-P to its reg.gov docket id.
for cms_id in cms_ids:
reg_docket = api.resolve_docket(cms_id)
if not reg_docket:
typer.echo(f" skip {cms_id}: no reg.gov docket found")
continue
typer.echo(f" {cms_id} → {reg_docket} ({doc.publication_date})")
rule.add_tag(f"reg-docket:{reg_docket}")
store.upsert(rule)
fr_docs = api.find_documents_in_docket(reg_docket)
per_docket_count = 0
scratch = Path(f".state/comments/{reg_docket}")
for fr_doc in fr_docs:
attrs = fr_doc.get("attributes") or {}
object_id = attrs.get("objectId")
# Skip docs with no objectId or no real comment
# window — final rules and internal display versions
# don't carry meaningful comment traffic.
if not object_id:
continue
if not attrs.get("commentEndDate"):
continue
for c in api.iter_comments(object_id):
if per_docket_limit and per_docket_count >= per_docket_limit:
break
key = upsert_comment(
store,
c,
cms_id=cms_id,
extra_tags=[f"reg-docket:{reg_docket}"],
)
if attachments and c.attachment_count:
for att in api.attachments_for(c.id):
path = api.download_attachment(
att.url,
scratch / c.id,
)
if path:
store.attach_file(key, path, title=att.filename)
per_docket_count += 1
if per_docket_count % 50 == 0:
typer.echo(f" {per_docket_count} comments")
store._con().commit() # noqa: SLF001
if per_docket_limit and per_docket_count >= per_docket_limit:
break
store._con().commit() # noqa: SLF001
typer.echo(f" docket total: {per_docket_count}")
@app.command(name="ingest-mail")
def ingest_mail(
user: str = typer.Option(
"cmsupdates@mail.fhirworx.io",
"--user",
help="Mailbox address to poll (must have a password in .state/mail/credentials.json).",
),
host: str = typer.Option("mail.fhirworx.io", "--host"),
port: int = typer.Option(993, "--port"),
folder: str = typer.Option("INBOX", "--folder"),
limit: int = typer.Option(0, "--limit", "-n"),
) -> None:
"""Pull UNSEEN mail from a mailbox via IMAPS, upsert as bib Sources.
Idempotent: marks each ingested message ``\\Seen`` on the server,
so re-runs only fetch new mail. Attachments ride along into bib
storage. After this, ``stack bib sync-zotero`` routes everything
tagged ``source:email`` into the appropriate Zotero collection.
"""
import json
from bib import connect
from bib.email_ingest import Mailbox, ingest
creds_path = Path(".state/mail/credentials.json")
if not creds_path.exists():
raise typer.BadParameter(
"no .state/mail/credentials.json — run `stack mail provision` first."
)
creds = json.loads(creds_path.read_text())
pw = creds.get(user) or creds.get(user.split("@", 1)[0])
if not pw:
raise typer.BadParameter(
f"no cached password for {user}.\nRun: stack mail rotate-creds {user}"
)
store = connect()
stats = ingest(
store,
Mailbox(host=host, port=port, username=user, password=pw, folder=folder),
limit=limit or None,
)
for k, v in stats.items():
typer.echo(f" {k}: {v}")
@app.command(name="backfill-comments")
def backfill_comments(
limit: int = typer.Option(
0,
"--limit",
"-n",
help="Cap items processed this run. 0 = no cap (multi-day crawl).",
),
sleep: float = typer.Option(
3.7,
"--sleep",
help="Seconds between API calls. 3.7s ≈ 970/hr — just under the "
"1000/hr reg.gov cap.",
),
log_path: Path = typer.Option(
Path("/tmp/bib-backfill-comments.log"),
"--log",
help="Append-only progress log (survives restarts).",
),
) -> None:
"""Enrich every reg-gov comment stub with body + attachments.
Walks items tagged ``source:regulations-gov`` whose ``abstract`` is
still empty and hits ``/v4/comments/{id}?include=attachments`` for
each one. Resumable: stop any time, start again, it picks up where
it left off by skipping already-enriched items. At 1000/hr against
164K stubs this is a ~7-day crawl; longer if attachments are big.
"""
from bib import connect
from bib.regulations_gov import Client, backfill_details
store = connect()
with Client(sleep=sleep) as api:
stats = backfill_details(
store,
api,
limit=limit or None,
log_path=log_path,
)
for k, v in stats.items():
typer.echo(f" {k}: {v}")
@app.command(name="ingest-iom")
def ingest_iom(
pubs: list[str] = typer.Option(
None,
"--pub",
"-p",
help="Limit to specific pub numbers (e.g. 100-02). Repeat for multiple.",
),
) -> None:
"""Crawl the CMS IOM index and upsert every chapter as a Manual.
Idempotent: re-running only adds missing chapters (dedup is by URL
in the store). Without --pub, ingests every manual the CMS index
advertises.
"""
from bib import connect
from bib.iom import ingest_all
store = connect()
with _http_client() as client:
summary = ingest_all(store, client, pubs=pubs)
for pub, n in sorted(summary.items()):
typer.echo(f" {pub}: {n} chapters")
typer.echo(
f"Total: {sum(summary.values())} chapters across {len(summary)} manuals."
)
@app.command(name="attach-iom")
def attach_iom(
pubs: list[str] = typer.Option(
None,
"--pub",
"-p",
help="Limit to specific pub numbers. Repeat for multiple.",
),
force: bool = typer.Option(
False,
"--force",
help="Re-download even if local SHA-256 matches.",
),
) -> None:
"""Download the PDF for every Manual and attach it to the record.
Idempotent by SHA-256 — only downloads files that have changed on
CMS since the last run. Safe to schedule on a timer.
"""
from bib import connect
from bib.iom import download_attachments
store = connect()
with _http_client() as client:
results = download_attachments(store, client, pubs=pubs, force=force)
for pub, n in sorted(results.items()):
typer.echo(f" {pub}: {n} new/updated")
typer.echo(f"Total: {sum(results.values())} attachments updated.")
@app.command(name="watch-iom")
def watch_iom() -> None:
"""Poll the CMS futurepdf.pdf schedule; attach if changed.
CMS publishes upcoming IOM transmittals at
``/manuals/downloads/futurepdf.pdf``. When its checksum changes,
chapters in the main manuals are probably about to update — run
``ingest-iom`` + ``attach-iom`` afterward to pick them up.
"""
from bib import connect
from bib.iom import check_future_updates
store = connect()
with _http_client() as client:
changed, digest = check_future_updates(store, client)
status = "CHANGED" if changed else "unchanged"
typer.echo(f"futurepdf.pdf: {status} (sha256={digest[:12]})")
@app.command(name="sync-zotero")
def sync_zotero(
tag: str = typer.Option("", "--tag", help="Only sync items matching a tag."),
item_type: str = typer.Option("", "--type", help="Only sync one item_type."),
hold_zotero: bool = typer.Option(
True,
"--hold-zotero/--no-hold",
help="Stop the 'zotero' container during sync so Zotero releases "
"its exclusive SQLite lock; restart afterwards.",
),
) -> None:
"""Push bib items (and their attachments) to the Zotero database.
Zotero holds an exclusive write lock on its SQLite file while
running. By default we stop the 'zotero' compose service for the
duration of the sync and start it again afterward.
"""
import subprocess
from bib import connect
from bib.sync import push_to_zotero
store = connect()
items = store.list_items(tag=tag, item_type=item_type)
def _docker(*args: str) -> None:
subprocess.run(
["docker", *args],
capture_output=True,
check=False,
timeout=30,
)
try:
if hold_zotero:
_docker("stop", "zotero")
stats = push_to_zotero(items, store=store)
finally:
if hold_zotero:
_docker("start", "zotero")
for k, v in stats.items():
typer.echo(f" {k}: {v}")
@app.command(name="ingest-oig")
def ingest_oig(
kinds: list[str] = typer.Option(
None,
"--kind",
"-k",
help="Restrict to a subset: 'cpg' and/or 'alerts'. Repeat for multiple.",
),
) -> None:
"""Crawl OIG Compliance Program Guidance + fraud alerts indexes."""
from bib import connect
from bib.oig import ingest_all
store = connect()
with _http_client() as client:
summary = ingest_all(store, client, kinds=kinds)
for k, n in summary.items():
typer.echo(f" {k}: {n} docs")
@app.command(name="attach-oig")
def attach_oig(
force: bool = typer.Option(False, "--force", help="Re-download even if unchanged."),
) -> None:
"""Download and attach the PDF/HTML for every OIG guidance record."""
from bib import connect
from bib.oig import download_attachments
store = connect()
with _http_client() as client:
n = download_attachments(store, client, force=force)
typer.echo(f" {n} attachments updated")
@app.command(name="refresh-oig")
def refresh_oig() -> None:
"""One-shot: ingest-oig → attach-oig → sync-zotero (OIG tag)."""
import subprocess
from bib import connect
from bib.oig import download_attachments, ingest_all
from bib.sync import push_to_zotero
store = connect()
with _http_client() as client:
typer.echo("==> ingest-oig")
summary = ingest_all(store, client)
typer.echo(f" {summary['cpg']} CPGs, {summary['alerts']} alerts")
typer.echo("==> attach-oig")
n = download_attachments(store, client)
typer.echo(f" {n} attachments new/changed")
typer.echo("==> sync-zotero")
items = store.list_items(tag="agency:oig")
try:
subprocess.run(
["docker", "stop", "zotero"], capture_output=True, check=False, timeout=30
)
stats = push_to_zotero(items, store=store)
finally:
subprocess.run(
["docker", "start", "zotero"], capture_output=True, check=False, timeout=30
)
typer.echo(
f" created={stats['created']} skipped={stats['skipped']} "
f"attachments={stats['attachments']}"
)
@app.command(name="refresh-iom")
def refresh_iom(
pubs: list[str] = typer.Option(None, "--pub", "-p"),
) -> None:
"""One-shot: ingest-iom → watch-iom → attach-iom → sync-zotero.
Safe to put behind a cron. Idempotent at every step:
- ingest: upserts by URL, prunes stale records
- watch: re-attaches futurepdf.pdf only when its SHA-256 changes
- attach: skips PDFs whose checksum matches the stored copy
- sync: skips Zotero items whose URL already exists
Zotero is held (stopped) during the sync phase only.
"""
import subprocess
from bib import connect
from bib.iom import check_future_updates, download_attachments, ingest_all
from bib.sync import push_to_zotero
store = connect()
with _http_client() as client:
typer.echo("==> ingest-iom")
ingest_summary = ingest_all(store, client, pubs=pubs)
typer.echo(
f" {sum(ingest_summary.values())} chapters across "
f"{len([p for p, n in ingest_summary.items() if n])} manuals"
)
typer.echo("==> watch-iom")
changed, digest = check_future_updates(store, client)
typer.echo(
f" futurepdf.pdf: {'CHANGED' if changed else 'unchanged'} "
f"(sha256={digest[:12]})"
)
typer.echo("==> attach-iom")
attach_summary = download_attachments(store, client, pubs=pubs)
typer.echo(f" {sum(attach_summary.values())} PDFs downloaded or refreshed")
typer.echo("==> sync-zotero")
items = store.list_items(item_type="manual")
try:
subprocess.run(
["docker", "stop", "zotero"], capture_output=True, check=False, timeout=30
)
stats = push_to_zotero(items, store=store)
finally:
subprocess.run(
["docker", "start", "zotero"], capture_output=True, check=False, timeout=30
)
typer.echo(
f" created={stats['created']} skipped={stats['skipped']} "
f"attachments={stats['attachments']}"
)

123
src/cli/mail.py Normal file
View File

@@ -0,0 +1,123 @@
"""`stack mail` — touchless Maddy mail server lifecycle."""
from __future__ import annotations
import typer
app = typer.Typer(no_args_is_help=True)
@app.command(name="provision")
def provision(
region: str = typer.Option("nyc3", help="DO region slug (one your ISP can reach)."),
) -> None:
"""End-to-end idempotent: up → wait → DNS → DKIM → smarthost → git env."""
from mail.droplet import provision as _provision
_provision(region=region)
@app.command(name="up")
def up(
region: str = typer.Option("nyc3", help="DO region slug."),
) -> None:
"""Adopt-or-create the mail droplet. Idempotent."""
from mail.droplet import up as _up
_up(region=region)
@app.command(name="down")
def down(
yes: bool = typer.Option(False, "--yes", help="Skip confirmation."),
) -> None:
"""Destroy the mail droplet. Idempotent no-op if none exists."""
from mail.droplet import down as _down
if not yes:
typer.confirm("Destroy the mail droplet?", abort=True)
_down(confirm=True)
@app.command(name="status")
def status() -> None:
"""Show droplet state from DO + PTR check."""
from mail.droplet import status as _status
info = _status()
if not info:
typer.echo("no mail droplet exists — `stack mail provision`")
return
for k in ("name", "id", "region", "public_ip", "status", "ptr", "ptr_match"):
typer.echo(f" {k:11s} {info.get(k)}")
@app.command(name="dns")
def dns(
claim_apex: bool = typer.Option(
False,
"--claim-apex",
help="Also write apex MX/SPF/DMARC. Skip when another provider "
"(Proton, Fastmail, Workspace) already owns the apex.",
),
) -> None:
"""Upsert mail DNS records in Cloudflare. Idempotent."""
from mail.droplet import apply_dns
apply_dns(claim_apex=claim_apex)
@app.command(name="dkim-export")
def dkim_export() -> None:
"""Publish Maddy's DKIM public key to Cloudflare. Idempotent."""
from mail.droplet import export_dkim
export_dkim()
@app.command(name="attach-smarthost")
def attach_smarthost() -> None:
"""Register at Resend, push DNS, install smarthost on droplet."""
from mail.droplet import attach_smarthost as _attach
_attach()
@app.command(name="rotate-creds")
def rotate_creds(
user: str = typer.Argument(
..., help="Mailbox to rotate (e.g. git, postmaster, cmsupdates)."
),
) -> None:
"""Mint a new password for *user*@fhirworx.io and push to droplet."""
from mail.droplet import rotate_creds as _rotate
_rotate(user)
@app.command(name="seed-mailboxes")
def seed_mailboxes() -> None:
"""Ensure the standard set of mailboxes exists (postmaster, git, cmsupdates).
Idempotent: only creates missing ones; existing mailboxes keep
their cached passwords. Useful after adding a new entry to
``DEFAULT_MAILBOXES`` in mail/droplet.py.
"""
from mail.droplet import seed_mailboxes as _seed
_seed()
@app.command(name="wire-git")
def wire_git() -> None:
"""Write .state/git/mailer.env from the mail droplet's credentials.
Then run ``docker compose up -d git`` to apply (a plain ``docker
restart`` won't re-read the new env_file directive in compose.yml).
"""
from mail.droplet import write_git_mailer_env
if write_git_mailer_env():
typer.echo(" git mailer env written → run: docker compose up -d git")
else:
typer.echo(" git mailer env unchanged or skipped")

527
src/cli/prisma.py Normal file
View File

@@ -0,0 +1,527 @@
"""stack prisma — LLM-driven PRISMA screening workflow.
Stages:
1 identified (implicit — everything in the project tag)
2 screen title/abstract LLM pass
3 eligibility full-text LLM pass
3+ extract structured data extraction on included items
4 included derived from tag state
Commands are idempotent: tags record the transition a row has already
made, and the screen / eligibility loops skip items already past that
stage.
"""
from __future__ import annotations
import subprocess
from pathlib import Path
import typer
app = typer.Typer(no_args_is_help=True)
_DEFAULT_ZOT = Path("data/zotero/data/zotero.sqlite")
_DEFAULT_STORAGE = Path("data/zotero/data/storage")
def _hold(fn, *, hold: bool):
if not hold:
return fn()
subprocess.run(
["docker", "stop", "zotero"], capture_output=True, check=False, timeout=30
)
try:
return fn()
finally:
subprocess.run(
["docker", "start", "zotero"], capture_output=True, check=False, timeout=30
)
@app.command(name="init")
def init_project(
name: str = typer.Argument(..., help="Project slug, e.g. skin-subs."),
db: Path = typer.Option(_DEFAULT_ZOT, "--db", help="Zotero SQLite path."),
hold: bool = typer.Option(True, "--hold/--no-hold"),
) -> None:
"""Create (or re-load) the project's three Zotero anchor items:
criteria, extraction-template, and the reasons codebook."""
from prisma.project import init as _init
from zot.db import Db
def _go():
with Db(str(db)) as zdb:
project = _init(zdb, name)
typer.echo(f"project: {project.name}")
typer.echo(f" criteria: {len(project.criteria):>6} chars")
typer.echo(
f" extraction_template: {len(project.extraction_template):>6} chars"
)
typer.echo(f" reasons: {len(project.reasons):>6} chars")
_hold(_go, hold=hold)
@app.command(name="export")
def export_item(
zot_id: int = typer.Argument(..., help="Zotero itemID to export."),
fulltext: bool = typer.Option(False, "--fulltext"),
db: Path = typer.Option(_DEFAULT_ZOT, "--db"),
storage: Path = typer.Option(_DEFAULT_STORAGE, "--storage"),
hold: bool = typer.Option(True, "--hold/--no-hold"),
) -> None:
"""Dump one Zotero item as the LLM-ready markdown snapshot."""
from prisma.export import load_item, to_markdown
from zot.db import Db
def _go() -> str:
with Db(str(db)) as zdb:
snap = load_item(zdb, zot_id, storage)
return to_markdown(snap, include_fulltext=fulltext)
typer.echo(_hold(_go, hold=hold))
@app.command(name="ping-llm")
def ping_llm() -> None:
"""Send a cheap ping through the configured provider.
Useful for verifying ``PRISMA_LLM_PROVIDER`` + auth before kicking
off a long screening run.
"""
from prisma.llm import LLMCall, LLMMessage, make_provider
provider = make_provider()
call = LLMCall(
messages=[
LLMMessage(role="user", content="Respond with exactly: pong."),
],
max_tokens=8,
)
result = provider.complete(call)
typer.echo(f"text: {result.text.strip()}")
typer.echo(f"usage: {result.usage}")
# ── Stage loops ─────────────────────────────────────────────────
@app.command(name="screen")
def screen(
name: str = typer.Argument(..., help="Project slug."),
limit: int = typer.Option(
0, "--limit", "-n", help="0 = screen every unscreened item."
),
db: Path = typer.Option(_DEFAULT_ZOT, "--db"),
storage: Path = typer.Option(_DEFAULT_STORAGE, "--storage"),
hold: bool = typer.Option(True, "--hold/--no-hold"),
) -> None:
"""Stage 2 — LLM-driven title/abstract screening pass."""
from prisma import screen as _screen
from prisma.llm import make_provider
from prisma.project import load
from zot.db import Db
def _go():
provider = make_provider()
with Db(str(db)) as zdb:
project = load(zdb, name)
return _screen.run(
zdb,
provider,
project,
storage_dir=storage,
limit=limit or None,
)
stats = _hold(_go, hold=hold)
for k, v in stats.items():
typer.echo(f" {k}: {v}")
@app.command(name="eligible")
def eligible(
name: str = typer.Argument(..., help="Project slug."),
limit: int = typer.Option(0, "--limit", "-n"),
db: Path = typer.Option(_DEFAULT_ZOT, "--db"),
storage: Path = typer.Option(_DEFAULT_STORAGE, "--storage"),
hold: bool = typer.Option(True, "--hold/--no-hold"),
) -> None:
"""Stage 3 — LLM-driven full-text eligibility pass."""
from prisma import eligibility
from prisma.llm import make_provider
from prisma.project import load
from zot.db import Db
def _go():
provider = make_provider()
with Db(str(db)) as zdb:
project = load(zdb, name)
return eligibility.run(
zdb,
provider,
project,
storage_dir=storage,
limit=limit or None,
)
stats = _hold(_go, hold=hold)
for k, v in stats.items():
typer.echo(f" {k}: {v}")
@app.command(name="extract")
def extract(
name: str = typer.Argument(..., help="Project slug."),
limit: int = typer.Option(0, "--limit", "-n"),
db: Path = typer.Option(_DEFAULT_ZOT, "--db"),
storage: Path = typer.Option(_DEFAULT_STORAGE, "--storage"),
hold: bool = typer.Option(True, "--hold/--no-hold"),
) -> None:
"""Stage 3+ — structured data extraction on included studies."""
from prisma import extract as _extract
from prisma.llm import make_provider
from prisma.project import load
from zot.db import Db
def _go():
provider = make_provider()
with Db(str(db)) as zdb:
project = load(zdb, name)
return _extract.run(
zdb,
provider,
project,
storage_dir=storage,
limit=limit or None,
)
stats = _hold(_go, hold=hold)
for k, v in stats.items():
typer.echo(f" {k}: {v}")
@app.command(name="flow")
def flow(
name: str = typer.Argument(..., help="Project slug."),
mermaid: bool = typer.Option(True, "--mermaid/--text"),
db: Path = typer.Option(_DEFAULT_ZOT, "--db"),
hold: bool = typer.Option(True, "--hold/--no-hold"),
) -> None:
"""Emit the PRISMA 2020 flow diagram from current tag counts."""
from prisma import flow as _flow
from zot.db import Db
def _go() -> str:
with Db(str(db)) as zdb:
counts = _flow.count(zdb, name)
return (
_flow.mermaid(counts, project=name)
if mermaid
else _flow.text_summary(counts)
)
typer.echo(_hold(_go, hold=hold))
@app.command(name="fetch")
def fetch(
name: str = typer.Argument(..., help="Project slug."),
limit: int = typer.Option(0, "--limit", "-n"),
db: Path = typer.Option(_DEFAULT_ZOT, "--db"),
storage: Path = typer.Option(_DEFAULT_STORAGE, "--storage"),
scratch: Path = typer.Option(Path(".state/prisma-fetch"), "--scratch"),
proxy: str = typer.Option(
"", "--proxy", help="Override PRISMA_FETCH_PROXY (else env)."
),
use_vpn: bool = typer.Option(
True,
"--vpn/--no-vpn",
help="Auto-open an SSH tunnel to the provisioned droplet when one "
"exists. --no-vpn skips the tunnel and uses only the "
"unproxied sources.",
),
verify: bool = typer.Option(
True,
"--verify/--no-verify",
help="Curl through the proxy once before fetching and confirm "
"egress is non-US.",
),
hold: bool = typer.Option(True, "--hold/--no-hold"),
) -> None:
"""Fetch PDFs for every non-excluded item lacking an attachment.
Source cascade: Unpaywall → PMC → fallback (via the VPN droplet).
Items excluded at stage 2 are untouched — the queue filter only
passes `screen:include` and `screen:uncertain` through.
Tunnel lifecycle is automatic: if a droplet is tracked in
``.state/prisma-vpn/`` this command opens an SSH port-forward for
the duration of the fetch and closes it on exit. No sudo required.
"""
import contextlib
import os
from prisma import fetch as _fetch
from prisma import vpn as _vpn
from zot.db import Db
email = os.environ.get("UNPAYWALL_EMAIL", "dev@fhirworx.io")
explicit_proxy = proxy or os.environ.get("PRISMA_FETCH_PROXY") or None
droplet_up = _vpn.status().get("status") == "up"
auto_tunnel = use_vpn and droplet_up and not explicit_proxy
@contextlib.contextmanager
def _proxy_cm():
if explicit_proxy:
yield explicit_proxy
elif auto_tunnel:
with _vpn.active() as url:
if verify:
info = _vpn.verify_egress(url)
typer.echo(
f" egress: {info.get('ip', '?')} ({info.get('country', '?')})"
)
yield url
else:
if not droplet_up:
typer.echo(
" note: no VPN droplet tracked — fallback tier disabled. "
"Run `stack prisma vpn up` to provision one."
)
yield None
def _go():
with _proxy_cm() as fetch_proxy, Db(str(db)) as zdb:
return _fetch.run(
zdb,
project=name,
storage_dir=storage,
scratch_dir=scratch,
email=email,
fetch_proxy=fetch_proxy,
limit=limit or None,
)
stats = _hold(_go, hold=hold)
for k, v in stats.items():
typer.echo(f" {k}: {v}")
# ── VPN droplet lifecycle ───────────────────────────────────────
vpn_app = typer.Typer(no_args_is_help=True)
app.add_typer(
vpn_app,
name="vpn",
help="DigitalOcean SOCKS5 exit for fallback fetch (SSH tunnel, no sudo needed).",
)
@vpn_app.command(name="up")
def vpn_up(
region: str = typer.Option(
"", "--region", help="DO region slug. Defaults to PRISMA_VPN_REGION."
),
attach_zotero: bool = typer.Option(
True,
"--attach-zotero/--no-attach-zotero",
help="Also start a sidecar SSH tunnel on the stack gateway network "
"and rewrite Zotero's prefs.js to proxy through it, so the "
"fallback plugin downloads via the droplet.",
),
) -> None:
"""Provision the droplet, wait for dante, wire Zotero up.
End state: Zotero's fallback-PDF plugin downloads through
``socks5://fetch-proxy:1080``; stack's own `fetch` command pulls
through the same route via ``vpn.active()``. Zero client-side sudo.
"""
from prisma import vpn as _vpn
info = _vpn.up(region=region or None, attach_zotero=attach_zotero)
typer.echo(f" droplet: {info['name']} ({info['droplet_id']})")
typer.echo(f" region: {info['region']}")
typer.echo(f" public IP: {info['public_ip']}")
typer.echo(f" ssh key: {info['ssh_key']}")
typer.echo(f" proxy URL: {info['proxy_url']}")
if info.get("sidecar"):
typer.echo(
f" zotero proxy: {info['zotero_proxy']} (sidecar: {info['sidecar']})"
)
typer.echo("")
typer.echo("Next: `stack prisma fetch <project>` — tunnel opens automatically.")
@vpn_app.command(name="down")
def vpn_down() -> None:
"""Destroy the droplet and wipe local state."""
from prisma import vpn as _vpn
info = _vpn.down()
for k, v in info.items():
typer.echo(f" {k}: {v}")
@vpn_app.command(name="status")
def vpn_status() -> None:
"""Report whether a PRISMA VPN droplet is currently tracked."""
from prisma import vpn as _vpn
for k, v in _vpn.status().items():
typer.echo(f" {k}: {v}")
@vpn_app.command(name="attach-zotero")
def vpn_attach_zotero() -> None:
"""Start the sidecar tunnel + rewire Zotero prefs.
Use when the droplet is already up but Zotero was never attached,
or to re-attach after a zotero container rebuild wiped prefs.
"""
import json
from prisma import vpn as _vpn
if not _vpn._DROPLET_JSON.is_file(): # noqa: SLF001
raise typer.BadParameter(
"no droplet tracked — run `stack prisma vpn up` first."
)
meta = json.loads(_vpn._DROPLET_JSON.read_text()) # noqa: SLF001
_vpn.attach_zotero_proxy(meta["public_ip"])
typer.echo(" sidecar started on gateway network as `fetch-proxy`")
typer.echo(" zotero prefs patched → socks5://fetch-proxy:1080")
typer.echo(" zotero restarted")
@vpn_app.command(name="detach-zotero")
def vpn_detach_zotero() -> None:
"""Stop the sidecar and restore Zotero's pre-proxy prefs."""
from prisma import vpn as _vpn
_vpn.detach_zotero_proxy()
typer.echo(" sidecar removed; prefs restored; zotero restarted.")
@vpn_app.command(name="verify")
def vpn_verify() -> None:
"""Open the tunnel, curl ifconfig.co, print egress, close.
Confirms the droplet is routable and reports its public IP /
country — the touchless way to sanity-check before a long fetch
run.
"""
from prisma import vpn as _vpn
with _vpn.active() as url:
info = _vpn.verify_egress(url)
typer.echo(f" proxy: {url}")
typer.echo(f" egress: {info.get('ip', '?')}")
typer.echo(
f" country: {info.get('country', '?')} ({info.get('country_iso', '?')})"
)
typer.echo(f" asn: {info.get('asn_org', '?')}")
# ── Orchestrated run ────────────────────────────────────────────
@app.command(name="run")
def run_all(
name: str = typer.Argument(..., help="Project slug."),
limit: int = typer.Option(
0, "--limit", "-n", help="Same limit applied to every stage."
),
skip_fetch: bool = typer.Option(False, "--skip-fetch"),
skip_extract: bool = typer.Option(False, "--skip-extract"),
db: Path = typer.Option(_DEFAULT_ZOT, "--db"),
storage: Path = typer.Option(_DEFAULT_STORAGE, "--storage"),
hold: bool = typer.Option(True, "--hold/--no-hold"),
) -> None:
"""Orchestrate all PRISMA stages in order.
screen → fetch → eligible → extract → flow, all inside one zotero
hold. Fetch only pulls PDFs for non-excluded items; stage 2
exclusions are terminal by design.
"""
import contextlib
import os
from prisma import (
eligibility,
)
from prisma import (
extract as _extract,
)
from prisma import (
fetch as _fetch,
)
from prisma import (
flow as _flow,
)
from prisma import (
screen as _screen,
)
from prisma import (
vpn as _vpn,
)
from prisma.llm import make_provider
from prisma.project import load
from zot.db import Db
def _proxy_cm():
explicit = os.environ.get("PRISMA_FETCH_PROXY")
if explicit:
return contextlib.nullcontext(explicit)
if _vpn.status().get("status") == "up":
return _vpn.active()
return contextlib.nullcontext(None)
def _go() -> None:
provider = make_provider()
with Db(str(db)) as zdb:
project = load(zdb, name)
typer.echo("==> stage 2 (title/abstract screen)")
s2 = _screen.run(
zdb, provider, project, storage_dir=storage, limit=limit or None
)
typer.echo(f" {s2}")
if not skip_fetch:
typer.echo("==> fetch (PDFs for non-excluded items)")
with _proxy_cm() as fetch_proxy:
fetch_stats = _fetch.run(
zdb,
project=name,
storage_dir=storage,
scratch_dir=Path(".state/prisma-fetch"),
email=os.environ.get("UNPAYWALL_EMAIL", "dev@fhirworx.io"),
fetch_proxy=fetch_proxy,
limit=limit or None,
)
typer.echo(f" {fetch_stats}")
typer.echo("==> stage 3 (full-text eligibility)")
s3 = eligibility.run(
zdb, provider, project, storage_dir=storage, limit=limit or None
)
typer.echo(f" {s3}")
if not skip_extract:
typer.echo("==> stage 3+ (data extraction)")
se = _extract.run(
zdb, provider, project, storage_dir=storage, limit=limit or None
)
typer.echo(f" {se}")
typer.echo("==> flow diagram")
counts = _flow.count(zdb, name)
typer.echo(_flow.text_summary(counts))
_hold(_go, hold=hold)

128
src/cli/zot.py Normal file
View File

@@ -0,0 +1,128 @@
"""stack zot — Zotero database maintenance operations.
All commands expect the ``zotero`` compose service to be stopped while
they run (SQLite file is WAL-locked by the desktop app). ``--hold``
(default on) handles that around each invocation.
"""
from __future__ import annotations
import subprocess
from pathlib import Path
import typer
app = typer.Typer(no_args_is_help=True)
_DEFAULT_DB = Path("data/zotero/data/zotero.sqlite")
def _hold_zotero(fn, hold: bool):
"""Stop the zotero container, run *fn*, restart, return fn's result."""
if not hold:
return fn()
subprocess.run(
["docker", "stop", "zotero"], capture_output=True, check=False, timeout=30
)
try:
return fn()
finally:
subprocess.run(
["docker", "start", "zotero"], capture_output=True, check=False, timeout=30
)
def _resolve(db_path: Path | None) -> Path:
path = db_path or _DEFAULT_DB
if not path.is_file():
raise typer.BadParameter(f"Zotero SQLite not found: {path}")
return path
@app.command(name="dump-schema")
def dump_schema(
db_path: Path = typer.Option(None, "--db", help=f"Default: {_DEFAULT_DB}"),
as_python: bool = typer.Option(
True,
"--python/--json",
help="Emit paste-ready Python literals for zot/db.py (default) or JSON.",
),
) -> None:
"""Dump itemType / field / creatorType IDs.
Regenerate after a Zotero upgrade: the output replaces TYPE_MAP,
FIELD_MAP, and CREATOR_TYPES in ``src/zot/db.py`` verbatim.
"""
import json as _json
from zot.ops import dump_schema as _dump
maps = _dump(_resolve(db_path))
if as_python:
for name, m in maps.items():
typer.echo(f"{name}: dict[str, int] = {{")
for k, v in m.items():
typer.echo(f' "{k}": {v},')
typer.echo("}")
typer.echo("")
else:
typer.echo(_json.dumps(maps, indent=2))
@app.command(name="fix-dates")
def fix_dates(
db_path: Path = typer.Option(None, "--db"),
hold: bool = typer.Option(True, "--hold/--no-hold"),
) -> None:
"""Normalize item timestamps to ISO 8601 + install auto-fix triggers."""
from zot.ops import fix_dates as _op
out = _hold_zotero(lambda: _op(_resolve(db_path)), hold)
for k, v in out.items():
typer.echo(f" {k}: {v}")
@app.command(name="fix-keys")
def fix_keys(
db_path: Path = typer.Option(None, "--db"),
hold: bool = typer.Option(True, "--hold/--no-hold"),
backup: bool = typer.Option(True, "--backup/--no-backup"),
) -> None:
"""Replace invalid Zotero object keys (0/1/O/lowercase → valid)."""
from zot.ops import fix_keys as _op
out = _hold_zotero(
lambda: _op(_resolve(db_path), backup=backup),
hold,
)
for k, v in out.items():
typer.echo(f" {k}: {v}")
@app.command(name="fix-fields")
def fix_fields(
db_path: Path = typer.Option(None, "--db"),
hold: bool = typer.Option(True, "--hold/--no-hold"),
) -> None:
"""Remap base-field rows to type-specific fieldIDs; drop orphans."""
from zot.ops import fix_fields as _op
out = _hold_zotero(lambda: _op(_resolve(db_path)), hold)
for k, v in out.items():
typer.echo(f" {k}: {v}")
@app.command(name="verify-parity")
def verify_parity(
db_path: Path = typer.Option(None, "--db"),
) -> None:
"""Check that ``zot.db`` constants match the live Zotero schema.
The check runs implicitly on every ``Db(...)`` construction; this
command just exposes it explicitly so CI can fail loudly if someone
bumps Zotero without regenerating the maps.
"""
from zot.db import Db
with Db(str(_resolve(db_path))):
typer.echo("parity OK")

12
src/mail/__init__.py Normal file
View File

@@ -0,0 +1,12 @@
"""Touchless mail server for fhirworx — Maddy on DO + Resend smarthost.
Mirrors the corwins.media pattern (port mappings, gotchas, state model
all documented in MEMORY.md → "corwins.media Maddy mail stack"):
provision a single DO droplet named ``mail.fhirworx.io`` (the FQDN-as-
name trick auto-sets PTR), let Maddy handle SMTP/IMAP/DKIM, route
outbound through ``smtp.resend.com:2587`` (DO blocks 25/465/587).
State lives in ``.state/mail/`` as a cache only — every command
re-discovers truth from the DO API by tag ``stack-mail``, so wiping
``.state`` never breaks future runs.
"""

32
src/mail/_helpers.py Normal file
View File

@@ -0,0 +1,32 @@
"""Tiny shared bits: paths, env, output. No third-party deps."""
from __future__ import annotations
import os
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2] # /home/kert/stack
STATE_DIR = ROOT / ".state" / "mail"
DOMAIN = "fhirworx.io"
HOSTNAME = f"mail.{DOMAIN}"
TAG = "stack-mail"
DROPLET_SIZE = "s-1vcpu-1gb"
IMAGE = "ubuntu-24-04-x64"
def env(name: str, default: str = "") -> str:
return os.environ.get(name) or default
def step(msg: str) -> None:
print(f"==> {msg}", flush=True, file=sys.stderr)
def ok(msg: str) -> None:
print(f" {msg}", flush=True, file=sys.stderr)
def warn(msg: str) -> None:
print(f" WARN: {msg}", flush=True, file=sys.stderr)

Some files were not shown because too many files have changed in this diff Show More