feat: full session — mail servers, comment pipeline, PRISMA fetch, email ingest
Some checks failed
CI / skinny-install (aco) (push) Successful in 1m12s
CI / skinny-install (api) (push) Successful in 30s
CI / skinny-install (bcda) (push) Successful in 36s
CI / skinny-install (bib) (push) Successful in 35s
CI / skinny-install (bls) (push) Successful in 27s
CI / skinny-install (ccw) (push) Successful in 32s
CI / skinny-install (cli) (push) Successful in 41s
CI / skinny-install (cms) (push) Successful in 37s
CI / skinny-install (conf) (push) Successful in 38s
CI / skinny-install (opps) (push) Successful in 33s
CI / skinny-install (perf) (push) Successful in 38s
CI / skinny-install (pfs) (push) Successful in 38s
CI / skinny-install (rex) (push) Successful in 34s
Deploy / build-scan-report (push) Failing after 46s
Infra CI / notebooks (push) Failing after 25s
Infra CI / zotero (push) Successful in 12s
Infra CI / docs (push) Failing after 16s
CI / lint-test (push) Failing after 11m2s
Infra CI / mc (push) Successful in 21s
Infra CI / api (push) Successful in 29s
Package Supply Chain / pkg-supply-chain (push) Failing after 41s
Mail: Maddy on DO (corwins.media+Resend, fhirworx.io+Postmark), touchless/stateless/idempotent. Gitea SMTP via env_file. CMS inbox at cmsupdates@mail.fhirworx.io with IMAP→bib poller. Bib: regulations.gov v4 client, Federal Register discovery, 164K comment backfill (running), IMAP email ingest, Zotero sync routing. PRISMA: altcha PoW solver, CrossRef DOI resolution, 83/129 PDFs. Zotero: schema parity, ops module, CLI, fail-fast guard. CI: docs.Dockerfile COPY glob fix (tracks #341). Infra: Gitea+marimo fhirworx themes, IOM/OIG modules.
@@ -13,6 +13,12 @@ tuva/
|
|||||||
dev/
|
dev/
|
||||||
tests/
|
tests/
|
||||||
infra/
|
infra/
|
||||||
|
# Carve out the pieces that feed image builds so they ride along in the
|
||||||
|
# build context. `infra/` is otherwise heavy (certs, state, configs).
|
||||||
|
!infra/marimo/theme/
|
||||||
|
!infra/marimo/theme/**
|
||||||
|
!infra/gitea/
|
||||||
|
!infra/gitea/**
|
||||||
assets/
|
assets/
|
||||||
cloud/
|
cloud/
|
||||||
mirrors/
|
mirrors/
|
||||||
|
|||||||
14
.env.example
@@ -51,3 +51,17 @@ GF_ADMIN_PASSWORD=admin
|
|||||||
|
|
||||||
# ── Woodpecker ──────────────────────────────────────
|
# ── Woodpecker ──────────────────────────────────────
|
||||||
WOODPECKER_ADMIN=kert
|
WOODPECKER_ADMIN=kert
|
||||||
|
|
||||||
|
# ── PRISMA / LLM ───────────────────────────────────
|
||||||
|
# Model-agnostic by design. Flip PRISMA_LLM_PROVIDER to "openai-compat"
|
||||||
|
# (with PRISMA_LLM_BASE_URL pointing at a vLLM/Ollama endpoint) to
|
||||||
|
# swap away from Anthropic — no code changes.
|
||||||
|
PRISMA_LLM_PROVIDER=anthropic
|
||||||
|
PRISMA_LLM_MODEL=claude-opus-4-6
|
||||||
|
PRISMA_LLM_BASE_URL=
|
||||||
|
ANTHROPIC_API_KEY=
|
||||||
|
OPENAI_API_KEY=
|
||||||
|
DIGITAL_OCEAN_PAT= # for `stack prisma vpn up` — droplet for fallback fetches
|
||||||
|
PRISMA_VPN_REGION=nyc3 # DO region slug (reachable from your ISP)
|
||||||
|
PRISMA_FETCH_PROXY= # populated by `stack prisma vpn up`; e.g. socks5://127.0.0.1:1080
|
||||||
|
UNPAYWALL_EMAIL=dev@fhirworx.io # required by Unpaywall API ToS
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ jobs:
|
|||||||
run: curl -sL https://github.com/google/go-containerregistry/releases/latest/download/go-containerregistry_Linux_x86_64.tar.gz | tar xz -C /usr/local/bin crane
|
run: curl -sL https://github.com/google/go-containerregistry/releases/latest/download/go-containerregistry_Linux_x86_64.tar.gz | tar xz -C /usr/local/bin crane
|
||||||
|
|
||||||
- name: Log in to registry
|
- name: Log in to registry
|
||||||
run: crane auth login gitea:3000 -u "${{ secrets.REGISTRY_USER }}" -p "${{ secrets.REGISTRY_TOKEN }}"
|
run: crane auth login git:3000 -u "${{ secrets.REGISTRY_USER }}" -p "${{ secrets.REGISTRY_TOKEN }}"
|
||||||
env:
|
env:
|
||||||
CRANE_INSECURE: "true"
|
CRANE_INSECURE: "true"
|
||||||
|
|
||||||
@@ -39,8 +39,8 @@ jobs:
|
|||||||
- name: Push notebooks
|
- name: Push notebooks
|
||||||
run: |
|
run: |
|
||||||
docker save local/notebooks:build -o /tmp/notebooks.tar
|
docker save local/notebooks:build -o /tmp/notebooks.tar
|
||||||
crane push /tmp/notebooks.tar gitea:3000/homelab/stack/notebooks:${{ env.SHORT_SHA }} --insecure
|
crane push /tmp/notebooks.tar git:3000/homelab/stack/notebooks:${{ env.SHORT_SHA }} --insecure
|
||||||
crane push /tmp/notebooks.tar gitea:3000/homelab/stack/notebooks:latest --insecure
|
crane push /tmp/notebooks.tar git:3000/homelab/stack/notebooks:latest --insecure
|
||||||
|
|
||||||
- name: Build zotero
|
- name: Build zotero
|
||||||
run: docker build -f infra/images/zotero.Dockerfile -t local/zotero:build data/zotero/
|
run: docker build -f infra/images/zotero.Dockerfile -t local/zotero:build data/zotero/
|
||||||
@@ -48,8 +48,8 @@ jobs:
|
|||||||
- name: Push zotero
|
- name: Push zotero
|
||||||
run: |
|
run: |
|
||||||
docker save local/zotero:build -o /tmp/zotero.tar
|
docker save local/zotero:build -o /tmp/zotero.tar
|
||||||
crane push /tmp/zotero.tar gitea:3000/homelab/stack/zotero:${{ env.SHORT_SHA }} --insecure
|
crane push /tmp/zotero.tar git:3000/homelab/stack/zotero:${{ env.SHORT_SHA }} --insecure
|
||||||
crane push /tmp/zotero.tar gitea:3000/homelab/stack/zotero:latest --insecure
|
crane push /tmp/zotero.tar git:3000/homelab/stack/zotero:latest --insecure
|
||||||
|
|
||||||
- name: Build docs
|
- name: Build docs
|
||||||
run: docker build -f infra/images/docs.Dockerfile -t local/docs:build .
|
run: docker build -f infra/images/docs.Dockerfile -t local/docs:build .
|
||||||
@@ -57,8 +57,8 @@ jobs:
|
|||||||
- name: Push docs
|
- name: Push docs
|
||||||
run: |
|
run: |
|
||||||
docker save local/docs:build -o /tmp/docs.tar
|
docker save local/docs:build -o /tmp/docs.tar
|
||||||
crane push /tmp/docs.tar gitea:3000/homelab/stack/docs:${{ env.SHORT_SHA }} --insecure
|
crane push /tmp/docs.tar git:3000/homelab/stack/docs:${{ env.SHORT_SHA }} --insecure
|
||||||
crane push /tmp/docs.tar gitea:3000/homelab/stack/docs:latest --insecure
|
crane push /tmp/docs.tar git:3000/homelab/stack/docs:latest --insecure
|
||||||
|
|
||||||
- name: Build api
|
- name: Build api
|
||||||
run: docker build -f infra/images/api.Dockerfile -t local/api:build .
|
run: docker build -f infra/images/api.Dockerfile -t local/api:build .
|
||||||
@@ -66,8 +66,8 @@ jobs:
|
|||||||
- name: Push api
|
- name: Push api
|
||||||
run: |
|
run: |
|
||||||
docker save local/api:build -o /tmp/api.tar
|
docker save local/api:build -o /tmp/api.tar
|
||||||
crane push /tmp/api.tar gitea:3000/homelab/stack/api:${{ env.SHORT_SHA }} --insecure
|
crane push /tmp/api.tar git:3000/homelab/stack/api:${{ env.SHORT_SHA }} --insecure
|
||||||
crane push /tmp/api.tar gitea:3000/homelab/stack/api:latest --insecure
|
crane push /tmp/api.tar git:3000/homelab/stack/api:latest --insecure
|
||||||
|
|
||||||
- name: Build mc
|
- name: Build mc
|
||||||
run: docker build -f infra/images/mc.Dockerfile -t local/mc:build infra/rustfs/
|
run: docker build -f infra/images/mc.Dockerfile -t local/mc:build infra/rustfs/
|
||||||
@@ -75,8 +75,8 @@ jobs:
|
|||||||
- name: Push mc
|
- name: Push mc
|
||||||
run: |
|
run: |
|
||||||
docker save local/mc:build -o /tmp/mc.tar
|
docker save local/mc:build -o /tmp/mc.tar
|
||||||
crane push /tmp/mc.tar gitea:3000/homelab/stack/mc:${{ env.SHORT_SHA }} --insecure
|
crane push /tmp/mc.tar git:3000/homelab/stack/mc:${{ env.SHORT_SHA }} --insecure
|
||||||
crane push /tmp/mc.tar gitea:3000/homelab/stack/mc:latest --insecure
|
crane push /tmp/mc.tar git:3000/homelab/stack/mc:latest --insecure
|
||||||
|
|
||||||
- name: Scan notebooks
|
- name: Scan notebooks
|
||||||
run: trivy image --severity HIGH,CRITICAL --exit-code 0 --format json -o notebooks-scan.json local/notebooks:build
|
run: trivy image --severity HIGH,CRITICAL --exit-code 0 --format json -o notebooks-scan.json local/notebooks:build
|
||||||
|
|||||||
@@ -19,7 +19,7 @@ jobs:
|
|||||||
run: curl -sL https://github.com/google/go-containerregistry/releases/latest/download/go-containerregistry_Linux_x86_64.tar.gz | tar xz -C /usr/local/bin crane
|
run: curl -sL https://github.com/google/go-containerregistry/releases/latest/download/go-containerregistry_Linux_x86_64.tar.gz | tar xz -C /usr/local/bin crane
|
||||||
|
|
||||||
- name: Log in to registry
|
- name: Log in to registry
|
||||||
run: crane auth login gitea:3000 -u "${{ secrets.REGISTRY_USER }}" -p "${{ secrets.REGISTRY_TOKEN }}"
|
run: crane auth login git:3000 -u "${{ secrets.REGISTRY_USER }}" -p "${{ secrets.REGISTRY_TOKEN }}"
|
||||||
env:
|
env:
|
||||||
CRANE_INSECURE: "true"
|
CRANE_INSECURE: "true"
|
||||||
|
|
||||||
@@ -37,8 +37,8 @@ jobs:
|
|||||||
- name: Push notebooks
|
- name: Push notebooks
|
||||||
run: |
|
run: |
|
||||||
docker save local/notebooks:build -o /tmp/notebooks.tar
|
docker save local/notebooks:build -o /tmp/notebooks.tar
|
||||||
crane push /tmp/notebooks.tar gitea:3000/homelab/stack/notebooks:hardened --insecure
|
crane push /tmp/notebooks.tar git:3000/homelab/stack/notebooks:hardened --insecure
|
||||||
crane push /tmp/notebooks.tar gitea:3000/homelab/stack/notebooks:latest --insecure
|
crane push /tmp/notebooks.tar git:3000/homelab/stack/notebooks:latest --insecure
|
||||||
|
|
||||||
- name: Build zotero
|
- name: Build zotero
|
||||||
run: docker build --no-cache -f infra/images/zotero.Dockerfile -t local/zotero:build data/zotero/
|
run: docker build --no-cache -f infra/images/zotero.Dockerfile -t local/zotero:build data/zotero/
|
||||||
@@ -46,8 +46,8 @@ jobs:
|
|||||||
- name: Push zotero
|
- name: Push zotero
|
||||||
run: |
|
run: |
|
||||||
docker save local/zotero:build -o /tmp/zotero.tar
|
docker save local/zotero:build -o /tmp/zotero.tar
|
||||||
crane push /tmp/zotero.tar gitea:3000/homelab/stack/zotero:hardened --insecure
|
crane push /tmp/zotero.tar git:3000/homelab/stack/zotero:hardened --insecure
|
||||||
crane push /tmp/zotero.tar gitea:3000/homelab/stack/zotero:latest --insecure
|
crane push /tmp/zotero.tar git:3000/homelab/stack/zotero:latest --insecure
|
||||||
|
|
||||||
- name: Build docs
|
- name: Build docs
|
||||||
run: docker build --no-cache -f infra/images/docs.Dockerfile -t local/docs:build .
|
run: docker build --no-cache -f infra/images/docs.Dockerfile -t local/docs:build .
|
||||||
@@ -55,8 +55,8 @@ jobs:
|
|||||||
- name: Push docs
|
- name: Push docs
|
||||||
run: |
|
run: |
|
||||||
docker save local/docs:build -o /tmp/docs.tar
|
docker save local/docs:build -o /tmp/docs.tar
|
||||||
crane push /tmp/docs.tar gitea:3000/homelab/stack/docs:hardened --insecure
|
crane push /tmp/docs.tar git:3000/homelab/stack/docs:hardened --insecure
|
||||||
crane push /tmp/docs.tar gitea:3000/homelab/stack/docs:latest --insecure
|
crane push /tmp/docs.tar git:3000/homelab/stack/docs:latest --insecure
|
||||||
|
|
||||||
- name: Build api
|
- name: Build api
|
||||||
run: docker build --no-cache -f infra/images/api.Dockerfile -t local/api:build .
|
run: docker build --no-cache -f infra/images/api.Dockerfile -t local/api:build .
|
||||||
@@ -64,8 +64,8 @@ jobs:
|
|||||||
- name: Push api
|
- name: Push api
|
||||||
run: |
|
run: |
|
||||||
docker save local/api:build -o /tmp/api.tar
|
docker save local/api:build -o /tmp/api.tar
|
||||||
crane push /tmp/api.tar gitea:3000/homelab/stack/api:hardened --insecure
|
crane push /tmp/api.tar git:3000/homelab/stack/api:hardened --insecure
|
||||||
crane push /tmp/api.tar gitea:3000/homelab/stack/api:latest --insecure
|
crane push /tmp/api.tar git:3000/homelab/stack/api:latest --insecure
|
||||||
|
|
||||||
- name: Build mc
|
- name: Build mc
|
||||||
run: docker build --no-cache -f infra/images/mc.Dockerfile -t local/mc:build infra/rustfs/
|
run: docker build --no-cache -f infra/images/mc.Dockerfile -t local/mc:build infra/rustfs/
|
||||||
@@ -73,8 +73,8 @@ jobs:
|
|||||||
- name: Push mc
|
- name: Push mc
|
||||||
run: |
|
run: |
|
||||||
docker save local/mc:build -o /tmp/mc.tar
|
docker save local/mc:build -o /tmp/mc.tar
|
||||||
crane push /tmp/mc.tar gitea:3000/homelab/stack/mc:hardened --insecure
|
crane push /tmp/mc.tar git:3000/homelab/stack/mc:hardened --insecure
|
||||||
crane push /tmp/mc.tar gitea:3000/homelab/stack/mc:latest --insecure
|
crane push /tmp/mc.tar git:3000/homelab/stack/mc:latest --insecure
|
||||||
|
|
||||||
- name: Scan notebooks
|
- name: Scan notebooks
|
||||||
run: trivy image --severity HIGH,CRITICAL --exit-code 0 --format json -o notebooks-scan.json local/notebooks:build
|
run: trivy image --severity HIGH,CRITICAL --exit-code 0 --format json -o notebooks-scan.json local/notebooks:build
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ jobs:
|
|||||||
run: curl -sL https://github.com/google/go-containerregistry/releases/latest/download/go-containerregistry_Linux_x86_64.tar.gz | tar xz -C /usr/local/bin crane
|
run: curl -sL https://github.com/google/go-containerregistry/releases/latest/download/go-containerregistry_Linux_x86_64.tar.gz | tar xz -C /usr/local/bin crane
|
||||||
|
|
||||||
- name: Log in to registry
|
- name: Log in to registry
|
||||||
run: crane auth login gitea:3000 -u "${{ secrets.REGISTRY_USER }}" -p "${{ secrets.REGISTRY_TOKEN }}"
|
run: crane auth login git:3000 -u "${{ secrets.REGISTRY_USER }}" -p "${{ secrets.REGISTRY_TOKEN }}"
|
||||||
env:
|
env:
|
||||||
CRANE_INSECURE: "true"
|
CRANE_INSECURE: "true"
|
||||||
|
|
||||||
@@ -38,8 +38,8 @@ jobs:
|
|||||||
- name: Push notebooks
|
- name: Push notebooks
|
||||||
run: |
|
run: |
|
||||||
docker save local/notebooks:build -o /tmp/notebooks.tar
|
docker save local/notebooks:build -o /tmp/notebooks.tar
|
||||||
crane push /tmp/notebooks.tar gitea:3000/homelab/stack/notebooks:${{ env.SHORT_SHA }} --insecure
|
crane push /tmp/notebooks.tar git:3000/homelab/stack/notebooks:${{ env.SHORT_SHA }} --insecure
|
||||||
crane push /tmp/notebooks.tar gitea:3000/homelab/stack/notebooks:latest --insecure
|
crane push /tmp/notebooks.tar git:3000/homelab/stack/notebooks:latest --insecure
|
||||||
|
|
||||||
- name: Build zotero
|
- name: Build zotero
|
||||||
run: docker build -f infra/images/zotero.Dockerfile -t local/zotero:build data/zotero/
|
run: docker build -f infra/images/zotero.Dockerfile -t local/zotero:build data/zotero/
|
||||||
@@ -47,8 +47,8 @@ jobs:
|
|||||||
- name: Push zotero
|
- name: Push zotero
|
||||||
run: |
|
run: |
|
||||||
docker save local/zotero:build -o /tmp/zotero.tar
|
docker save local/zotero:build -o /tmp/zotero.tar
|
||||||
crane push /tmp/zotero.tar gitea:3000/homelab/stack/zotero:${{ env.SHORT_SHA }} --insecure
|
crane push /tmp/zotero.tar git:3000/homelab/stack/zotero:${{ env.SHORT_SHA }} --insecure
|
||||||
crane push /tmp/zotero.tar gitea:3000/homelab/stack/zotero:latest --insecure
|
crane push /tmp/zotero.tar git:3000/homelab/stack/zotero:latest --insecure
|
||||||
|
|
||||||
- name: Build docs
|
- name: Build docs
|
||||||
run: docker build -f infra/images/docs.Dockerfile -t local/docs:build .
|
run: docker build -f infra/images/docs.Dockerfile -t local/docs:build .
|
||||||
@@ -56,8 +56,8 @@ jobs:
|
|||||||
- name: Push docs
|
- name: Push docs
|
||||||
run: |
|
run: |
|
||||||
docker save local/docs:build -o /tmp/docs.tar
|
docker save local/docs:build -o /tmp/docs.tar
|
||||||
crane push /tmp/docs.tar gitea:3000/homelab/stack/docs:${{ env.SHORT_SHA }} --insecure
|
crane push /tmp/docs.tar git:3000/homelab/stack/docs:${{ env.SHORT_SHA }} --insecure
|
||||||
crane push /tmp/docs.tar gitea:3000/homelab/stack/docs:latest --insecure
|
crane push /tmp/docs.tar git:3000/homelab/stack/docs:latest --insecure
|
||||||
|
|
||||||
- name: Build api
|
- name: Build api
|
||||||
run: docker build -f infra/images/api.Dockerfile -t local/api:build .
|
run: docker build -f infra/images/api.Dockerfile -t local/api:build .
|
||||||
@@ -65,8 +65,8 @@ jobs:
|
|||||||
- name: Push api
|
- name: Push api
|
||||||
run: |
|
run: |
|
||||||
docker save local/api:build -o /tmp/api.tar
|
docker save local/api:build -o /tmp/api.tar
|
||||||
crane push /tmp/api.tar gitea:3000/homelab/stack/api:${{ env.SHORT_SHA }} --insecure
|
crane push /tmp/api.tar git:3000/homelab/stack/api:${{ env.SHORT_SHA }} --insecure
|
||||||
crane push /tmp/api.tar gitea:3000/homelab/stack/api:latest --insecure
|
crane push /tmp/api.tar git:3000/homelab/stack/api:latest --insecure
|
||||||
|
|
||||||
- name: Build mc
|
- name: Build mc
|
||||||
run: docker build -f infra/images/mc.Dockerfile -t local/mc:build infra/rustfs/
|
run: docker build -f infra/images/mc.Dockerfile -t local/mc:build infra/rustfs/
|
||||||
@@ -74,8 +74,8 @@ jobs:
|
|||||||
- name: Push mc
|
- name: Push mc
|
||||||
run: |
|
run: |
|
||||||
docker save local/mc:build -o /tmp/mc.tar
|
docker save local/mc:build -o /tmp/mc.tar
|
||||||
crane push /tmp/mc.tar gitea:3000/homelab/stack/mc:${{ env.SHORT_SHA }} --insecure
|
crane push /tmp/mc.tar git:3000/homelab/stack/mc:${{ env.SHORT_SHA }} --insecure
|
||||||
crane push /tmp/mc.tar gitea:3000/homelab/stack/mc:latest --insecure
|
crane push /tmp/mc.tar git:3000/homelab/stack/mc:latest --insecure
|
||||||
|
|
||||||
- name: Scan notebooks
|
- name: Scan notebooks
|
||||||
run: trivy image --severity HIGH,CRITICAL --exit-code 0 --format json -o notebooks-scan.json local/notebooks:build
|
run: trivy image --severity HIGH,CRITICAL --exit-code 0 --format json -o notebooks-scan.json local/notebooks:build
|
||||||
|
|||||||
1
.gitignore
vendored
@@ -50,3 +50,4 @@ docs/node_modules/
|
|||||||
docs/build/
|
docs/build/
|
||||||
docs/.docusaurus/
|
docs/.docusaurus/
|
||||||
hw/node_modules/
|
hw/node_modules/
|
||||||
|
infra/marimo/src/
|
||||||
|
|||||||
@@ -1,380 +0,0 @@
|
|||||||
/* HTI-5 — fhirworx design system for Gitea */
|
|
||||||
|
|
||||||
@import url('https://fonts.googleapis.com/css2?family=Playfair+Display:wght@400;600;700&family=Source+Serif+4:ital,wght@0,300;0,400;0,600;1,400&family=JetBrains+Mono:wght@400;500;600&display=swap');
|
|
||||||
|
|
||||||
/* Theme metadata */
|
|
||||||
gitea-theme-meta-info {
|
|
||||||
--theme-display-name: "Fhirworx";
|
|
||||||
}
|
|
||||||
|
|
||||||
:root {
|
|
||||||
--is-dark-theme: false;
|
|
||||||
|
|
||||||
/* HTI-5 Color Palette */
|
|
||||||
--background: #F7F5F0;
|
|
||||||
--foreground: #1A1A18;
|
|
||||||
--card: #FAFAF7;
|
|
||||||
--primary: #1C2B3A;
|
|
||||||
--primary-foreground: #F7F5F0;
|
|
||||||
--secondary: #EDEBE6;
|
|
||||||
--muted-foreground: #6B6B68;
|
|
||||||
--border: #D4D0C8;
|
|
||||||
--sidebar: #1C2B3A;
|
|
||||||
--sidebar-foreground: #B8C5D0;
|
|
||||||
--sidebar-accent: #253748;
|
|
||||||
--destructive: #C0392B;
|
|
||||||
--chart-4: #2E8B6E;
|
|
||||||
--chart-5: #C8702A;
|
|
||||||
|
|
||||||
--font-display: "Playfair Display", Georgia, serif;
|
|
||||||
--font-body: "Source Serif 4", Georgia, serif;
|
|
||||||
--font-mono: "JetBrains Mono", "Fira Code", monospace;
|
|
||||||
|
|
||||||
/* Gitea color mappings — derived from hti5 tokens */
|
|
||||||
--color-primary: var(--primary);
|
|
||||||
--color-primary-light: #2E3D8F;
|
|
||||||
--color-primary-dark: #111c26;
|
|
||||||
--color-primary-alpha-10: color-mix(in srgb, var(--primary) 10%, transparent);
|
|
||||||
--color-primary-alpha-20: color-mix(in srgb, var(--primary) 20%, transparent);
|
|
||||||
--color-primary-alpha-40: color-mix(in srgb, var(--primary) 40%, transparent);
|
|
||||||
|
|
||||||
--color-secondary: var(--muted-foreground);
|
|
||||||
--color-secondary-light: #9B9B98;
|
|
||||||
--color-secondary-dark: var(--foreground);
|
|
||||||
|
|
||||||
/* Background colors */
|
|
||||||
--color-body: var(--background);
|
|
||||||
--color-box-body: var(--card);
|
|
||||||
--color-box-header: var(--secondary);
|
|
||||||
|
|
||||||
/* Text colors */
|
|
||||||
--color-text: var(--foreground);
|
|
||||||
--color-text-light: var(--muted-foreground);
|
|
||||||
--color-text-dark: var(--foreground);
|
|
||||||
|
|
||||||
/* Input colors */
|
|
||||||
--color-input-text: var(--foreground);
|
|
||||||
--color-input-background: var(--card);
|
|
||||||
--color-input-border: var(--border);
|
|
||||||
|
|
||||||
/* Status colors */
|
|
||||||
--color-success: var(--chart-4);
|
|
||||||
--color-success-text: #fff;
|
|
||||||
--color-warning: var(--chart-5);
|
|
||||||
--color-warning-text: #fff;
|
|
||||||
--color-error: var(--destructive);
|
|
||||||
--color-error-text: #fff;
|
|
||||||
--color-info: var(--primary);
|
|
||||||
--color-info-text: var(--primary-foreground);
|
|
||||||
|
|
||||||
/* Border */
|
|
||||||
--color-border: var(--border);
|
|
||||||
|
|
||||||
/* Link colors */
|
|
||||||
--color-link: var(--primary);
|
|
||||||
--color-link-hover: var(--foreground);
|
|
||||||
|
|
||||||
/* Code */
|
|
||||||
--color-code-bg: var(--secondary);
|
|
||||||
--color-code-border: var(--border);
|
|
||||||
|
|
||||||
/* Shadows */
|
|
||||||
--color-shadow: rgba(0, 0, 0, 0.1);
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Global styles */
|
|
||||||
body {
|
|
||||||
font-family: var(--font-body) !important;
|
|
||||||
background: var(--background) !important;
|
|
||||||
color: var(--foreground) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Headings */
|
|
||||||
h1, h2, h3, h4, h5, h6,
|
|
||||||
.header-wrapper .header {
|
|
||||||
font-family: var(--font-display) !important;
|
|
||||||
font-weight: 600 !important;
|
|
||||||
line-height: 1.3 !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
h1 { font-size: 22px !important; color: var(--foreground) !important; }
|
|
||||||
h2 { font-size: 17px !important; color: var(--primary) !important; }
|
|
||||||
h3 { font-size: 14px !important; color: var(--foreground) !important; }
|
|
||||||
|
|
||||||
/* Navbar */
|
|
||||||
.ui.secondary.menu.navbar,
|
|
||||||
.navbar {
|
|
||||||
background: var(--sidebar) !important;
|
|
||||||
border-bottom: 3px solid var(--foreground) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.navbar .item {
|
|
||||||
font-family: var(--font-body) !important;
|
|
||||||
color: var(--sidebar-foreground) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.navbar .item:hover {
|
|
||||||
background: var(--sidebar-accent) !important;
|
|
||||||
color: var(--sidebar-accent-foreground, #EEE9E0) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.navbar .item.active {
|
|
||||||
background: var(--sidebar-accent) !important;
|
|
||||||
color: #fff !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Buttons */
|
|
||||||
.ui.button,
|
|
||||||
.button {
|
|
||||||
font-family: var(--font-body) !important;
|
|
||||||
font-weight: 500 !important;
|
|
||||||
border-radius: var(--radius, 0.2rem) !important;
|
|
||||||
transition: all 0.15s ease !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.primary.button,
|
|
||||||
.ui.blue.button {
|
|
||||||
background: var(--primary) !important;
|
|
||||||
color: var(--primary-foreground) !important;
|
|
||||||
border: 1px solid var(--primary) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.primary.button:hover,
|
|
||||||
.ui.blue.button:hover {
|
|
||||||
background: var(--sidebar-accent) !important;
|
|
||||||
color: #fff !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.green.button {
|
|
||||||
background: var(--chart-4) !important;
|
|
||||||
color: #fff !important;
|
|
||||||
border: 1px solid var(--chart-4) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.green.button:hover {
|
|
||||||
opacity: 0.85;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.red.button {
|
|
||||||
background: var(--destructive) !important;
|
|
||||||
color: #fff !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Segments and boxes */
|
|
||||||
.ui.segment,
|
|
||||||
.ui.segments,
|
|
||||||
.repository,
|
|
||||||
.box {
|
|
||||||
background: var(--card) !important;
|
|
||||||
border: 1px solid var(--border) !important;
|
|
||||||
border-radius: var(--radius, 0.2rem) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.attached.header,
|
|
||||||
.box-header {
|
|
||||||
background: var(--secondary) !important;
|
|
||||||
border-bottom: 1px solid var(--border) !important;
|
|
||||||
font-family: var(--font-display) !important;
|
|
||||||
font-size: 13px !important;
|
|
||||||
color: var(--foreground) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Tables */
|
|
||||||
.ui.table {
|
|
||||||
background: var(--card) !important;
|
|
||||||
border: 1px solid var(--border) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.table th {
|
|
||||||
background: var(--primary) !important;
|
|
||||||
color: var(--primary-foreground) !important;
|
|
||||||
font-family: var(--font-mono) !important;
|
|
||||||
font-size: 10px !important;
|
|
||||||
text-transform: uppercase !important;
|
|
||||||
border: 1px solid var(--sidebar-accent) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.table td {
|
|
||||||
background: var(--card) !important;
|
|
||||||
color: var(--foreground) !important;
|
|
||||||
border: 1px solid var(--border) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.table tr:hover td {
|
|
||||||
background: var(--secondary) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Forms */
|
|
||||||
.ui.form input,
|
|
||||||
.ui.form textarea,
|
|
||||||
.ui.form select,
|
|
||||||
.ui.input input {
|
|
||||||
font-family: var(--font-body) !important;
|
|
||||||
background: var(--card) !important;
|
|
||||||
color: var(--foreground) !important;
|
|
||||||
border: 1px solid var(--border) !important;
|
|
||||||
border-radius: var(--radius, 0.2rem) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.form input:focus,
|
|
||||||
.ui.form textarea:focus,
|
|
||||||
.ui.input input:focus {
|
|
||||||
border-color: var(--primary) !important;
|
|
||||||
background: var(--card) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Labels — mono for metadata labels (issue numbers, ref badges) */
|
|
||||||
.ui.label {
|
|
||||||
font-family: var(--font-mono) !important;
|
|
||||||
font-size: 11px !important;
|
|
||||||
border-radius: var(--radius, 0.2rem) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Repo/issue titles — Playfair */
|
|
||||||
.repository .header,
|
|
||||||
.issue-title,
|
|
||||||
.repository h1,
|
|
||||||
.repository h2,
|
|
||||||
.repository h3 {
|
|
||||||
font-family: var(--font-display) !important;
|
|
||||||
font-weight: 700 !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* PR and issue descriptions — serif body */
|
|
||||||
.comment-content,
|
|
||||||
.issue-content,
|
|
||||||
.render-content {
|
|
||||||
font-family: var(--font-body) !important;
|
|
||||||
font-size: 15px !important;
|
|
||||||
line-height: 1.65 !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Commit hashes, branch names, file paths — mono */
|
|
||||||
.commit-sha,
|
|
||||||
.sha,
|
|
||||||
[class*="sha"],
|
|
||||||
.branch-name,
|
|
||||||
.tag-name,
|
|
||||||
.file-name,
|
|
||||||
code.ref {
|
|
||||||
font-family: var(--font-mono) !important;
|
|
||||||
font-size: 12px !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Dropdowns */
|
|
||||||
.ui.dropdown .menu {
|
|
||||||
background: var(--card) !important;
|
|
||||||
border: 1px solid var(--border) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.dropdown .menu .item {
|
|
||||||
color: var(--foreground) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.dropdown .menu .item:hover {
|
|
||||||
background: var(--secondary) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Code and diffs */
|
|
||||||
.code-view,
|
|
||||||
.file-view,
|
|
||||||
.CodeMirror,
|
|
||||||
.highlight {
|
|
||||||
font-family: var(--font-mono) !important;
|
|
||||||
background: var(--card) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.diff-file-box .diff-file-header {
|
|
||||||
background: var(--secondary) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Commit graph */
|
|
||||||
.repository .commit-list .commit {
|
|
||||||
border-bottom: 1px solid var(--border) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Issues and PRs */
|
|
||||||
.issue-list .item,
|
|
||||||
.pull-list .item {
|
|
||||||
border-bottom: 1px solid var(--border) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Footer */
|
|
||||||
.footer {
|
|
||||||
background: var(--secondary) !important;
|
|
||||||
border-top: 3px solid var(--foreground) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Cards */
|
|
||||||
.ui.card,
|
|
||||||
.ui.cards .card {
|
|
||||||
background: var(--card) !important;
|
|
||||||
border: 1px solid var(--border) !important;
|
|
||||||
border-radius: var(--radius, 0.2rem) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Messages */
|
|
||||||
.ui.message {
|
|
||||||
font-family: var(--font-body) !important;
|
|
||||||
border-radius: var(--radius, 0.2rem) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.positive.message,
|
|
||||||
.ui.success.message {
|
|
||||||
background: color-mix(in srgb, var(--chart-4) 12%, var(--background)) !important;
|
|
||||||
border: 1px solid var(--chart-4) !important;
|
|
||||||
color: var(--chart-4) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.negative.message,
|
|
||||||
.ui.error.message {
|
|
||||||
background: color-mix(in srgb, var(--destructive) 10%, var(--background)) !important;
|
|
||||||
border: 1px solid var(--destructive) !important;
|
|
||||||
color: var(--destructive) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.warning.message {
|
|
||||||
background: color-mix(in srgb, var(--chart-5) 10%, var(--background)) !important;
|
|
||||||
border: 1px solid var(--chart-5) !important;
|
|
||||||
color: var(--chart-5) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
.ui.info.message {
|
|
||||||
background: color-mix(in srgb, var(--primary) 10%, var(--background)) !important;
|
|
||||||
border: 1px solid var(--primary) !important;
|
|
||||||
color: var(--primary) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Scrollbars */
|
|
||||||
::-webkit-scrollbar {
|
|
||||||
width: 10px;
|
|
||||||
height: 10px;
|
|
||||||
}
|
|
||||||
|
|
||||||
::-webkit-scrollbar-track {
|
|
||||||
background: var(--secondary);
|
|
||||||
}
|
|
||||||
|
|
||||||
::-webkit-scrollbar-thumb {
|
|
||||||
background: var(--border);
|
|
||||||
border-radius: 4px;
|
|
||||||
}
|
|
||||||
|
|
||||||
::-webkit-scrollbar-thumb:hover {
|
|
||||||
background: var(--muted-foreground);
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Selection */
|
|
||||||
::selection {
|
|
||||||
background: var(--primary) !important;
|
|
||||||
color: var(--primary-foreground) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Links */
|
|
||||||
a {
|
|
||||||
color: var(--primary) !important;
|
|
||||||
}
|
|
||||||
|
|
||||||
a:hover {
|
|
||||||
color: var(--foreground) !important;
|
|
||||||
}
|
|
||||||
77
compose.yml
@@ -129,13 +129,30 @@ services:
|
|||||||
- no-new-privileges:true
|
- no-new-privileges:true
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|
||||||
gitea:
|
git:
|
||||||
image: gitea/gitea:1.25.4-rootless
|
build:
|
||||||
container_name: gitea
|
context: ./infra/gitea
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
args:
|
||||||
|
GITEA_VERSION: v1.25.4
|
||||||
|
# BUILD_TAG bumps `?v=...` on every asset URL → busts browser/CDN
|
||||||
|
# cache. Use BUILD_TAG=$(date +%s) when invoking docker compose build,
|
||||||
|
# or the Dockerfile defaults to current epoch.
|
||||||
|
BUILD_TAG: "${BUILD_TAG:-}"
|
||||||
|
image: fhirworx/git:v1.25.4
|
||||||
|
container_name: git
|
||||||
networks:
|
networks:
|
||||||
- gateway
|
- gateway
|
||||||
- storage
|
- storage
|
||||||
- ci
|
- ci
|
||||||
|
env_file:
|
||||||
|
# Written by `stack mail wire-git` (or `stack mail provision`) from
|
||||||
|
# the mail droplet's credentials cache. Marked optional so cold-start
|
||||||
|
# works before the mail droplet exists; once it does, re-running
|
||||||
|
# wire-git populates this file and `docker compose up -d git`
|
||||||
|
# picks it up.
|
||||||
|
- path: .state/git/mailer.env
|
||||||
|
required: false
|
||||||
environment:
|
environment:
|
||||||
- GITEA__database__DB_TYPE=postgres
|
- GITEA__database__DB_TYPE=postgres
|
||||||
- GITEA__database__HOST=postgres:5432
|
- GITEA__database__HOST=postgres:5432
|
||||||
@@ -165,13 +182,12 @@ services:
|
|||||||
- GITEA__server__ROOT_URL=https://git.${DOMAIN:-fhirworx.io}/
|
- GITEA__server__ROOT_URL=https://git.${DOMAIN:-fhirworx.io}/
|
||||||
- GITEA__server__SSH_DOMAIN=git.${DOMAIN:-fhirworx.io}
|
- GITEA__server__SSH_DOMAIN=git.${DOMAIN:-fhirworx.io}
|
||||||
- GITEA__webhook__ALLOWED_HOST_LIST=woodpecker-server,ci.${DOMAIN:-fhirworx.io},${HOST_IP:-192.168.1.192},172.19.0.0/16
|
- GITEA__webhook__ALLOWED_HOST_LIST=woodpecker-server,ci.${DOMAIN:-fhirworx.io},${HOST_IP:-192.168.1.192},172.19.0.0/16
|
||||||
- GITEA__ui__THEMES=gitea-auto,gitea-light,gitea-dark,fhirworx
|
- GITEA__ui__THEMES=fhirworx,fhirworx-dark
|
||||||
- GITEA__ui__DEFAULT_THEME=fhirworx
|
- GITEA__ui__DEFAULT_THEME=fhirworx
|
||||||
volumes:
|
volumes:
|
||||||
- gitea_data:/var/lib/gitea
|
- gitea_data:/var/lib/gitea
|
||||||
- gitea_config:/etc/gitea
|
- gitea_config:/etc/gitea
|
||||||
- ./infra/gitea/custom:/var/lib/gitea/custom
|
- ./infra/gitea/custom:/var/lib/gitea/custom
|
||||||
- ./assets/css/gitea.css:/var/lib/gitea/custom/public/assets/css/theme-fhirworx.css:ro
|
|
||||||
ports:
|
ports:
|
||||||
- "2222:2222"
|
- "2222:2222"
|
||||||
- "3000:3000"
|
- "3000:3000"
|
||||||
@@ -208,7 +224,7 @@ services:
|
|||||||
- woodpecker_data:/var/lib/woodpecker
|
- woodpecker_data:/var/lib/woodpecker
|
||||||
- ./assets/css/woodpecker.css:/etc/woodpecker/custom.css:ro
|
- ./assets/css/woodpecker.css:/etc/woodpecker/custom.css:ro
|
||||||
depends_on:
|
depends_on:
|
||||||
- gitea
|
- git
|
||||||
- postgres
|
- postgres
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|
||||||
@@ -240,7 +256,7 @@ services:
|
|||||||
- ci
|
- ci
|
||||||
- storage
|
- storage
|
||||||
environment:
|
environment:
|
||||||
- GITEA_INSTANCE_URL=http://gitea:3000
|
- GITEA_INSTANCE_URL=http://git:3000
|
||||||
- GITEA_RUNNER_REGISTRATION_TOKEN=${ACT_RUNNER_TOKEN}
|
- GITEA_RUNNER_REGISTRATION_TOKEN=${ACT_RUNNER_TOKEN}
|
||||||
- GITEA_RUNNER_NAME=homelab-runner
|
- GITEA_RUNNER_NAME=homelab-runner
|
||||||
- GITEA_RUNNER_LABELS=ubuntu-latest:docker://catthehacker/ubuntu:act-latest
|
- GITEA_RUNNER_LABELS=ubuntu-latest:docker://catthehacker/ubuntu:act-latest
|
||||||
@@ -250,7 +266,7 @@ services:
|
|||||||
- act_runner_data:/data
|
- act_runner_data:/data
|
||||||
- ./infra/act-runner/config.yaml:/config.yaml:ro
|
- ./infra/act-runner/config.yaml:/config.yaml:ro
|
||||||
depends_on:
|
depends_on:
|
||||||
- gitea
|
- git
|
||||||
security_opt:
|
security_opt:
|
||||||
- no-new-privileges:true
|
- no-new-privileges:true
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
@@ -276,8 +292,10 @@ services:
|
|||||||
- POLARIS_ROOT_SECRET=${POLARIS_ROOT_SECRET}
|
- POLARIS_ROOT_SECRET=${POLARIS_ROOT_SECRET}
|
||||||
volumes:
|
volumes:
|
||||||
- ./notebooks:/home/kert/notebooks
|
- ./notebooks:/home/kert/notebooks
|
||||||
- ./infra/marimo:/home/kert/.config/marimo
|
# Mount only user-editable config files, not the whole infra/marimo
|
||||||
- ./infra/marimo/home-page-patched.js:/home/kert/workspace/.venv/lib/python3.13/site-packages/marimo/_static/assets/home-page-itW0tRmv.js:ro
|
# directory — the theme and source lives inside the baked image now.
|
||||||
|
- ./infra/marimo/marimo.toml:/home/kert/.config/marimo/marimo.toml
|
||||||
|
- ./infra/marimo/snippets:/home/kert/.config/marimo/snippets:ro
|
||||||
- ./data:/home/kert/data
|
- ./data:/home/kert/data
|
||||||
- ./data/zotero/data:/home/kert/zotero:ro
|
- ./data/zotero/data:/home/kert/zotero:ro
|
||||||
- ./src:/home/kert/src:ro
|
- ./src:/home/kert/src:ro
|
||||||
@@ -512,6 +530,39 @@ services:
|
|||||||
- no-new-privileges:true
|
- no-new-privileges:true
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|
||||||
|
# IMAP → bib poller. Pulls UNSEEN mail from `cmsupdates@mail.fhirworx.io`
|
||||||
|
# every MAIL_POLL_INTERVAL seconds and upserts each as a Source item.
|
||||||
|
# Idempotent (server-side `\Seen` flag), so frequency is purely a
|
||||||
|
# latency knob — 600s = ~10min average from arrival → bib.
|
||||||
|
mail-poller:
|
||||||
|
image: ${IMAGE_PREFIX:-fhirworx}/api:${COMMIT_SHA:-latest}
|
||||||
|
pull_policy: if_not_present
|
||||||
|
container_name: mail-poller
|
||||||
|
# Needs `gateway` for outbound DNS + IMAPS to mail.fhirworx.io;
|
||||||
|
# `data` to share bib.sqlite with the api/lake services.
|
||||||
|
networks:
|
||||||
|
- gateway
|
||||||
|
- data
|
||||||
|
environment:
|
||||||
|
- MAIL_POLL_INTERVAL=${MAIL_POLL_INTERVAL:-600}
|
||||||
|
volumes:
|
||||||
|
- ./data:/app/data
|
||||||
|
- ./.state:/app/.state
|
||||||
|
# Mount source live so adding mailboxes / tweaking the poller
|
||||||
|
# doesn't require an image rebuild — the api image's baked venv
|
||||||
|
# provides interpreter + deps; --no-sync keeps `uv run` from
|
||||||
|
# going back to pypi.
|
||||||
|
- ./src:/app/src:ro
|
||||||
|
- ./pyproject.toml:/app/pyproject.toml:ro
|
||||||
|
command: >
|
||||||
|
sh -c 'while true; do
|
||||||
|
uv run --no-sync stack bib ingest-mail || true;
|
||||||
|
sleep $${MAIL_POLL_INTERVAL};
|
||||||
|
done'
|
||||||
|
security_opt:
|
||||||
|
- no-new-privileges:true
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
# Observability Stack
|
# Observability Stack
|
||||||
jaeger:
|
jaeger:
|
||||||
image: jaegertracing/all-in-one:latest
|
image: jaegertracing/all-in-one:latest
|
||||||
@@ -709,8 +760,8 @@ services:
|
|||||||
- OAUTH2_PROXY_OIDC_ISSUER_URL=https://git.${DOMAIN:-fhirworx.io}/
|
- OAUTH2_PROXY_OIDC_ISSUER_URL=https://git.${DOMAIN:-fhirworx.io}/
|
||||||
- OAUTH2_PROXY_SKIP_OIDC_DISCOVERY=true
|
- OAUTH2_PROXY_SKIP_OIDC_DISCOVERY=true
|
||||||
- OAUTH2_PROXY_LOGIN_URL=https://git.${DOMAIN:-fhirworx.io}/login/oauth/authorize
|
- OAUTH2_PROXY_LOGIN_URL=https://git.${DOMAIN:-fhirworx.io}/login/oauth/authorize
|
||||||
- OAUTH2_PROXY_REDEEM_URL=http://gitea:3000/login/oauth/access_token
|
- OAUTH2_PROXY_REDEEM_URL=http://git:3000/login/oauth/access_token
|
||||||
- OAUTH2_PROXY_OIDC_JWKS_URL=http://gitea:3000/login/oauth/keys
|
- OAUTH2_PROXY_OIDC_JWKS_URL=http://git:3000/login/oauth/keys
|
||||||
- OAUTH2_PROXY_INSECURE_OIDC_SKIP_ISSUER_VERIFICATION=true
|
- OAUTH2_PROXY_INSECURE_OIDC_SKIP_ISSUER_VERIFICATION=true
|
||||||
- OAUTH2_PROXY_REDIRECT_URL=https://auth.${DOMAIN:-fhirworx.io}/oauth2/callback
|
- OAUTH2_PROXY_REDIRECT_URL=https://auth.${DOMAIN:-fhirworx.io}/oauth2/callback
|
||||||
- OAUTH2_PROXY_COOKIE_DOMAINS=.${DOMAIN:-fhirworx.io}
|
- OAUTH2_PROXY_COOKIE_DOMAINS=.${DOMAIN:-fhirworx.io}
|
||||||
@@ -724,7 +775,7 @@ services:
|
|||||||
- OAUTH2_PROXY_SKIP_PROVIDER_BUTTON=true
|
- OAUTH2_PROXY_SKIP_PROVIDER_BUTTON=true
|
||||||
- OAUTH2_PROXY_CUSTOM_SIGN_IN_LOGO=-
|
- OAUTH2_PROXY_CUSTOM_SIGN_IN_LOGO=-
|
||||||
depends_on:
|
depends_on:
|
||||||
- gitea
|
- git
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD", "oauth2-proxy", "--version"]
|
test: ["CMD", "oauth2-proxy", "--version"]
|
||||||
interval: 30s
|
interval: 30s
|
||||||
|
|||||||
@@ -33,20 +33,24 @@ CARRIER_FILES = {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def add_carrier_year(db: Db, year: int, info: dict) -> None:
|
def add_carrier_year(db: Db, year: int, info: dict) -> None:
|
||||||
"""Add one carrier year to Zotero: parent item + file attachments."""
|
"""Add one carrier year to Zotero: parent item + file attachments."""
|
||||||
now = now_iso()
|
now = now_iso()
|
||||||
|
|
||||||
# Create parent webpage item
|
# Create parent webpage item
|
||||||
parent_id = db.create_item(TYPE_MAP["webpage"], now=now)
|
parent_id = db.create_item(TYPE_MAP["webpage"], now=now)
|
||||||
db.set_fields(parent_id, {
|
db.set_fields(
|
||||||
"title": info["title"],
|
parent_id,
|
||||||
"date": f"{year}-01-01",
|
{
|
||||||
"url": info["url"],
|
"title": info["title"],
|
||||||
"accessDate": now,
|
"date": f"{year}-01-01",
|
||||||
"websiteType": "Government Data Portal",
|
"url": info["url"],
|
||||||
"websiteTitle": "Centers for Medicare & Medicaid Services",
|
"accessDate": now,
|
||||||
})
|
"websiteType": "Government Data Portal",
|
||||||
|
"websiteTitle": "Centers for Medicare & Medicaid Services",
|
||||||
|
},
|
||||||
|
)
|
||||||
db.sync_tags(parent_id, ["module:pfs", f"year:{year}"])
|
db.sync_tags(parent_id, ["module:pfs", f"year:{year}"])
|
||||||
|
|
||||||
print(f"Created parent item for {info['title']}")
|
print(f"Created parent item for {info['title']}")
|
||||||
|
|||||||
140
dev/scripts/add_cf_history_to_zotero.py
Normal file
@@ -0,0 +1,140 @@
|
|||||||
|
"""Add the AMA Medicare Physician CF history PDF to Zotero.
|
||||||
|
|
||||||
|
The document is the authoritative public source for the historical
|
||||||
|
record of CMS PFS conversion factors from CY1992 through present,
|
||||||
|
including the four-CF split (QP APM / non-APM / Anesthesia × 2)
|
||||||
|
that took effect CY2026. It explains that the Final Rule's
|
||||||
|
conversion factor for a given year is derived from the prior year's
|
||||||
|
CF via update factor × budget-neutrality adjustor × performance
|
||||||
|
adjustment, and then **baked into** the published value — which is
|
||||||
|
why ``pfs.rules.RULES[year].conversion_factor`` is already BN-
|
||||||
|
adjusted and should never be multiplied by a separate BN term at
|
||||||
|
payment-calculation time.
|
||||||
|
|
||||||
|
Source:
|
||||||
|
https://www.ama-assn.org/system/files/cf-history.pdf
|
||||||
|
|
||||||
|
Tags:
|
||||||
|
- module:pfs
|
||||||
|
- source:ama
|
||||||
|
- file:cf-history
|
||||||
|
- year:1992 … year:2026 (range covered)
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
uv run python dev/scripts/add_cf_history_to_zotero.py [--pdf PATH]
|
||||||
|
|
||||||
|
The PDF is downloaded from AMA if no local path is supplied.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import subprocess
|
||||||
|
import urllib.request
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from conf import path as _conf_path
|
||||||
|
from zot.db import TYPE_MAP, Db, generate_key, now_iso
|
||||||
|
|
||||||
|
ZOTERO_DB = str(_conf_path("db.zotero"))
|
||||||
|
ZOTERO_STORAGE = str(_conf_path("storage.zotero"))
|
||||||
|
|
||||||
|
CF_HISTORY_URL = "https://www.ama-assn.org/system/files/cf-history.pdf"
|
||||||
|
CF_HISTORY_TITLE = "History of Medicare Physician Payment Schedule Conversion Factors"
|
||||||
|
CF_HISTORY_YEARS = range(1992, 2027) # 1992 → 2026
|
||||||
|
|
||||||
|
|
||||||
|
def _download(dest: Path) -> None:
|
||||||
|
"""Fetch the PDF from AMA if missing."""
|
||||||
|
if dest.exists() and dest.stat().st_size > 0:
|
||||||
|
print(f"Using existing {dest}")
|
||||||
|
return
|
||||||
|
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
print(f"Downloading {CF_HISTORY_URL} → {dest}")
|
||||||
|
urllib.request.urlretrieve(CF_HISTORY_URL, dest) # noqa: S310
|
||||||
|
print(f" {dest.stat().st_size:,} bytes")
|
||||||
|
|
||||||
|
|
||||||
|
def _add_parent_item(db: Db) -> int:
|
||||||
|
"""Create the parent report item + tags. Returns item id."""
|
||||||
|
now = now_iso()
|
||||||
|
item_id = db.create_item(TYPE_MAP["report"], now=now)
|
||||||
|
db.set_fields(
|
||||||
|
item_id,
|
||||||
|
{
|
||||||
|
"title": CF_HISTORY_TITLE,
|
||||||
|
"date": "2026-01-01",
|
||||||
|
"url": CF_HISTORY_URL,
|
||||||
|
"accessDate": now,
|
||||||
|
"institution": "American Medical Association",
|
||||||
|
"reportType": "Data Provenance",
|
||||||
|
"abstractNote": (
|
||||||
|
"Authoritative historical record of the Medicare Physician "
|
||||||
|
"Payment Schedule conversion factor from CY1992 through the "
|
||||||
|
"present. For CY2026, CMS finalised FOUR conversion factors "
|
||||||
|
"(APM / non-APM × standard / anesthesia) under MACRA 2015. "
|
||||||
|
"The published CF for each year already reflects the "
|
||||||
|
"statutory update factor, budget-neutrality adjustor, and "
|
||||||
|
"any performance adjustment — callers should NOT apply BN "
|
||||||
|
"as a separate multiplier at payment-calculation time."
|
||||||
|
),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
tags = ["module:pfs", "source:ama", "file:cf-history"] + [
|
||||||
|
f"year:{y}" for y in CF_HISTORY_YEARS
|
||||||
|
]
|
||||||
|
db.sync_tags(item_id, tags)
|
||||||
|
print(f"Created parent item {item_id} with {len(tags)} tags")
|
||||||
|
return item_id
|
||||||
|
|
||||||
|
|
||||||
|
def _attach_pdf(db: Db, parent_id: int, pdf_path: Path) -> None:
|
||||||
|
"""Copy the PDF into Zotero storage and create the attachment."""
|
||||||
|
att_key = generate_key()
|
||||||
|
storage_dir = Path(ZOTERO_STORAGE) / att_key
|
||||||
|
subprocess.run(["sudo", "mkdir", "-p", str(storage_dir)], check=True)
|
||||||
|
dest = storage_dir / pdf_path.name
|
||||||
|
subprocess.run(["sudo", "cp", str(pdf_path), str(dest)], check=True)
|
||||||
|
subprocess.run(
|
||||||
|
["sudo", "chown", "-R", "100999:100999", str(storage_dir)],
|
||||||
|
check=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
att_id = db.add_attachment(
|
||||||
|
parent_id,
|
||||||
|
key=att_key,
|
||||||
|
content_type="application/pdf",
|
||||||
|
path=f"storage:{pdf_path.name}",
|
||||||
|
)
|
||||||
|
db.set_field(att_id, "title", pdf_path.name)
|
||||||
|
print(f"Attached {pdf_path.name} (attachment id {att_id}, key {att_key})")
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__ or "")
|
||||||
|
ap.add_argument(
|
||||||
|
"--pdf",
|
||||||
|
type=Path,
|
||||||
|
default=Path("/tmp/cf_history/cf-history.pdf"),
|
||||||
|
help="Local path to cf-history.pdf (downloads from AMA if missing).",
|
||||||
|
)
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
_download(args.pdf)
|
||||||
|
|
||||||
|
with Db(ZOTERO_DB) as db:
|
||||||
|
parent_id = _add_parent_item(db)
|
||||||
|
_attach_pdf(db, parent_id, args.pdf)
|
||||||
|
parent_key = db.con.execute(
|
||||||
|
"SELECT key FROM items WHERE itemID = ?", (parent_id,)
|
||||||
|
).fetchone()[0]
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
print()
|
||||||
|
print(f"Done. Parent item key: {parent_key}")
|
||||||
|
print("Cite in docstrings with:")
|
||||||
|
print(f" :pincite:`{parent_key}`")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
164
dev/scripts/add_pprrvu_historical_to_zotero.py
Normal file
@@ -0,0 +1,164 @@
|
|||||||
|
"""Backfill ALL quarterly PPRRVU releases for CY2015–CY2024 into Zotero.
|
||||||
|
|
||||||
|
CMS publishes a fresh PPRRVU file each calendar quarter. Sometimes
|
||||||
|
the CF and/or RVUs carry the same values across all four quarters;
|
||||||
|
sometimes a mid-year CAA retroactive update changes them. Example:
|
||||||
|
|
||||||
|
2015-2023: CF constant across Q1/Q2/Q3/Q4 for a given year.
|
||||||
|
2024: Q1 CF = 32.7442 (original Final Rule),
|
||||||
|
Q2/Q3/Q4 CF = 33.2875 (after CAA 2024 adjustment).
|
||||||
|
|
||||||
|
We archive ALL four quarters per year to preserve provenance and let
|
||||||
|
downstream code pick the release that matches a given carrier file's
|
||||||
|
publication date. See :pincite:`VVBEVYLC` — the AMA CF history table
|
||||||
|
— for the authoritative summary across years.
|
||||||
|
|
||||||
|
Each year's four releases are added to Zotero as a SINGLE parent
|
||||||
|
webpage item with ``year:YYYY`` + ``file:rvu`` + ``source:cms-website``
|
||||||
|
+ ``release:q1/q2/q3/q4`` tags; every release's xlsx/csv/txt trio
|
||||||
|
becomes an attachment on that parent.
|
||||||
|
|
||||||
|
Source zip URLs live in ``/tmp/rvu_urls_final.txt`` (tab-separated
|
||||||
|
``year\\tquarter\\turl``) produced by the CMS PFS RVU index scraper.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
uv run python dev/scripts/add_pprrvu_historical_to_zotero.py
|
||||||
|
|
||||||
|
Assumes the zips have already been downloaded + extracted to
|
||||||
|
``/tmp/pprrvu_dl/all/extracted/{year}{q}/``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from conf import path as _conf_path
|
||||||
|
from zot.db import TYPE_MAP, Db, generate_key, now_iso
|
||||||
|
|
||||||
|
ZOTERO_DB = str(_conf_path("db.zotero"))
|
||||||
|
ZOTERO_STORAGE = str(_conf_path("storage.zotero"))
|
||||||
|
|
||||||
|
URL_TSV = Path("/tmp/rvu_urls_final.txt")
|
||||||
|
EXTRACTED_ROOT = Path("/tmp/pprrvu_dl/all/extracted")
|
||||||
|
|
||||||
|
QUARTER_LABEL = {"a": "Q1", "b": "Q2", "c": "Q3", "d": "Q4"}
|
||||||
|
|
||||||
|
|
||||||
|
def _load_url_map() -> dict[int, dict[str, str]]:
|
||||||
|
"""Parse the tab-separated URL list into {year: {quarter: url}}."""
|
||||||
|
urls: dict[int, dict[str, str]] = {}
|
||||||
|
for line in URL_TSV.read_text().splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if not line or line.startswith("#"):
|
||||||
|
continue
|
||||||
|
parts = line.split("\t")
|
||||||
|
if len(parts) != 3:
|
||||||
|
continue
|
||||||
|
year, q, url = parts
|
||||||
|
urls.setdefault(int(year), {})[q] = url
|
||||||
|
return urls
|
||||||
|
|
||||||
|
|
||||||
|
def add_quarter(db: Db, year: int, q: str, url: str) -> int:
|
||||||
|
"""Create one parent item per (year, quarter).
|
||||||
|
|
||||||
|
Unambiguous ``release:q1..q4`` tag on the parent is what the
|
||||||
|
pipe loader uses to dedupe to the latest quarter per year.
|
||||||
|
"""
|
||||||
|
now = now_iso()
|
||||||
|
parent_id = db.create_item(TYPE_MAP["webpage"], now=now)
|
||||||
|
db.set_fields(
|
||||||
|
parent_id,
|
||||||
|
{
|
||||||
|
"title": (
|
||||||
|
f"CY {year} PFS Q{'abcd'.index(q) + 1} — PPRRVU "
|
||||||
|
f"({QUARTER_LABEL[q]} release)"
|
||||||
|
),
|
||||||
|
"date": f"{year}-01-01",
|
||||||
|
"url": url,
|
||||||
|
"accessDate": now,
|
||||||
|
"websiteType": "Government Data Portal",
|
||||||
|
"websiteTitle": "Centers for Medicare & Medicaid Services",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
db.sync_tags(
|
||||||
|
parent_id,
|
||||||
|
[
|
||||||
|
"module:pfs",
|
||||||
|
"file:rvu",
|
||||||
|
"source:cms-website",
|
||||||
|
f"year:{year}",
|
||||||
|
f"release:{QUARTER_LABEL[q].lower()}",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
extracted_dir = EXTRACTED_ROOT / f"{year}{q}"
|
||||||
|
if not extracted_dir.exists():
|
||||||
|
print(f" {year} {QUARTER_LABEL[q]}: no extracted dir")
|
||||||
|
return parent_id
|
||||||
|
|
||||||
|
attached = 0
|
||||||
|
# Walk recursively; older zips nest files under a subfolder.
|
||||||
|
for filepath in sorted(extracted_dir.rglob("*")):
|
||||||
|
if not filepath.is_file():
|
||||||
|
continue
|
||||||
|
fn = filepath.name
|
||||||
|
if "PPRRVU" not in fn.upper():
|
||||||
|
continue
|
||||||
|
if filepath.suffix.lower() not in (".xlsx", ".csv", ".txt"):
|
||||||
|
continue
|
||||||
|
|
||||||
|
att_key = generate_key()
|
||||||
|
storage_dir = Path(ZOTERO_STORAGE) / att_key
|
||||||
|
subprocess.run(["sudo", "mkdir", "-p", str(storage_dir)], check=True)
|
||||||
|
dest = storage_dir / fn
|
||||||
|
subprocess.run(["sudo", "cp", str(filepath), str(dest)], check=True)
|
||||||
|
subprocess.run(
|
||||||
|
["sudo", "chown", "-R", "100999:100999", str(storage_dir)],
|
||||||
|
check=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
ext = filepath.suffix.lower()
|
||||||
|
content_type = {
|
||||||
|
".xlsx": (
|
||||||
|
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
|
||||||
|
),
|
||||||
|
".csv": "text/csv",
|
||||||
|
".txt": "text/plain",
|
||||||
|
}[ext]
|
||||||
|
|
||||||
|
att_id = db.add_attachment(
|
||||||
|
parent_id,
|
||||||
|
key=att_key,
|
||||||
|
content_type=content_type,
|
||||||
|
path=f"storage:{fn}",
|
||||||
|
)
|
||||||
|
db.set_field(att_id, "title", fn)
|
||||||
|
attached += 1
|
||||||
|
|
||||||
|
print(f" {year} {QUARTER_LABEL[q]}: parent {parent_id}, {attached} files")
|
||||||
|
return parent_id
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
urls = _load_url_map()
|
||||||
|
if not urls:
|
||||||
|
raise SystemExit(f"No URLs parsed from {URL_TSV}")
|
||||||
|
|
||||||
|
if not EXTRACTED_ROOT.exists():
|
||||||
|
raise SystemExit(
|
||||||
|
f"Extracted dir {EXTRACTED_ROOT} not found — download+unzip the zips first."
|
||||||
|
)
|
||||||
|
|
||||||
|
with Db(ZOTERO_DB) as db:
|
||||||
|
for year in sorted(urls):
|
||||||
|
for q in sorted(urls[year]):
|
||||||
|
add_quarter(db, year, q, urls[year][q])
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
print("\nDone. Restart Zotero + re-run pfs ingestion.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -7,13 +7,12 @@ Usage:
|
|||||||
uv run python dev/scripts/add_zipcode_to_zotero.py
|
uv run python dev/scripts/add_zipcode_to_zotero.py
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import sqlite3
|
|
||||||
import subprocess
|
import subprocess
|
||||||
import zipfile
|
import zipfile
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from conf import path as _conf_path
|
from conf import path as _conf_path
|
||||||
from zot.db import FIELD_MAP, TYPE_MAP, Db, generate_key, now_iso
|
from zot.db import TYPE_MAP, Db, generate_key, now_iso
|
||||||
|
|
||||||
ZOTERO_DB = str(_conf_path("db.zotero"))
|
ZOTERO_DB = str(_conf_path("db.zotero"))
|
||||||
ZOTERO_STORAGE = str(_conf_path("storage.zotero"))
|
ZOTERO_STORAGE = str(_conf_path("storage.zotero"))
|
||||||
@@ -76,6 +75,7 @@ ZIPCODE_FILES = {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def add_zipcode_year(db: Db, year: int, info: dict) -> None:
|
def add_zipcode_year(db: Db, year: int, info: dict) -> None:
|
||||||
now = now_iso()
|
now = now_iso()
|
||||||
|
|
||||||
@@ -87,14 +87,17 @@ def add_zipcode_year(db: Db, year: int, info: dict) -> None:
|
|||||||
|
|
||||||
# Create parent webpage item
|
# Create parent webpage item
|
||||||
parent_id = db.create_item(TYPE_MAP["webpage"], now=now)
|
parent_id = db.create_item(TYPE_MAP["webpage"], now=now)
|
||||||
db.set_fields(parent_id, {
|
db.set_fields(
|
||||||
"title": info["title"],
|
parent_id,
|
||||||
"date": f"{year}-01-01",
|
{
|
||||||
"url": info["url"],
|
"title": info["title"],
|
||||||
"accessDate": now,
|
"date": f"{year}-01-01",
|
||||||
"websiteType": "Government Data Portal",
|
"url": info["url"],
|
||||||
"websiteTitle": "Centers for Medicare & Medicaid Services",
|
"accessDate": now,
|
||||||
})
|
"websiteType": "Government Data Portal",
|
||||||
|
"websiteTitle": "Centers for Medicare & Medicaid Services",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
# Tags: module:pfs + year:YYYY
|
# Tags: module:pfs + year:YYYY
|
||||||
db.sync_tags(parent_id, ["module:pfs", f"year:{year}"])
|
db.sync_tags(parent_id, ["module:pfs", f"year:{year}"])
|
||||||
|
|||||||
@@ -135,7 +135,9 @@ def _docker_login_step(registry: str) -> str:
|
|||||||
# ── Workflow generators ──────────────────────────────────────────
|
# ── Workflow generators ──────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
def _gen_ci(runner: str, uv_version: str, coverage_threshold: int = 99, **_kw: object) -> tuple[str, str]:
|
def _gen_ci(
|
||||||
|
runner: str, uv_version: str, coverage_threshold: int = 99, **_kw: object
|
||||||
|
) -> tuple[str, str]:
|
||||||
content = f"""\
|
content = f"""\
|
||||||
{_HEADER}
|
{_HEADER}
|
||||||
name: CI
|
name: CI
|
||||||
|
|||||||
@@ -129,7 +129,9 @@ def _failure_step(workflow_name: str, job_name: str) -> str:
|
|||||||
# ── Workflow generators ───────────────────────────────────────────
|
# ── Workflow generators ───────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
def _gen_ci(runner: str, uv_version: str, coverage_threshold: int = 99, **_kw: object) -> tuple[str, str]:
|
def _gen_ci(
|
||||||
|
runner: str, uv_version: str, coverage_threshold: int = 99, **_kw: object
|
||||||
|
) -> tuple[str, str]:
|
||||||
content = f"""\
|
content = f"""\
|
||||||
{_HEADER}
|
{_HEADER}
|
||||||
name: CI
|
name: CI
|
||||||
|
|||||||
@@ -38,6 +38,7 @@ def _headers(token: str) -> dict:
|
|||||||
# State persistence
|
# State persistence
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
def _load_state() -> dict:
|
def _load_state() -> dict:
|
||||||
if OAUTH_STATE.exists():
|
if OAUTH_STATE.exists():
|
||||||
return json.loads(OAUTH_STATE.read_text())
|
return json.loads(OAUTH_STATE.read_text())
|
||||||
@@ -65,6 +66,7 @@ def _write_env_file(path: Path, content: str) -> bool:
|
|||||||
# Admin user
|
# Admin user
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
def _ensure_admin(client: httpx.Client) -> str | None:
|
def _ensure_admin(client: httpx.Client) -> str | None:
|
||||||
r = client.get(f"{GITEA_API}/user", auth=(ADMIN_USER, ADMIN_PASS))
|
r = client.get(f"{GITEA_API}/user", auth=(ADMIN_USER, ADMIN_PASS))
|
||||||
if r.status_code == 200:
|
if r.status_code == 200:
|
||||||
@@ -72,22 +74,45 @@ def _ensure_admin(client: httpx.Client) -> str | None:
|
|||||||
elif r.status_code == 401:
|
elif r.status_code == 401:
|
||||||
# Try to create; if user exists, change the password instead
|
# Try to create; if user exists, change the password instead
|
||||||
result = subprocess.run(
|
result = subprocess.run(
|
||||||
["docker", "exec", "gitea", "gitea", "admin", "user", "create",
|
[
|
||||||
"--username", ADMIN_USER,
|
"docker",
|
||||||
"--password", ADMIN_PASS,
|
"exec",
|
||||||
"--email", f"{ADMIN_USER}@{DOMAIN}",
|
"gitea",
|
||||||
"--admin"],
|
"gitea",
|
||||||
capture_output=True, text=True,
|
"admin",
|
||||||
|
"user",
|
||||||
|
"create",
|
||||||
|
"--username",
|
||||||
|
ADMIN_USER,
|
||||||
|
"--password",
|
||||||
|
ADMIN_PASS,
|
||||||
|
"--email",
|
||||||
|
f"{ADMIN_USER}@{DOMAIN}",
|
||||||
|
"--admin",
|
||||||
|
],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
)
|
)
|
||||||
if result.returncode == 0:
|
if result.returncode == 0:
|
||||||
print(f" ok: admin '{ADMIN_USER}' created")
|
print(f" ok: admin '{ADMIN_USER}' created")
|
||||||
elif "already exists" in result.stderr:
|
elif "already exists" in result.stderr:
|
||||||
subprocess.run(
|
subprocess.run(
|
||||||
["docker", "exec", "gitea", "gitea", "admin", "user",
|
[
|
||||||
"change-password", "--username", ADMIN_USER,
|
"docker",
|
||||||
"--password", ADMIN_PASS,
|
"exec",
|
||||||
"--must-change-password=false"],
|
"gitea",
|
||||||
capture_output=True, text=True,
|
"gitea",
|
||||||
|
"admin",
|
||||||
|
"user",
|
||||||
|
"change-password",
|
||||||
|
"--username",
|
||||||
|
ADMIN_USER,
|
||||||
|
"--password",
|
||||||
|
ADMIN_PASS,
|
||||||
|
"--must-change-password=false",
|
||||||
|
],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
)
|
)
|
||||||
print(f" ok: admin '{ADMIN_USER}' password synced")
|
print(f" ok: admin '{ADMIN_USER}' password synced")
|
||||||
else:
|
else:
|
||||||
@@ -139,8 +164,12 @@ def _ensure_token(client: httpx.Client) -> str | None:
|
|||||||
# OAuth2 application
|
# OAuth2 application
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
def _ensure_oauth_app(
|
def _ensure_oauth_app(
|
||||||
client: httpx.Client, token: str, name: str, redirect_uri: str,
|
client: httpx.Client,
|
||||||
|
token: str,
|
||||||
|
name: str,
|
||||||
|
redirect_uri: str,
|
||||||
) -> tuple[str, str] | None:
|
) -> tuple[str, str] | None:
|
||||||
headers = _headers(token)
|
headers = _headers(token)
|
||||||
state = _load_state()
|
state = _load_state()
|
||||||
@@ -191,6 +220,7 @@ def _ensure_oauth_app(
|
|||||||
# Downstream wiring
|
# Downstream wiring
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
def _write_oauth2_proxy_env(client_id: str, client_secret: str) -> bool:
|
def _write_oauth2_proxy_env(client_id: str, client_secret: str) -> bool:
|
||||||
state = _load_state()
|
state = _load_state()
|
||||||
cookie_secret = state.get("oauth2_proxy_cookie_secret")
|
cookie_secret = state.get("oauth2_proxy_cookie_secret")
|
||||||
@@ -198,11 +228,14 @@ def _write_oauth2_proxy_env(client_id: str, client_secret: str) -> bool:
|
|||||||
cookie_secret = secrets.token_hex(16)
|
cookie_secret = secrets.token_hex(16)
|
||||||
state["oauth2_proxy_cookie_secret"] = cookie_secret
|
state["oauth2_proxy_cookie_secret"] = cookie_secret
|
||||||
_save_state(state)
|
_save_state(state)
|
||||||
return _write_env_file(OAUTH2_PROXY_ENV, (
|
return _write_env_file(
|
||||||
f"OAUTH2_PROXY_CLIENT_ID={client_id}\n"
|
OAUTH2_PROXY_ENV,
|
||||||
f"OAUTH2_PROXY_CLIENT_SECRET={client_secret}\n"
|
(
|
||||||
f"OAUTH2_PROXY_COOKIE_SECRET={cookie_secret}\n"
|
f"OAUTH2_PROXY_CLIENT_ID={client_id}\n"
|
||||||
))
|
f"OAUTH2_PROXY_CLIENT_SECRET={client_secret}\n"
|
||||||
|
f"OAUTH2_PROXY_COOKIE_SECRET={cookie_secret}\n"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -210,9 +243,26 @@ def _write_oauth2_proxy_env(client_id: str, client_secret: str) -> bool:
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
SUBDOMAINS = [
|
SUBDOMAINS = [
|
||||||
"", "dashboard", "docs", "git", "ci", "notebooks", "zotero",
|
"",
|
||||||
"webdav", "api", "nessie", "trino", "polaris", "grafana",
|
"dashboard",
|
||||||
"prometheus", "jaeger", "loki", "s3", "s3console", "traefik", "auth",
|
"docs",
|
||||||
|
"git",
|
||||||
|
"ci",
|
||||||
|
"notebooks",
|
||||||
|
"zotero",
|
||||||
|
"webdav",
|
||||||
|
"api",
|
||||||
|
"nessie",
|
||||||
|
"trino",
|
||||||
|
"polaris",
|
||||||
|
"grafana",
|
||||||
|
"prometheus",
|
||||||
|
"jaeger",
|
||||||
|
"loki",
|
||||||
|
"s3",
|
||||||
|
"s3console",
|
||||||
|
"traefik",
|
||||||
|
"auth",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
@@ -261,16 +311,21 @@ def _sync_tunnel_dns(client: httpx.Client) -> None:
|
|||||||
ingress = []
|
ingress = []
|
||||||
for sub in SUBDOMAINS:
|
for sub in SUBDOMAINS:
|
||||||
hostname = f"{sub}.{DOMAIN}" if sub else DOMAIN
|
hostname = f"{sub}.{DOMAIN}" if sub else DOMAIN
|
||||||
ingress.append({
|
ingress.append(
|
||||||
"hostname": hostname,
|
{
|
||||||
"service": "http://traefik:80",
|
"hostname": hostname,
|
||||||
"originRequest": {},
|
"service": "http://traefik:80",
|
||||||
})
|
"originRequest": {},
|
||||||
|
}
|
||||||
|
)
|
||||||
ingress.append({"service": "http_status:404", "originRequest": {}})
|
ingress.append({"service": "http_status:404", "originRequest": {}})
|
||||||
|
|
||||||
r = client.put(
|
r = client.put(
|
||||||
api,
|
api,
|
||||||
headers={"Authorization": f"Bearer {token}", "Content-Type": "application/json"},
|
headers={
|
||||||
|
"Authorization": f"Bearer {token}",
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
},
|
||||||
json={"config": {"ingress": ingress}},
|
json={"config": {"ingress": ingress}},
|
||||||
)
|
)
|
||||||
if r.status_code == 200 and r.json().get("success"):
|
if r.status_code == 200 and r.json().get("success"):
|
||||||
@@ -283,6 +338,7 @@ def _sync_tunnel_dns(client: httpx.Client) -> None:
|
|||||||
# Entry point
|
# Entry point
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
if not ADMIN_PASS:
|
if not ADMIN_PASS:
|
||||||
print("ERROR: GITEA_ADMIN_PASSWORD not set")
|
print("ERROR: GITEA_ADMIN_PASSWORD not set")
|
||||||
@@ -291,6 +347,7 @@ def main() -> None:
|
|||||||
print("==> Waiting for Gitea...")
|
print("==> Waiting for Gitea...")
|
||||||
with httpx.Client(timeout=10) as client:
|
with httpx.Client(timeout=10) as client:
|
||||||
import time
|
import time
|
||||||
|
|
||||||
for _ in range(60):
|
for _ in range(60):
|
||||||
try:
|
try:
|
||||||
r = client.get(f"{GITEA_API}/settings/api")
|
r = client.get(f"{GITEA_API}/settings/api")
|
||||||
@@ -309,7 +366,9 @@ def main() -> None:
|
|||||||
sys.exit(1)
|
sys.exit(1)
|
||||||
|
|
||||||
creds = _ensure_oauth_app(
|
creds = _ensure_oauth_app(
|
||||||
client, token, "platform-sso",
|
client,
|
||||||
|
token,
|
||||||
|
"platform-sso",
|
||||||
f"https://auth.{DOMAIN}/oauth2/callback",
|
f"https://auth.{DOMAIN}/oauth2/callback",
|
||||||
)
|
)
|
||||||
if creds:
|
if creds:
|
||||||
@@ -345,7 +404,10 @@ def _ensure_dns(client: httpx.Client) -> None:
|
|||||||
zone = "f8553bde1ddb415b8c3e5dbec4b28330"
|
zone = "f8553bde1ddb415b8c3e5dbec4b28330"
|
||||||
tunnel = "1389035e-d3ba-4a4f-969d-a369c07ee057"
|
tunnel = "1389035e-d3ba-4a4f-969d-a369c07ee057"
|
||||||
tunnel_cname = f"{tunnel}.cfargotunnel.com"
|
tunnel_cname = f"{tunnel}.cfargotunnel.com"
|
||||||
headers = {"Authorization": f"Bearer {cf_token}", "Content-Type": "application/json"}
|
headers = {
|
||||||
|
"Authorization": f"Bearer {cf_token}",
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
}
|
||||||
|
|
||||||
# Fetch existing records
|
# Fetch existing records
|
||||||
r = client.get(
|
r = client.get(
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ def main() -> None:
|
|||||||
con = duckdb.connect(str(DUCKDB_PATH))
|
con = duckdb.connect(str(DUCKDB_PATH))
|
||||||
|
|
||||||
con.execute("DROP TABLE IF EXISTS skin_subs.product_susceptibility")
|
con.execute("DROP TABLE IF EXISTS skin_subs.product_susceptibility")
|
||||||
con.execute(f"""
|
con.execute("""
|
||||||
CREATE TABLE skin_subs.product_susceptibility AS
|
CREATE TABLE skin_subs.product_susceptibility AS
|
||||||
WITH product_base AS (
|
WITH product_base AS (
|
||||||
SELECT
|
SELECT
|
||||||
@@ -198,7 +198,9 @@ def main() -> None:
|
|||||||
FROM skin_subs.product_susceptibility
|
FROM skin_subs.product_susceptibility
|
||||||
GROUP BY susceptibility_tier ORDER BY avg_score DESC
|
GROUP BY susceptibility_tier ORDER BY avg_score DESC
|
||||||
""").fetchall():
|
""").fetchall():
|
||||||
print(f" {r[0]:10s} n={r[1]:3d} avg_score={r[2]:6.3f} avg_asp=${r[3] or 0:>8}")
|
print(
|
||||||
|
f" {r[0]:10s} n={r[1]:3d} avg_score={r[2]:6.3f} avg_asp=${r[3] or 0:>8}"
|
||||||
|
)
|
||||||
|
|
||||||
# Top 20 most susceptible
|
# Top 20 most susceptible
|
||||||
print("\n Top 20 most susceptible products:")
|
print("\n Top 20 most susceptible products:")
|
||||||
@@ -210,9 +212,11 @@ def main() -> None:
|
|||||||
WHERE susceptibility_score IS NOT NULL
|
WHERE susceptibility_score IS NOT NULL
|
||||||
ORDER BY susceptibility_score DESC LIMIT 20
|
ORDER BY susceptibility_score DESC LIMIT 20
|
||||||
""").fetchall():
|
""").fetchall():
|
||||||
print(f" {r[0]} {(r[1] or ''):25s} {(r[2] or ''):20s} "
|
print(
|
||||||
f"score={r[4]:5.3f} ({r[5]}) asp=${r[6] or 0:>8} "
|
f" {r[0]} {(r[1] or ''):25s} {(r[2] or ''):20s} "
|
||||||
f"rcts={r[7]:2d} provs={r[8]:3d} office={r[9] or 0}%")
|
f"score={r[4]:5.3f} ({r[5]}) asp=${r[6] or 0:>8} "
|
||||||
|
f"rcts={r[7]:2d} provs={r[8]:3d} office={r[9] or 0}%"
|
||||||
|
)
|
||||||
|
|
||||||
# Cross-reference: do susceptible products appear in enforcement?
|
# Cross-reference: do susceptible products appear in enforcement?
|
||||||
print("\n Susceptibility by category:")
|
print("\n Susceptibility by category:")
|
||||||
@@ -227,8 +231,10 @@ def main() -> None:
|
|||||||
HAVING count(*) >= 2
|
HAVING count(*) >= 2
|
||||||
ORDER BY avg_score DESC
|
ORDER BY avg_score DESC
|
||||||
""").fetchall():
|
""").fetchall():
|
||||||
print(f" {(r[0] or ''):40s} n={r[1]:3d} score={r[2]:6.3f} "
|
print(
|
||||||
f"asp=${r[3] or 0:>8} rcts={r[4]:3d}")
|
f" {(r[0] or ''):40s} n={r[1]:3d} score={r[2]:6.3f} "
|
||||||
|
f"asp=${r[3] or 0:>8} rcts={r[4]:3d}"
|
||||||
|
)
|
||||||
|
|
||||||
# Final table count
|
# Final table count
|
||||||
print("\n All skin_subs tables:")
|
print("\n All skin_subs tables:")
|
||||||
|
|||||||
@@ -93,23 +93,27 @@ def build_asp_trajectories(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
ORDER BY hcpcs_code, quarter
|
ORDER BY hcpcs_code, quarter
|
||||||
""")
|
""")
|
||||||
|
|
||||||
count = con.execute(
|
count = con.execute("SELECT count(*) FROM skin_subs.asp_trajectories").fetchone()[0]
|
||||||
"SELECT count(*) FROM skin_subs.asp_trajectories"
|
|
||||||
).fetchone()[0]
|
|
||||||
print(f" Rows: {count}")
|
print(f" Rows: {count}")
|
||||||
|
|
||||||
# Summary stats
|
# Summary stats
|
||||||
for label, q in [
|
for label, q in [
|
||||||
("Products with anomalous spikes",
|
(
|
||||||
"SELECT count(DISTINCT hcpcs_code) FROM skin_subs.asp_trajectories WHERE anomalous_spike"),
|
"Products with anomalous spikes",
|
||||||
("Products that lose under flat rate (latest quarter)",
|
"SELECT count(DISTINCT hcpcs_code) FROM skin_subs.asp_trajectories WHERE anomalous_spike",
|
||||||
"""SELECT count(DISTINCT hcpcs_code) FROM skin_subs.asp_trajectories
|
),
|
||||||
|
(
|
||||||
|
"Products that lose under flat rate (latest quarter)",
|
||||||
|
"""SELECT count(DISTINCT hcpcs_code) FROM skin_subs.asp_trajectories
|
||||||
WHERE flat_rate_impact = 'loses'
|
WHERE flat_rate_impact = 'loses'
|
||||||
AND quarter = (SELECT max(quarter) FROM skin_subs.asp_trajectories)"""),
|
AND quarter = (SELECT max(quarter) FROM skin_subs.asp_trajectories)""",
|
||||||
("Products that gain under flat rate (latest quarter)",
|
),
|
||||||
"""SELECT count(DISTINCT hcpcs_code) FROM skin_subs.asp_trajectories
|
(
|
||||||
|
"Products that gain under flat rate (latest quarter)",
|
||||||
|
"""SELECT count(DISTINCT hcpcs_code) FROM skin_subs.asp_trajectories
|
||||||
WHERE flat_rate_impact = 'gains'
|
WHERE flat_rate_impact = 'gains'
|
||||||
AND quarter = (SELECT max(quarter) FROM skin_subs.asp_trajectories)"""),
|
AND quarter = (SELECT max(quarter) FROM skin_subs.asp_trajectories)""",
|
||||||
|
),
|
||||||
]:
|
]:
|
||||||
val = con.execute(q).fetchone()[0]
|
val = con.execute(q).fetchone()[0]
|
||||||
print(f" {label}: {val}")
|
print(f" {label}: {val}")
|
||||||
@@ -123,7 +127,9 @@ def build_asp_trajectories(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
WHERE quarter = (SELECT max(quarter) FROM skin_subs.asp_trajectories)
|
WHERE quarter = (SELECT max(quarter) FROM skin_subs.asp_trajectories)
|
||||||
ORDER BY flat_rate_delta DESC LIMIT 10
|
ORDER BY flat_rate_delta DESC LIMIT 10
|
||||||
""").fetchall():
|
""").fetchall():
|
||||||
print(f" {r[0]} {(r[1] or ''):30s} {(r[2] or ''):20s} pay=${r[3]:>8} delta=${r[4]:>+8}")
|
print(
|
||||||
|
f" {r[0]} {(r[1] or ''):30s} {(r[2] or ''):20s} pay=${r[3]:>8} delta=${r[4]:>+8}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def build_market_segmentation(con: duckdb.DuckDBPyConnection) -> None:
|
def build_market_segmentation(con: duckdb.DuckDBPyConnection) -> None:
|
||||||
@@ -202,9 +208,9 @@ def build_market_segmentation(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
ORDER BY cs.total_paid DESC NULLS LAST
|
ORDER BY cs.total_paid DESC NULLS LAST
|
||||||
""")
|
""")
|
||||||
|
|
||||||
count = con.execute(
|
count = con.execute("SELECT count(*) FROM skin_subs.products_enriched").fetchone()[
|
||||||
"SELECT count(*) FROM skin_subs.products_enriched"
|
0
|
||||||
).fetchone()[0]
|
]
|
||||||
print(f" Products: {count}")
|
print(f" Products: {count}")
|
||||||
|
|
||||||
# Manufacturer summary
|
# Manufacturer summary
|
||||||
@@ -236,7 +242,9 @@ def build_market_segmentation(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
ORDER BY total_revenue DESC LIMIT 10
|
ORDER BY total_revenue DESC LIMIT 10
|
||||||
""").fetchall():
|
""").fetchall():
|
||||||
rev = f"${r[3]:,.0f}" if r[3] else "n/a"
|
rev = f"${r[3]:,.0f}" if r[3] else "n/a"
|
||||||
print(f" {(r[0] or ''):30s} prods={r[1]:3d} active={r[2]:3d} rev={rev:>12} avg_asp=${r[4] or 0:>8}")
|
print(
|
||||||
|
f" {(r[0] or ''):30s} prods={r[1]:3d} active={r[2]:3d} rev={rev:>12} avg_asp=${r[4] or 0:>8}"
|
||||||
|
)
|
||||||
|
|
||||||
print("\n Products by category:")
|
print("\n Products by category:")
|
||||||
for r in con.execute("""
|
for r in con.execute("""
|
||||||
@@ -247,7 +255,9 @@ def build_market_segmentation(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
WHERE category IS NOT NULL
|
WHERE category IS NOT NULL
|
||||||
GROUP BY category ORDER BY n DESC
|
GROUP BY category ORDER BY n DESC
|
||||||
""").fetchall():
|
""").fetchall():
|
||||||
print(f" {(r[0] or ''):40s} n={r[1]:3d} claims={r[2]:3d} avg_asp=${r[3] or 0:>8}")
|
print(
|
||||||
|
f" {(r[0] or ''):40s} n={r[1]:3d} claims={r[2]:3d} avg_asp=${r[3] or 0:>8}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def build_utilization_metrics(con: duckdb.DuckDBPyConnection) -> None:
|
def build_utilization_metrics(con: duckdb.DuckDBPyConnection) -> None:
|
||||||
@@ -327,8 +337,12 @@ def build_utilization_metrics(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
|
|
||||||
# Print summaries
|
# Print summaries
|
||||||
prov_count = con.execute("SELECT count(*) FROM skin_subs.providers").fetchone()[0]
|
prov_count = con.execute("SELECT count(*) FROM skin_subs.providers").fetchone()[0]
|
||||||
bene_count = con.execute("SELECT count(*) FROM skin_subs.beneficiaries").fetchone()[0]
|
bene_count = con.execute("SELECT count(*) FROM skin_subs.beneficiaries").fetchone()[
|
||||||
util_count = con.execute("SELECT count(*) FROM skin_subs.utilization_summary").fetchone()[0]
|
0
|
||||||
|
]
|
||||||
|
util_count = con.execute(
|
||||||
|
"SELECT count(*) FROM skin_subs.utilization_summary"
|
||||||
|
).fetchone()[0]
|
||||||
print(f" Providers: {prov_count}")
|
print(f" Providers: {prov_count}")
|
||||||
print(f" Beneficiaries: {bene_count}")
|
print(f" Beneficiaries: {bene_count}")
|
||||||
print(f" Utilization summary rows: {util_count}")
|
print(f" Utilization summary rows: {util_count}")
|
||||||
@@ -339,7 +353,9 @@ def build_utilization_metrics(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
unique_patients, total_paid, paid_per_patient
|
unique_patients, total_paid, paid_per_patient
|
||||||
FROM skin_subs.providers ORDER BY total_paid DESC LIMIT 10
|
FROM skin_subs.providers ORDER BY total_paid DESC LIMIT 10
|
||||||
""").fetchall():
|
""").fetchall():
|
||||||
print(f" NPI {r[0]} {r[1]:15s} {r[2]} pts={r[3]:3d} paid=${r[4]:>10,.2f} per_pt=${r[5]:>8,.2f}")
|
print(
|
||||||
|
f" NPI {r[0]} {r[1]:15s} {r[2]} pts={r[3]:3d} paid=${r[4]:>10,.2f} per_pt=${r[5]:>8,.2f}"
|
||||||
|
)
|
||||||
|
|
||||||
print("\n Utilization by setting:")
|
print("\n Utilization by setting:")
|
||||||
for r in con.execute("""
|
for r in con.execute("""
|
||||||
@@ -348,7 +364,9 @@ def build_utilization_metrics(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
FROM skin_subs.utilization_summary
|
FROM skin_subs.utilization_summary
|
||||||
GROUP BY setting ORDER BY paid DESC
|
GROUP BY setting ORDER BY paid DESC
|
||||||
""").fetchall():
|
""").fetchall():
|
||||||
print(f" {r[0]:15s} lines={r[1]:5d} paid=${r[2]:>12,.2f} per_bene=${r[3]:>8,.2f}")
|
print(
|
||||||
|
f" {r[0]:15s} lines={r[1]:5d} paid=${r[2]:>12,.2f} per_bene=${r[3]:>8,.2f}"
|
||||||
|
)
|
||||||
|
|
||||||
print("\n Utilization by specialty:")
|
print("\n Utilization by specialty:")
|
||||||
for r in con.execute("""
|
for r in con.execute("""
|
||||||
|
|||||||
@@ -9,7 +9,6 @@ Usage:
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import math
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import duckdb
|
import duckdb
|
||||||
@@ -41,10 +40,10 @@ KNOWN_FRAUD_PATTERNS = {
|
|||||||
|
|
||||||
# Setting risk weights (higher = more fraud-prone based on OIG findings)
|
# Setting risk weights (higher = more fraud-prone based on OIG findings)
|
||||||
SETTING_RISK = {
|
SETTING_RISK = {
|
||||||
"office": 1.5, # Lowest oversight, wound care mills
|
"office": 1.5, # Lowest oversight, wound care mills
|
||||||
"snf": 1.3, # Kickback-prone, captive patients
|
"snf": 1.3, # Kickback-prone, captive patients
|
||||||
"asc": 0.8, # Moderate oversight
|
"asc": 0.8, # Moderate oversight
|
||||||
"hopd": 0.5, # Institutional controls, auditable
|
"hopd": 0.5, # Institutional controls, auditable
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -120,10 +119,10 @@ def build_anomaly_scores(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
ELSE 0 END AS units_zscore,
|
ELSE 0 END AS units_zscore,
|
||||||
-- Setting risk weight
|
-- Setting risk weight
|
||||||
CASE ps.primary_setting
|
CASE ps.primary_setting
|
||||||
WHEN 'office' THEN {SETTING_RISK['office']}
|
WHEN 'office' THEN {SETTING_RISK["office"]}
|
||||||
WHEN 'snf' THEN {SETTING_RISK['snf']}
|
WHEN 'snf' THEN {SETTING_RISK["snf"]}
|
||||||
WHEN 'asc' THEN {SETTING_RISK['asc']}
|
WHEN 'asc' THEN {SETTING_RISK["asc"]}
|
||||||
WHEN 'hopd' THEN {SETTING_RISK['hopd']}
|
WHEN 'hopd' THEN {SETTING_RISK["hopd"]}
|
||||||
ELSE 1.0 END AS setting_risk,
|
ELSE 1.0 END AS setting_risk,
|
||||||
-- Geographic risk (state per-bene spend z-score)
|
-- Geographic risk (state per-bene spend z-score)
|
||||||
(SELECT round((g.paid_per_bene - agg.m) / nullif(agg.s, 0), 2)
|
(SELECT round((g.paid_per_bene - agg.m) / nullif(agg.s, 0), 2)
|
||||||
@@ -178,7 +177,9 @@ def build_anomaly_scores(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
FROM skin_subs.anomaly_scores
|
FROM skin_subs.anomaly_scores
|
||||||
GROUP BY risk_tier ORDER BY avg_score DESC
|
GROUP BY risk_tier ORDER BY avg_score DESC
|
||||||
""").fetchall():
|
""").fetchall():
|
||||||
print(f" {r[0]:10s} n={r[1]:3d} avg_paid=${r[2]:>10,.2f} avg_score={r[3]}")
|
print(
|
||||||
|
f" {r[0]:10s} n={r[1]:3d} avg_paid=${r[2]:>10,.2f} avg_score={r[3]}"
|
||||||
|
)
|
||||||
|
|
||||||
# Top risk providers
|
# Top risk providers
|
||||||
print("\n Top 10 highest-risk providers:")
|
print("\n Top 10 highest-risk providers:")
|
||||||
@@ -189,9 +190,11 @@ def build_anomaly_scores(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
FROM skin_subs.anomaly_scores
|
FROM skin_subs.anomaly_scores
|
||||||
ORDER BY composite_risk DESC LIMIT 10
|
ORDER BY composite_risk DESC LIMIT 10
|
||||||
""").fetchall():
|
""").fetchall():
|
||||||
print(f" NPI {r[0]} {r[1]:15s} {r[2]} {r[3]:8s} "
|
print(
|
||||||
f"paid=${r[4]:>10,.2f} per_pt=${r[5]:>8,.2f} "
|
f" NPI {r[0]} {r[1]:15s} {r[2]} {r[3]:8s} "
|
||||||
f"risk={r[6]:5.2f} ({r[7]}) vol_z={r[8]:+5.2f} int_z={r[9]:+5.2f}")
|
f"paid=${r[4]:>10,.2f} per_pt=${r[5]:>8,.2f} "
|
||||||
|
f"risk={r[6]:5.2f} ({r[7]}) vol_z={r[8]:+5.2f} int_z={r[9]:+5.2f}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def build_enforcement_correlation(con: duckdb.DuckDBPyConnection) -> None:
|
def build_enforcement_correlation(con: duckdb.DuckDBPyConnection) -> None:
|
||||||
@@ -258,7 +261,9 @@ def build_enforcement_correlation(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
WHERE fraud_pattern_match IS NOT NULL
|
WHERE fraud_pattern_match IS NOT NULL
|
||||||
GROUP BY fraud_pattern_match ORDER BY avg_risk DESC
|
GROUP BY fraud_pattern_match ORDER BY avg_risk DESC
|
||||||
""").fetchall():
|
""").fetchall():
|
||||||
print(f" {r[0]:15s} n={r[1]:3d} avg_risk={r[2]:5.2f} avg_paid=${r[3]:>10,.2f}")
|
print(
|
||||||
|
f" {r[0]:15s} n={r[1]:3d} avg_risk={r[2]:5.2f} avg_paid=${r[3]:>10,.2f}"
|
||||||
|
)
|
||||||
|
|
||||||
# Do fraud-pattern providers score higher than peers?
|
# Do fraud-pattern providers score higher than peers?
|
||||||
print("\n Risk scores: fraud-pattern vs non-match:")
|
print("\n Risk scores: fraud-pattern vs non-match:")
|
||||||
@@ -273,10 +278,14 @@ def build_enforcement_correlation(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
FROM skin_subs.provider_risk_profile
|
FROM skin_subs.provider_risk_profile
|
||||||
GROUP BY group_name
|
GROUP BY group_name
|
||||||
""").fetchall():
|
""").fetchall():
|
||||||
print(f" {r[0]:10s} n={r[1]:3d} risk={r[2]:6.3f} vol_z={r[3]:+6.3f} int_z={r[4]:+6.3f}")
|
print(
|
||||||
|
f" {r[0]:10s} n={r[1]:3d} risk={r[2]:6.3f} vol_z={r[3]:+6.3f} int_z={r[4]:+6.3f}"
|
||||||
|
)
|
||||||
|
|
||||||
# Benford analysis summary
|
# Benford analysis summary
|
||||||
print("\n Benford's law chi² (lower = more conformant, >15.5 = suspicious at p<0.05):")
|
print(
|
||||||
|
"\n Benford's law chi² (lower = more conformant, >15.5 = suspicious at p<0.05):"
|
||||||
|
)
|
||||||
for r in con.execute("""
|
for r in con.execute("""
|
||||||
SELECT risk_tier, count(*) as n,
|
SELECT risk_tier, count(*) as n,
|
||||||
round(avg(benford_chi2), 4) as avg_chi2,
|
round(avg(benford_chi2), 4) as avg_chi2,
|
||||||
|
|||||||
@@ -56,26 +56,73 @@ SKIN_SUBS_COLLECTIONS = {
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
MANUFACTURERS = [
|
MANUFACTURERS = [
|
||||||
"organogenesis", "mimedx", "smith nephew", "smith & nephew",
|
"organogenesis",
|
||||||
"integra", "solsys", "amnioexcel", "derma sciences",
|
"mimedx",
|
||||||
"healthpoint", "shire", "acelity", "kci", "3m",
|
"smith nephew",
|
||||||
"molnlycke", "medline", "hollister", "coloplast",
|
"smith & nephew",
|
||||||
"stryker", "zimmer biomet", "wright medical",
|
"integra",
|
||||||
"solventum", "apria", "anika", "musculoskeletal transplant",
|
"solsys",
|
||||||
"surmodics", "tissue regenix", "nuo therapeutics",
|
"amnioexcel",
|
||||||
"sanara medtech", "kerecis", "human bioprocessing",
|
"derma sciences",
|
||||||
"alphatec", "biosig technologies",
|
"healthpoint",
|
||||||
|
"shire",
|
||||||
|
"acelity",
|
||||||
|
"kci",
|
||||||
|
"3m",
|
||||||
|
"molnlycke",
|
||||||
|
"medline",
|
||||||
|
"hollister",
|
||||||
|
"coloplast",
|
||||||
|
"stryker",
|
||||||
|
"zimmer biomet",
|
||||||
|
"wright medical",
|
||||||
|
"solventum",
|
||||||
|
"apria",
|
||||||
|
"anika",
|
||||||
|
"musculoskeletal transplant",
|
||||||
|
"surmodics",
|
||||||
|
"tissue regenix",
|
||||||
|
"nuo therapeutics",
|
||||||
|
"sanara medtech",
|
||||||
|
"kerecis",
|
||||||
|
"human bioprocessing",
|
||||||
|
"alphatec",
|
||||||
|
"biosig technologies",
|
||||||
]
|
]
|
||||||
|
|
||||||
# Brand names that indicate manufacturer-linked studies
|
# Brand names that indicate manufacturer-linked studies
|
||||||
BRAND_NAMES = [
|
BRAND_NAMES = [
|
||||||
"apligraf", "dermagraft", "epifix", "grafix", "amnioexcel",
|
"apligraf",
|
||||||
"dermacell", "oasis", "primatrix", "integra", "graftjacket",
|
"dermagraft",
|
||||||
"dermapure", "affinity", "biovance", "cytal", "endoform",
|
"epifix",
|
||||||
"kerecis omega3", "novafix", "puraply", "restorigin",
|
"grafix",
|
||||||
"surgicraft", "theraskin", "amnioburn", "clarix",
|
"amnioexcel",
|
||||||
"epicord", "genesis", "grafix core", "grafix prime",
|
"dermacell",
|
||||||
"innovamatrix", "nushield", "stravix", "woundex",
|
"oasis",
|
||||||
|
"primatrix",
|
||||||
|
"integra",
|
||||||
|
"graftjacket",
|
||||||
|
"dermapure",
|
||||||
|
"affinity",
|
||||||
|
"biovance",
|
||||||
|
"cytal",
|
||||||
|
"endoform",
|
||||||
|
"kerecis omega3",
|
||||||
|
"novafix",
|
||||||
|
"puraply",
|
||||||
|
"restorigin",
|
||||||
|
"surgicraft",
|
||||||
|
"theraskin",
|
||||||
|
"amnioburn",
|
||||||
|
"clarix",
|
||||||
|
"epicord",
|
||||||
|
"genesis",
|
||||||
|
"grafix core",
|
||||||
|
"grafix prime",
|
||||||
|
"innovamatrix",
|
||||||
|
"nushield",
|
||||||
|
"stravix",
|
||||||
|
"woundex",
|
||||||
]
|
]
|
||||||
|
|
||||||
# Patterns suggesting industry funding
|
# Patterns suggesting industry funding
|
||||||
@@ -206,9 +253,7 @@ def detect_coi(extra: str, abstract: str, title: str) -> list[str]:
|
|||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
parser = argparse.ArgumentParser(
|
parser = argparse.ArgumentParser(description="Build skin-subs evidence base")
|
||||||
description="Build skin-subs evidence base"
|
|
||||||
)
|
|
||||||
parser.add_argument("--dry-run", action="store_true")
|
parser.add_argument("--dry-run", action="store_true")
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
@@ -261,19 +306,13 @@ def main() -> None:
|
|||||||
|
|
||||||
for row in rows:
|
for row in rows:
|
||||||
tags = item_tags.get(row["id"], [])
|
tags = item_tags.get(row["id"], [])
|
||||||
col_name = assign_collection(
|
col_name = assign_collection(tags, row["title"] or "", row["abstract"] or "")
|
||||||
tags, row["title"] or "", row["abstract"] or ""
|
|
||||||
)
|
|
||||||
if col_name in col_map:
|
if col_name in col_map:
|
||||||
assignments.append((row["id"], col_map[col_name]))
|
assignments.append((row["id"], col_map[col_name]))
|
||||||
collection_counts[col_name] = (
|
collection_counts[col_name] = collection_counts.get(col_name, 0) + 1
|
||||||
collection_counts.get(col_name, 0) + 1
|
|
||||||
)
|
|
||||||
|
|
||||||
print(" Assignment distribution:")
|
print(" Assignment distribution:")
|
||||||
for name, count in sorted(
|
for name, count in sorted(collection_counts.items(), key=lambda x: -x[1]):
|
||||||
collection_counts.items(), key=lambda x: -x[1]
|
|
||||||
):
|
|
||||||
print(f" {name:30s}: {count:>5}")
|
print(f" {name:30s}: {count:>5}")
|
||||||
|
|
||||||
# --- Step 4: COI / funding enrichment ---
|
# --- Step 4: COI / funding enrichment ---
|
||||||
@@ -326,8 +365,7 @@ def main() -> None:
|
|||||||
for tag in new_tags:
|
for tag in new_tags:
|
||||||
tag_id = store._ensure_tag(tag)
|
tag_id = store._ensure_tag(tag)
|
||||||
con.execute(
|
con.execute(
|
||||||
"INSERT OR IGNORE INTO item_tags (item_id, tag_id) "
|
"INSERT OR IGNORE INTO item_tags (item_id, tag_id) VALUES (?, ?)",
|
||||||
"VALUES (?, ?)",
|
|
||||||
(item_id, tag_id),
|
(item_id, tag_id),
|
||||||
)
|
)
|
||||||
total_tags_added += 1
|
total_tags_added += 1
|
||||||
@@ -340,13 +378,27 @@ def main() -> None:
|
|||||||
# Tag schema coverage
|
# Tag schema coverage
|
||||||
expected_tags = [
|
expected_tags = [
|
||||||
"module:skin-subs",
|
"module:skin-subs",
|
||||||
"source:pubmed", "source:oig", "source:cms", "source:court",
|
"source:pubmed",
|
||||||
"source:doj", "source:gao", "source:medpac",
|
"source:oig",
|
||||||
"source:mac-lcd", "source:industry",
|
"source:cms",
|
||||||
"type:clinical", "type:economic", "type:fraud",
|
"source:court",
|
||||||
"type:rct", "type:review", "type:meta-analysis",
|
"source:doj",
|
||||||
"type:report", "type:rule", "type:filing",
|
"source:gao",
|
||||||
"type:lcd", "type:position", "type:press-release",
|
"source:medpac",
|
||||||
|
"source:mac-lcd",
|
||||||
|
"source:industry",
|
||||||
|
"type:clinical",
|
||||||
|
"type:economic",
|
||||||
|
"type:fraud",
|
||||||
|
"type:rct",
|
||||||
|
"type:review",
|
||||||
|
"type:meta-analysis",
|
||||||
|
"type:report",
|
||||||
|
"type:rule",
|
||||||
|
"type:filing",
|
||||||
|
"type:lcd",
|
||||||
|
"type:position",
|
||||||
|
"type:press-release",
|
||||||
]
|
]
|
||||||
for tag in expected_tags:
|
for tag in expected_tags:
|
||||||
count = con.execute(
|
count = con.execute(
|
||||||
|
|||||||
@@ -23,7 +23,21 @@ MAC_JURISDICTIONS = {
|
|||||||
"CGS (J15)": ["KY", "OH"],
|
"CGS (J15)": ["KY", "OH"],
|
||||||
"WPS (J5/J8)": ["IA", "IN", "KS", "MI", "MO", "NE"],
|
"WPS (J5/J8)": ["IA", "IN", "KS", "MI", "MO", "NE"],
|
||||||
"NGS (J6/JK)": ["CT", "IL", "MA", "ME", "MN", "NH", "NY", "RI", "VT", "WI"],
|
"NGS (J6/JK)": ["CT", "IL", "MA", "ME", "MN", "NH", "NY", "RI", "VT", "WI"],
|
||||||
"Noridian (JE/JF)": ["AK", "AZ", "CA", "HI", "ID", "MT", "ND", "NV", "OR", "SD", "UT", "WA", "WY"],
|
"Noridian (JE/JF)": [
|
||||||
|
"AK",
|
||||||
|
"AZ",
|
||||||
|
"CA",
|
||||||
|
"HI",
|
||||||
|
"ID",
|
||||||
|
"MT",
|
||||||
|
"ND",
|
||||||
|
"NV",
|
||||||
|
"OR",
|
||||||
|
"SD",
|
||||||
|
"UT",
|
||||||
|
"WA",
|
||||||
|
"WY",
|
||||||
|
],
|
||||||
}
|
}
|
||||||
|
|
||||||
# Invert: state → MAC
|
# Invert: state → MAC
|
||||||
@@ -72,15 +86,14 @@ def build_geographic_summary(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
ORDER BY total_paid DESC
|
ORDER BY total_paid DESC
|
||||||
""")
|
""")
|
||||||
|
|
||||||
count = con.execute(
|
count = con.execute("SELECT count(*) FROM skin_subs.geographic_summary").fetchone()[
|
||||||
"SELECT count(*) FROM skin_subs.geographic_summary"
|
0
|
||||||
).fetchone()[0]
|
]
|
||||||
print(f" States: {count}")
|
print(f" States: {count}")
|
||||||
|
|
||||||
# Add MAC jurisdiction column
|
# Add MAC jurisdiction column
|
||||||
mac_cases = " ".join(
|
mac_cases = " ".join(
|
||||||
f"WHEN state = '{st}' THEN '{mac}'"
|
f"WHEN state = '{st}' THEN '{mac}'" for st, mac in STATE_TO_MAC.items()
|
||||||
for st, mac in STATE_TO_MAC.items()
|
|
||||||
)
|
)
|
||||||
con.execute(f"""
|
con.execute(f"""
|
||||||
ALTER TABLE skin_subs.geographic_summary
|
ALTER TABLE skin_subs.geographic_summary
|
||||||
@@ -110,9 +123,11 @@ def build_geographic_summary(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
|
|
||||||
print("\n By MAC jurisdiction:")
|
print("\n By MAC jurisdiction:")
|
||||||
for r in con.execute("SELECT * FROM skin_subs.mac_summary").fetchall():
|
for r in con.execute("SELECT * FROM skin_subs.mac_summary").fetchall():
|
||||||
print(f" {r[0]:25s} states={r[1]:2d} lines={r[2]:5d} "
|
print(
|
||||||
f"paid=${r[5]:>10,.2f} per_bene=${r[6]:>8,.2f} "
|
f" {r[0]:25s} states={r[1]:2d} lines={r[2]:5d} "
|
||||||
f"office={r[7]}% podiatry={r[8]}%")
|
f"paid=${r[5]:>10,.2f} per_bene=${r[6]:>8,.2f} "
|
||||||
|
f"office={r[7]}% podiatry={r[8]}%"
|
||||||
|
)
|
||||||
|
|
||||||
print("\n Top 5 states by spend per beneficiary:")
|
print("\n Top 5 states by spend per beneficiary:")
|
||||||
for r in con.execute("""
|
for r in con.execute("""
|
||||||
@@ -121,8 +136,10 @@ def build_geographic_summary(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
FROM skin_subs.geographic_summary
|
FROM skin_subs.geographic_summary
|
||||||
ORDER BY paid_per_bene DESC LIMIT 5
|
ORDER BY paid_per_bene DESC LIMIT 5
|
||||||
""").fetchall():
|
""").fetchall():
|
||||||
print(f" {r[0]} {r[1]:25s} benes={r[2]:4d} "
|
print(
|
||||||
f"per_bene=${r[4]:>8,.2f} office={r[5]}% podiatry={r[6]}%")
|
f" {r[0]} {r[1]:25s} benes={r[2]:4d} "
|
||||||
|
f"per_bene=${r[4]:>8,.2f} office={r[5]}% podiatry={r[6]}%"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def build_setting_analysis(con: duckdb.DuckDBPyConnection) -> None:
|
def build_setting_analysis(con: duckdb.DuckDBPyConnection) -> None:
|
||||||
@@ -175,9 +192,7 @@ def build_setting_analysis(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
ORDER BY total_paid DESC
|
ORDER BY total_paid DESC
|
||||||
""")
|
""")
|
||||||
|
|
||||||
count = con.execute(
|
count = con.execute("SELECT count(*) FROM skin_subs.setting_analysis").fetchone()[0]
|
||||||
"SELECT count(*) FROM skin_subs.setting_analysis"
|
|
||||||
).fetchone()[0]
|
|
||||||
print(f" Setting×specialty×product combinations: {count}")
|
print(f" Setting×specialty×product combinations: {count}")
|
||||||
|
|
||||||
# Setting summary
|
# Setting summary
|
||||||
@@ -191,8 +206,10 @@ def build_setting_analysis(con: duckdb.DuckDBPyConnection) -> None:
|
|||||||
FROM skin_subs.setting_analysis
|
FROM skin_subs.setting_analysis
|
||||||
GROUP BY setting ORDER BY paid DESC
|
GROUP BY setting ORDER BY paid DESC
|
||||||
""").fetchall():
|
""").fetchall():
|
||||||
print(f" {r[0]:15s} lines={r[1]:5d} provs={r[2]:4d} "
|
print(
|
||||||
f"paid=${r[3]:>12,.2f} units={r[4]:4.1f} per_bene=${r[5]:>8,.2f}")
|
f" {r[0]:15s} lines={r[1]:5d} provs={r[2]:4d} "
|
||||||
|
f"paid=${r[3]:>12,.2f} units={r[4]:4.1f} per_bene=${r[5]:>8,.2f}"
|
||||||
|
)
|
||||||
|
|
||||||
# Specialty×setting cross-tab
|
# Specialty×setting cross-tab
|
||||||
print("\n Specialty × setting (total paid):")
|
print("\n Specialty × setting (total paid):")
|
||||||
|
|||||||
@@ -14,7 +14,6 @@ import sqlite3
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from conf import path as conf_path
|
from conf import path as conf_path
|
||||||
from zot.db import FIELD_MAP
|
|
||||||
|
|
||||||
ZOTERO_DB = str(conf_path("db.zotero"))
|
ZOTERO_DB = str(conf_path("db.zotero"))
|
||||||
|
|
||||||
@@ -54,7 +53,9 @@ def main() -> None:
|
|||||||
|
|
||||||
print(f"Total skin-subs items: {total:>6}")
|
print(f"Total skin-subs items: {total:>6}")
|
||||||
print(f"Items with DOI: {with_doi:>6} ({with_doi * 100 // total}%)")
|
print(f"Items with DOI: {with_doi:>6} ({with_doi * 100 // total}%)")
|
||||||
print(f"Items with PDF attached: {with_pdf:>6} ({with_pdf * 100 // max(total, 1)}%)")
|
print(
|
||||||
|
f"Items with PDF attached: {with_pdf:>6} ({with_pdf * 100 // max(total, 1)}%)"
|
||||||
|
)
|
||||||
print(f"DOIs without PDF: {with_doi - with_pdf:>6}")
|
print(f"DOIs without PDF: {with_doi - with_pdf:>6}")
|
||||||
print(f"Coverage (of DOI items): {with_pdf * 100 // max(with_doi, 1)}%")
|
print(f"Coverage (of DOI items): {with_pdf * 100 // max(with_doi, 1)}%")
|
||||||
|
|
||||||
@@ -75,7 +76,9 @@ def main() -> None:
|
|||||||
ORDER BY total DESC
|
ORDER BY total DESC
|
||||||
""").fetchall():
|
""").fetchall():
|
||||||
pct = r["with_pdf"] * 100 // max(r["total"], 1)
|
pct = r["with_pdf"] * 100 // max(r["total"], 1)
|
||||||
print(f" {r['typeName']:20s} total={r['total']:>5} pdf={r['with_pdf']:>5} ({pct}%)")
|
print(
|
||||||
|
f" {r['typeName']:20s} total={r['total']:>5} pdf={r['with_pdf']:>5} ({pct}%)"
|
||||||
|
)
|
||||||
|
|
||||||
# Storage size
|
# Storage size
|
||||||
storage = Path(conf_path("storage.zotero"))
|
storage = Path(conf_path("storage.zotero"))
|
||||||
|
|||||||
@@ -20,7 +20,7 @@ import argparse
|
|||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
|
||||||
from bib.item import Rule, Source
|
from bib.item import Source
|
||||||
from bib.store import Store
|
from bib.store import Store
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -158,8 +158,7 @@ CMS_RULES = [
|
|||||||
),
|
),
|
||||||
GreyLitEntry(
|
GreyLitEntry(
|
||||||
title=(
|
title=(
|
||||||
"CY 2024 PFS Final Rule (CMS-1784-F) — Skin Substitute "
|
"CY 2024 PFS Final Rule (CMS-1784-F) — Skin Substitute Payment Under Part B"
|
||||||
"Payment Under Part B"
|
|
||||||
),
|
),
|
||||||
url="https://www.federalregister.gov/documents/2023/11/16/2023-24184/medicare-and-medicaid-programs-cy-2024-payment-policies-under-the-physician-fee-schedule",
|
url="https://www.federalregister.gov/documents/2023/11/16/2023-24184/medicare-and-medicaid-programs-cy-2024-payment-policies-under-the-physician-fee-schedule",
|
||||||
source_tag="cms",
|
source_tag="cms",
|
||||||
@@ -296,8 +295,7 @@ DOJ_ENFORCEMENT = [
|
|||||||
),
|
),
|
||||||
GreyLitEntry(
|
GreyLitEntry(
|
||||||
title=(
|
title=(
|
||||||
"USA v. Gehrke & King (D. Ariz.) — $1.2B Mobile Wound Care "
|
"USA v. Gehrke & King (D. Ariz.) — $1.2B Mobile Wound Care Fraud Scheme"
|
||||||
"Fraud Scheme"
|
|
||||||
),
|
),
|
||||||
url="https://www.justice.gov/usao-az/pr/two-individuals-charged-12-billion-health-care-fraud-scheme-involving-mobile-wound-care",
|
url="https://www.justice.gov/usao-az/pr/two-individuals-charged-12-billion-health-care-fraud-scheme-involving-mobile-wound-care",
|
||||||
source_tag="court",
|
source_tag="court",
|
||||||
@@ -314,10 +312,7 @@ DOJ_ENFORCEMENT = [
|
|||||||
extra_tags=["case:gehrke-king", "entity:daz"],
|
extra_tags=["case:gehrke-king", "entity:daz"],
|
||||||
),
|
),
|
||||||
GreyLitEntry(
|
GreyLitEntry(
|
||||||
title=(
|
title=("USA v. Azar Nasser (E.D. Mich.) — $60M Skin Substitute Fraud Ring"),
|
||||||
"USA v. Azar Nasser (E.D. Mich.) — $60M Skin Substitute "
|
|
||||||
"Fraud Ring"
|
|
||||||
),
|
|
||||||
url="https://www.justice.gov/usao-edmi/pr/metro-detroit-physician-charged-60-million-health-care-fraud-scheme",
|
url="https://www.justice.gov/usao-edmi/pr/metro-detroit-physician-charged-60-million-health-care-fraud-scheme",
|
||||||
source_tag="court",
|
source_tag="court",
|
||||||
type_tag="filing",
|
type_tag="filing",
|
||||||
@@ -403,8 +398,7 @@ GAO_MEDPAC = [
|
|||||||
),
|
),
|
||||||
GreyLitEntry(
|
GreyLitEntry(
|
||||||
title=(
|
title=(
|
||||||
"MedPAC March 2025 Report to Congress — Payment for "
|
"MedPAC March 2025 Report to Congress — Payment for Wound Care Products"
|
||||||
"Wound Care Products"
|
|
||||||
),
|
),
|
||||||
url="https://www.medpac.gov/document/march-2025-report-to-the-congress/",
|
url="https://www.medpac.gov/document/march-2025-report-to-the-congress/",
|
||||||
source_tag="medpac",
|
source_tag="medpac",
|
||||||
@@ -559,7 +553,9 @@ INDUSTRY = [
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
ALL_ENTRIES = OIG_REPORTS + CMS_RULES + DOJ_ENFORCEMENT + GAO_MEDPAC + MAC_LCDS + INDUSTRY
|
ALL_ENTRIES = (
|
||||||
|
OIG_REPORTS + CMS_RULES + DOJ_ENFORCEMENT + GAO_MEDPAC + MAC_LCDS + INDUSTRY
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -598,8 +594,11 @@ def entry_to_item(entry: GreyLitEntry) -> Source:
|
|||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
parser = argparse.ArgumentParser(description="Grey literature collection")
|
parser = argparse.ArgumentParser(description="Grey literature collection")
|
||||||
parser.add_argument("--dry-run", action="store_true",
|
parser.add_argument(
|
||||||
help="Print catalogue only, don't write to bib.sqlite")
|
"--dry-run",
|
||||||
|
action="store_true",
|
||||||
|
help="Print catalogue only, don't write to bib.sqlite",
|
||||||
|
)
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
print("=" * 70)
|
print("=" * 70)
|
||||||
|
|||||||
@@ -60,8 +60,9 @@ def download_file(url: str, dest: Path) -> bool:
|
|||||||
return True
|
return True
|
||||||
try:
|
try:
|
||||||
print(f" GET {dest.name} ...", end="", flush=True)
|
print(f" GET {dest.name} ...", end="", flush=True)
|
||||||
resp = httpx.get(url, headers={"User-Agent": USER_AGENT},
|
resp = httpx.get(
|
||||||
timeout=60, follow_redirects=True)
|
url, headers={"User-Agent": USER_AGENT}, timeout=60, follow_redirects=True
|
||||||
|
)
|
||||||
if resp.status_code == 200:
|
if resp.status_code == 200:
|
||||||
dest.write_bytes(resp.content)
|
dest.write_bytes(resp.content)
|
||||||
size_mb = len(resp.content) / 1024 / 1024
|
size_mb = len(resp.content) / 1024 / 1024
|
||||||
@@ -112,7 +113,9 @@ def main() -> None:
|
|||||||
print(f"Downloaded: {downloaded}/{total} files")
|
print(f"Downloaded: {downloaded}/{total} files")
|
||||||
print(f"Location: {OPPS_DIR}")
|
print(f"Location: {OPPS_DIR}")
|
||||||
if set(OPPS_ADDENDA_URLS) != set(str(y) for y in range(2014, 2027)):
|
if set(OPPS_ADDENDA_URLS) != set(str(y) for y in range(2014, 2027)):
|
||||||
missing = sorted(set(str(y) for y in range(2014, 2027)) - set(OPPS_ADDENDA_URLS))
|
missing = sorted(
|
||||||
|
set(str(y) for y in range(2014, 2027)) - set(OPPS_ADDENDA_URLS)
|
||||||
|
)
|
||||||
print(f"\nMissing years (purged from CMS): {', '.join(missing)}")
|
print(f"\nMissing years (purged from CMS): {', '.join(missing)}")
|
||||||
print(" These may be recoverable from the Wayback Machine.")
|
print(" These may be recoverable from the Wayback Machine.")
|
||||||
|
|
||||||
@@ -122,8 +125,10 @@ def main() -> None:
|
|||||||
files = list(year_dir.glob("*.zip"))
|
files = list(year_dir.glob("*.zip"))
|
||||||
if files:
|
if files:
|
||||||
total_size = sum(f.stat().st_size for f in files)
|
total_size = sum(f.stat().st_size for f in files)
|
||||||
print(f" {year_dir.name}: {len(files)} files "
|
print(
|
||||||
f"({total_size / 1024 / 1024:.1f} MB)")
|
f" {year_dir.name}: {len(files)} files "
|
||||||
|
f"({total_size / 1024 / 1024:.1f} MB)"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
29
dev/scripts/dump_zot_schema.py
Executable file
@@ -0,0 +1,29 @@
|
|||||||
|
"""Thin shim — logic moved to :mod:`zot.ops`.
|
||||||
|
|
||||||
|
Prefer: ``uv run stack zot dump-schema``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
from zot.ops import dump_schema
|
||||||
|
|
||||||
|
if len(sys.argv) != 2:
|
||||||
|
print(f"usage: {sys.argv[0]} <path/to/zotero.sqlite>", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
maps = dump_schema(Path(sys.argv[1]))
|
||||||
|
for name, m in maps.items():
|
||||||
|
print(f"{name}: dict[str, int] = {{")
|
||||||
|
for k, v in m.items():
|
||||||
|
print(f' "{k}": {v},')
|
||||||
|
print("}")
|
||||||
|
print()
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -134,9 +134,7 @@ def main() -> None:
|
|||||||
# --- Step 2: Assign snowball articles to collections ---
|
# --- Step 2: Assign snowball articles to collections ---
|
||||||
print("\n--- Assigning snowball articles to collections ---")
|
print("\n--- Assigning snowball articles to collections ---")
|
||||||
# Find collection keys
|
# Find collection keys
|
||||||
col_rows = con.execute(
|
col_rows = con.execute("SELECT key, name FROM collections").fetchall()
|
||||||
"SELECT key, name FROM collections"
|
|
||||||
).fetchall()
|
|
||||||
col_name_to_key = {r["name"]: r["key"] for r in col_rows}
|
col_name_to_key = {r["name"]: r["key"] for r in col_rows}
|
||||||
|
|
||||||
snowball_items = con.execute(
|
snowball_items = con.execute(
|
||||||
@@ -163,7 +161,9 @@ def main() -> None:
|
|||||||
(col_id_row["id"], item_row["id"]),
|
(col_id_row["id"], item_row["id"]),
|
||||||
)
|
)
|
||||||
con.commit()
|
con.commit()
|
||||||
print(f" Assigned {len(snowball_items)} snowball articles to Clinical Evidence")
|
print(
|
||||||
|
f" Assigned {len(snowball_items)} snowball articles to Clinical Evidence"
|
||||||
|
)
|
||||||
else:
|
else:
|
||||||
print(" No unassigned snowball articles or collection not found")
|
print(" No unassigned snowball articles or collection not found")
|
||||||
|
|
||||||
@@ -183,67 +183,69 @@ def main() -> None:
|
|||||||
).fetchall()
|
).fetchall()
|
||||||
tag_list = [t["name"] for t in fresh_tags]
|
tag_list = [t["name"] for t in fresh_tags]
|
||||||
|
|
||||||
evidence_rows.append({
|
evidence_rows.append(
|
||||||
"bib_key": row["key"],
|
{
|
||||||
"title": (row["title"] or "")[:300],
|
"bib_key": row["key"],
|
||||||
"url": row["url"] or "",
|
"title": (row["title"] or "")[:300],
|
||||||
"date_published": row["date_published"] or "",
|
"url": row["url"] or "",
|
||||||
"institution": row["institution"] or "",
|
"date_published": row["date_published"] or "",
|
||||||
"item_type": row["item_type"] or "",
|
"institution": row["institution"] or "",
|
||||||
"tags": "; ".join(sorted(tag_list)),
|
"item_type": row["item_type"] or "",
|
||||||
"source_tag": next(
|
"tags": "; ".join(sorted(tag_list)),
|
||||||
(t for t in tag_list if t.startswith("source:")), ""
|
"source_tag": next(
|
||||||
),
|
(t for t in tag_list if t.startswith("source:")), ""
|
||||||
"type_tags": "; ".join(
|
),
|
||||||
t for t in tag_list if t.startswith("type:")
|
"type_tags": "; ".join(t for t in tag_list if t.startswith("type:")),
|
||||||
),
|
"entity_tags": "; ".join(
|
||||||
"entity_tags": "; ".join(
|
t for t in tag_list if t.startswith("entity:")
|
||||||
t for t in tag_list if t.startswith("entity:")
|
),
|
||||||
),
|
"is_snowball": "source:snowball" in tag_list,
|
||||||
"is_snowball": "source:snowball" in tag_list,
|
"has_abstract": bool(row["abstract"]),
|
||||||
"has_abstract": bool(row["abstract"]),
|
}
|
||||||
})
|
)
|
||||||
|
|
||||||
import pyarrow as pa
|
import pyarrow as pa
|
||||||
|
|
||||||
schema = pa.schema([
|
schema = pa.schema(
|
||||||
("bib_key", pa.string()),
|
[
|
||||||
("title", pa.string()),
|
("bib_key", pa.string()),
|
||||||
("url", pa.string()),
|
("title", pa.string()),
|
||||||
("date_published", pa.string()),
|
("url", pa.string()),
|
||||||
("institution", pa.string()),
|
("date_published", pa.string()),
|
||||||
("item_type", pa.string()),
|
("institution", pa.string()),
|
||||||
("tags", pa.string()),
|
("item_type", pa.string()),
|
||||||
("source_tag", pa.string()),
|
("tags", pa.string()),
|
||||||
("type_tags", pa.string()),
|
("source_tag", pa.string()),
|
||||||
("entity_tags", pa.string()),
|
("type_tags", pa.string()),
|
||||||
("is_snowball", pa.bool_()),
|
("entity_tags", pa.string()),
|
||||||
("has_abstract", pa.bool_()),
|
("is_snowball", pa.bool_()),
|
||||||
])
|
("has_abstract", pa.bool_()),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
arrays = [pa.array([r[f.name] for r in evidence_rows]) for f in schema]
|
arrays = [pa.array([r[f.name] for r in evidence_rows]) for f in schema]
|
||||||
arrow_tbl = pa.table(
|
arrow_tbl = pa.table(dict(zip([f.name for f in schema], arrays)), schema=schema)
|
||||||
dict(zip([f.name for f in schema], arrays)), schema=schema
|
|
||||||
)
|
|
||||||
|
|
||||||
ddb = duckdb.connect(str(DUCKDB_PATH))
|
ddb = duckdb.connect(str(DUCKDB_PATH))
|
||||||
ddb.execute("CREATE SCHEMA IF NOT EXISTS skin_subs")
|
ddb.execute("CREATE SCHEMA IF NOT EXISTS skin_subs")
|
||||||
ddb.execute("DROP TABLE IF EXISTS skin_subs.evidence_base")
|
ddb.execute("DROP TABLE IF EXISTS skin_subs.evidence_base")
|
||||||
ddb.register("arrow_tbl", arrow_tbl)
|
ddb.register("arrow_tbl", arrow_tbl)
|
||||||
ddb.execute(
|
ddb.execute("CREATE TABLE skin_subs.evidence_base AS SELECT * FROM arrow_tbl")
|
||||||
"CREATE TABLE skin_subs.evidence_base AS SELECT * FROM arrow_tbl"
|
count = ddb.execute("SELECT count(*) FROM skin_subs.evidence_base").fetchone()[0]
|
||||||
)
|
|
||||||
count = ddb.execute(
|
|
||||||
"SELECT count(*) FROM skin_subs.evidence_base"
|
|
||||||
).fetchone()[0]
|
|
||||||
print(f" Loaded {count} rows into skin_subs.evidence_base")
|
print(f" Loaded {count} rows into skin_subs.evidence_base")
|
||||||
|
|
||||||
# Validation
|
# Validation
|
||||||
print("\n Validation:")
|
print("\n Validation:")
|
||||||
for q, label in [
|
for q, label in [
|
||||||
("SELECT source_tag, count(*) c FROM skin_subs.evidence_base GROUP BY 1 ORDER BY c DESC LIMIT 5", "by_source"),
|
(
|
||||||
|
"SELECT source_tag, count(*) c FROM skin_subs.evidence_base GROUP BY 1 ORDER BY c DESC LIMIT 5",
|
||||||
|
"by_source",
|
||||||
|
),
|
||||||
("SELECT count(*) FROM skin_subs.evidence_base WHERE is_snowball", "snowball"),
|
("SELECT count(*) FROM skin_subs.evidence_base WHERE is_snowball", "snowball"),
|
||||||
("SELECT count(*) FROM skin_subs.evidence_base WHERE entity_tags != ''", "with_entities"),
|
(
|
||||||
|
"SELECT count(*) FROM skin_subs.evidence_base WHERE entity_tags != ''",
|
||||||
|
"with_entities",
|
||||||
|
),
|
||||||
]:
|
]:
|
||||||
print(f" {label}: {ddb.execute(q).fetchall()}")
|
print(f" {label}: {ddb.execute(q).fetchall()}")
|
||||||
|
|
||||||
@@ -254,9 +256,7 @@ def main() -> None:
|
|||||||
"WHERE table_schema = 'skin_subs' ORDER BY table_name"
|
"WHERE table_schema = 'skin_subs' ORDER BY table_name"
|
||||||
).fetchall()
|
).fetchall()
|
||||||
for t in tbls:
|
for t in tbls:
|
||||||
cnt = ddb.execute(
|
cnt = ddb.execute(f"SELECT count(*) FROM skin_subs.{t[0]}").fetchone()[0]
|
||||||
f"SELECT count(*) FROM skin_subs.{t[0]}"
|
|
||||||
).fetchone()[0]
|
|
||||||
print(f" skin_subs.{t[0]:30s}: {cnt:>6} rows")
|
print(f" skin_subs.{t[0]:30s}: {cnt:>6} rows")
|
||||||
|
|
||||||
ddb.close()
|
ddb.close()
|
||||||
|
|||||||
@@ -232,23 +232,25 @@ def main() -> None:
|
|||||||
if "type:fraud" in tag_set:
|
if "type:fraud" in tag_set:
|
||||||
domains.append("fraud")
|
domains.append("fraud")
|
||||||
|
|
||||||
chars.append({
|
chars.append(
|
||||||
"pmid": pmid,
|
{
|
||||||
"bib_key": row["key"],
|
"pmid": pmid,
|
||||||
"first_author": first_author,
|
"bib_key": row["key"],
|
||||||
"year": year,
|
"first_author": first_author,
|
||||||
"pub_type": pub_type,
|
"year": year,
|
||||||
"journal": journal,
|
"pub_type": pub_type,
|
||||||
"title": title[:200],
|
"journal": journal,
|
||||||
"products": "; ".join(products) if products else "",
|
"title": title[:200],
|
||||||
"product_count": len(products),
|
"products": "; ".join(products) if products else "",
|
||||||
"sample_size": sample_size,
|
"product_count": len(products),
|
||||||
"is_industry_linked": is_industry_linked,
|
"sample_size": sample_size,
|
||||||
"is_single_product": is_single_product,
|
"is_industry_linked": is_industry_linked,
|
||||||
"stance": stance,
|
"is_single_product": is_single_product,
|
||||||
"domains": "; ".join(domains),
|
"stance": stance,
|
||||||
"url": row["url"] or "",
|
"domains": "; ".join(domains),
|
||||||
})
|
"url": row["url"] or "",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
store.close()
|
store.close()
|
||||||
|
|
||||||
@@ -269,7 +271,7 @@ def main() -> None:
|
|||||||
# Top products
|
# Top products
|
||||||
product_counts: dict[str, int] = {}
|
product_counts: dict[str, int] = {}
|
||||||
for c in chars:
|
for c in chars:
|
||||||
for p in (c["products"].split("; ") if c["products"] else []):
|
for p in c["products"].split("; ") if c["products"] else []:
|
||||||
product_counts[p] = product_counts.get(p, 0) + 1
|
product_counts[p] = product_counts.get(p, 0) + 1
|
||||||
print("\n Top 15 products mentioned:")
|
print("\n Top 15 products mentioned:")
|
||||||
for p, ct in sorted(product_counts.items(), key=lambda x: -x[1])[:15]:
|
for p, ct in sorted(product_counts.items(), key=lambda x: -x[1])[:15]:
|
||||||
@@ -285,23 +287,25 @@ def main() -> None:
|
|||||||
# Register Python list as table
|
# Register Python list as table
|
||||||
import pyarrow as pa
|
import pyarrow as pa
|
||||||
|
|
||||||
schema = pa.schema([
|
schema = pa.schema(
|
||||||
("pmid", pa.string()),
|
[
|
||||||
("bib_key", pa.string()),
|
("pmid", pa.string()),
|
||||||
("first_author", pa.string()),
|
("bib_key", pa.string()),
|
||||||
("year", pa.string()),
|
("first_author", pa.string()),
|
||||||
("pub_type", pa.string()),
|
("year", pa.string()),
|
||||||
("journal", pa.string()),
|
("pub_type", pa.string()),
|
||||||
("title", pa.string()),
|
("journal", pa.string()),
|
||||||
("products", pa.string()),
|
("title", pa.string()),
|
||||||
("product_count", pa.int32()),
|
("products", pa.string()),
|
||||||
("sample_size", pa.int32()),
|
("product_count", pa.int32()),
|
||||||
("is_industry_linked", pa.bool_()),
|
("sample_size", pa.int32()),
|
||||||
("is_single_product", pa.bool_()),
|
("is_industry_linked", pa.bool_()),
|
||||||
("stance", pa.string()),
|
("is_single_product", pa.bool_()),
|
||||||
("domains", pa.string()),
|
("stance", pa.string()),
|
||||||
("url", pa.string()),
|
("domains", pa.string()),
|
||||||
])
|
("url", pa.string()),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
arrays = [
|
arrays = [
|
||||||
pa.array([c["pmid"] for c in chars]),
|
pa.array([c["pmid"] for c in chars]),
|
||||||
@@ -334,10 +338,22 @@ def main() -> None:
|
|||||||
# Quick validation queries
|
# Quick validation queries
|
||||||
print("\n Validation:")
|
print("\n Validation:")
|
||||||
for q, label in [
|
for q, label in [
|
||||||
("SELECT pub_type, count(*) c FROM skin_subs.study_characteristics GROUP BY 1 ORDER BY c DESC LIMIT 5", "pub_type"),
|
(
|
||||||
("SELECT count(*) FROM skin_subs.study_characteristics WHERE product_count > 0", "with_products"),
|
"SELECT pub_type, count(*) c FROM skin_subs.study_characteristics GROUP BY 1 ORDER BY c DESC LIMIT 5",
|
||||||
("SELECT count(*) FROM skin_subs.study_characteristics WHERE sample_size IS NOT NULL", "with_sample_size"),
|
"pub_type",
|
||||||
("SELECT count(*) FROM skin_subs.study_characteristics WHERE is_industry_linked", "industry_linked"),
|
),
|
||||||
|
(
|
||||||
|
"SELECT count(*) FROM skin_subs.study_characteristics WHERE product_count > 0",
|
||||||
|
"with_products",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"SELECT count(*) FROM skin_subs.study_characteristics WHERE sample_size IS NOT NULL",
|
||||||
|
"with_sample_size",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"SELECT count(*) FROM skin_subs.study_characteristics WHERE is_industry_linked",
|
||||||
|
"industry_linked",
|
||||||
|
),
|
||||||
]:
|
]:
|
||||||
result = ddb.execute(q).fetchall()
|
result = ddb.execute(q).fetchall()
|
||||||
print(f" {label}: {result}")
|
print(f" {label}: {result}")
|
||||||
|
|||||||
@@ -19,7 +19,6 @@ Usage:
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
import hashlib
|
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
import sqlite3
|
import sqlite3
|
||||||
@@ -47,8 +46,12 @@ def fetch_unpaywall(doi: str) -> str | None:
|
|||||||
"""Get PDF URL from Unpaywall for a DOI. Returns URL or None."""
|
"""Get PDF URL from Unpaywall for a DOI. Returns URL or None."""
|
||||||
url = f"https://api.unpaywall.org/v2/{doi}"
|
url = f"https://api.unpaywall.org/v2/{doi}"
|
||||||
try:
|
try:
|
||||||
resp = httpx.get(url, params={"email": UNPAYWALL_EMAIL},
|
resp = httpx.get(
|
||||||
headers={"User-Agent": USER_AGENT}, timeout=15)
|
url,
|
||||||
|
params={"email": UNPAYWALL_EMAIL},
|
||||||
|
headers={"User-Agent": USER_AGENT},
|
||||||
|
timeout=15,
|
||||||
|
)
|
||||||
if resp.status_code != 200:
|
if resp.status_code != 200:
|
||||||
return None
|
return None
|
||||||
data = resp.json()
|
data = resp.json()
|
||||||
@@ -68,7 +71,7 @@ def doi_to_pmcid(dois: list[str]) -> dict[str, str]:
|
|||||||
result = {}
|
result = {}
|
||||||
# API accepts up to 200 IDs per request
|
# API accepts up to 200 IDs per request
|
||||||
for i in range(0, len(dois), 200):
|
for i in range(0, len(dois), 200):
|
||||||
batch = dois[i:i + 200]
|
batch = dois[i : i + 200]
|
||||||
try:
|
try:
|
||||||
resp = httpx.get(
|
resp = httpx.get(
|
||||||
"https://www.ncbi.nlm.nih.gov/pmc/utils/idconv/v1.0/",
|
"https://www.ncbi.nlm.nih.gov/pmc/utils/idconv/v1.0/",
|
||||||
@@ -190,7 +193,9 @@ def fetch_scihub(doi: str) -> str | None:
|
|||||||
|
|
||||||
for mirror in SCIHUB_MIRRORS:
|
for mirror in SCIHUB_MIRRORS:
|
||||||
try:
|
try:
|
||||||
with httpx.Client(timeout=15, follow_redirects=True, **client_kwargs) as client:
|
with httpx.Client(
|
||||||
|
timeout=15, follow_redirects=True, **client_kwargs
|
||||||
|
) as client:
|
||||||
resp = client.get(
|
resp = client.get(
|
||||||
f"{mirror}/{doi}",
|
f"{mirror}/{doi}",
|
||||||
headers={"User-Agent": "Mozilla/5.0"},
|
headers={"User-Agent": "Mozilla/5.0"},
|
||||||
@@ -273,7 +278,12 @@ def attach_pdf_to_item(
|
|||||||
"""INSERT INTO itemAttachments
|
"""INSERT INTO itemAttachments
|
||||||
(itemID, parentItemID, linkMode, contentType, path, storageModTime)
|
(itemID, parentItemID, linkMode, contentType, path, storageModTime)
|
||||||
VALUES (?, ?, 0, 'application/pdf', ?, ?)""",
|
VALUES (?, ?, 0, 'application/pdf', ?, ?)""",
|
||||||
(att_item_id, item_id, f"storage:{key}/{pdf_path.name}", int(time.time() * 1000)),
|
(
|
||||||
|
att_item_id,
|
||||||
|
item_id,
|
||||||
|
f"storage:{key}/{pdf_path.name}",
|
||||||
|
int(time.time() * 1000),
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
# Move PDF to Zotero storage
|
# Move PDF to Zotero storage
|
||||||
@@ -282,6 +292,7 @@ def attach_pdf_to_item(
|
|||||||
dest = storage_dir / pdf_path.name
|
dest = storage_dir / pdf_path.name
|
||||||
if pdf_path != dest:
|
if pdf_path != dest:
|
||||||
import shutil
|
import shutil
|
||||||
|
|
||||||
shutil.copy2(pdf_path, dest)
|
shutil.copy2(pdf_path, dest)
|
||||||
|
|
||||||
con.commit()
|
con.commit()
|
||||||
@@ -295,7 +306,8 @@ def attach_pdf_to_item(
|
|||||||
|
|
||||||
def get_items_needing_pdfs(con: sqlite3.Connection) -> list[dict]:
|
def get_items_needing_pdfs(con: sqlite3.Connection) -> list[dict]:
|
||||||
"""Get skin-subs items with DOI but no PDF attachment."""
|
"""Get skin-subs items with DOI but no PDF attachment."""
|
||||||
rows = con.execute("""
|
rows = con.execute(
|
||||||
|
"""
|
||||||
SELECT DISTINCT i.itemID,
|
SELECT DISTINCT i.itemID,
|
||||||
(SELECT idv.value FROM itemData id
|
(SELECT idv.value FROM itemData id
|
||||||
JOIN itemDataValues idv ON id.valueID = idv.valueID
|
JOIN itemDataValues idv ON id.valueID = idv.valueID
|
||||||
@@ -310,17 +322,26 @@ def get_items_needing_pdfs(con: sqlite3.Connection) -> list[dict]:
|
|||||||
WHERE parentItemID IS NOT NULL
|
WHERE parentItemID IS NOT NULL
|
||||||
AND contentType = 'application/pdf'
|
AND contentType = 'application/pdf'
|
||||||
)
|
)
|
||||||
""", (FIELD_MAP["DOI"], TYPE_MAP["journalArticle"])).fetchall()
|
""",
|
||||||
return [{"item_id": r[0], "doi": r[1]}
|
(FIELD_MAP["DOI"], TYPE_MAP["journalArticle"]),
|
||||||
for r in rows if r[1]]
|
).fetchall()
|
||||||
|
return [{"item_id": r[0], "doi": r[1]} for r in rows if r[1]]
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
parser = argparse.ArgumentParser(description="Headless PDF retrieval")
|
parser = argparse.ArgumentParser(description="Headless PDF retrieval")
|
||||||
parser.add_argument("--limit", type=int, default=0, help="Max items to process (0=all)")
|
parser.add_argument(
|
||||||
parser.add_argument("--source", choices=["all", "unpaywall", "pmc", "s2", "worker", "scihub"],
|
"--limit", type=int, default=0, help="Max items to process (0=all)"
|
||||||
default="all", help="Which source to use")
|
)
|
||||||
parser.add_argument("--proxy", help="SOCKS/HTTP proxy for SciHub (e.g. socks5://localhost:1080)")
|
parser.add_argument(
|
||||||
|
"--source",
|
||||||
|
choices=["all", "unpaywall", "pmc", "s2", "worker", "scihub"],
|
||||||
|
default="all",
|
||||||
|
help="Which source to use",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--proxy", help="SOCKS/HTTP proxy for SciHub (e.g. socks5://localhost:1080)"
|
||||||
|
)
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
global SCIHUB_PROXY
|
global SCIHUB_PROXY
|
||||||
@@ -338,7 +359,7 @@ def main() -> None:
|
|||||||
|
|
||||||
items = get_items_needing_pdfs(con)
|
items = get_items_needing_pdfs(con)
|
||||||
if args.limit:
|
if args.limit:
|
||||||
items = items[:args.limit]
|
items = items[: args.limit]
|
||||||
print(f"\nItems needing PDFs: {len(items)}")
|
print(f"\nItems needing PDFs: {len(items)}")
|
||||||
|
|
||||||
tmp_dir = Path("/tmp/pdf_downloads")
|
tmp_dir = Path("/tmp/pdf_downloads")
|
||||||
@@ -427,8 +448,10 @@ def main() -> None:
|
|||||||
else:
|
else:
|
||||||
consecutive_failures += 1
|
consecutive_failures += 1
|
||||||
if (i + 1) % 25 == 0:
|
if (i + 1) % 25 == 0:
|
||||||
print(f" {i + 1}/{len(remaining)} worker={stats.get('worker', 0)} "
|
print(
|
||||||
f"fails={consecutive_failures}")
|
f" {i + 1}/{len(remaining)} worker={stats.get('worker', 0)} "
|
||||||
|
f"fails={consecutive_failures}"
|
||||||
|
)
|
||||||
time.sleep(1.5) # Be polite to Worker + SciHub
|
time.sleep(1.5) # Be polite to Worker + SciHub
|
||||||
print(f" CF Worker: {stats.get('worker', 0)} PDFs")
|
print(f" CF Worker: {stats.get('worker', 0)} PDFs")
|
||||||
|
|
||||||
@@ -442,7 +465,9 @@ def main() -> None:
|
|||||||
consecutive_failures = 0
|
consecutive_failures = 0
|
||||||
for i, item in enumerate(remaining):
|
for i, item in enumerate(remaining):
|
||||||
if consecutive_failures >= 10:
|
if consecutive_failures >= 10:
|
||||||
print(f" Stopping: {consecutive_failures} consecutive failures (CAPTCHA?)")
|
print(
|
||||||
|
f" Stopping: {consecutive_failures} consecutive failures (CAPTCHA?)"
|
||||||
|
)
|
||||||
break
|
break
|
||||||
|
|
||||||
pdf_url = fetch_scihub(item["doi"])
|
pdf_url = fetch_scihub(item["doi"])
|
||||||
@@ -459,15 +484,22 @@ def main() -> None:
|
|||||||
consecutive_failures += 1
|
consecutive_failures += 1
|
||||||
|
|
||||||
if (i + 1) % 25 == 0:
|
if (i + 1) % 25 == 0:
|
||||||
print(f" {i + 1}/{len(remaining)} scihub={stats['scihub']} "
|
print(
|
||||||
f"fails={consecutive_failures}")
|
f" {i + 1}/{len(remaining)} scihub={stats['scihub']} "
|
||||||
|
f"fails={consecutive_failures}"
|
||||||
|
)
|
||||||
time.sleep(1) # Be polite to SciHub
|
time.sleep(1) # Be polite to SciHub
|
||||||
|
|
||||||
print(f" SciHub: {stats['scihub']} PDFs")
|
print(f" SciHub: {stats['scihub']} PDFs")
|
||||||
|
|
||||||
stats["failed"] = len([it for it in items if not it.get("_done")])
|
stats["failed"] = len([it for it in items if not it.get("_done")])
|
||||||
total_found = (stats["unpaywall"] + stats["pmc"] + stats.get("s2", 0)
|
total_found = (
|
||||||
+ stats.get("worker", 0) + stats.get("scihub", 0))
|
stats["unpaywall"]
|
||||||
|
+ stats["pmc"]
|
||||||
|
+ stats.get("s2", 0)
|
||||||
|
+ stats.get("worker", 0)
|
||||||
|
+ stats.get("scihub", 0)
|
||||||
|
)
|
||||||
|
|
||||||
print(f"\n{'=' * 60}")
|
print(f"\n{'=' * 60}")
|
||||||
print(f"Results: {total_found} PDFs downloaded")
|
print(f"Results: {total_found} PDFs downloaded")
|
||||||
|
|||||||
@@ -1,79 +1,22 @@
|
|||||||
"""Fix and prevent non-ISO 8601 dates in Zotero's SQLite database.
|
"""Thin shim — logic moved to :mod:`zot.ops`.
|
||||||
|
|
||||||
Zotero internally writes timestamps as 'YYYY-MM-DD HH:MM:SS' (SQL format)
|
Prefer: ``uv run stack zot fix-dates``.
|
||||||
but its sync engine requires 'YYYY-MM-DDTHH:MM:SSZ' (ISO 8601).
|
|
||||||
This script:
|
|
||||||
|
|
||||||
1. Converts all existing space-separated timestamps to ISO 8601
|
|
||||||
2. Installs SQLite triggers that auto-convert on every INSERT/UPDATE
|
|
||||||
|
|
||||||
Safe to run multiple times (triggers use CREATE IF NOT EXISTS).
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
uv run python dev/scripts/fix_zotero_dates.py
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import sqlite3
|
|
||||||
import sys
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
DB_PATH = "zotero/data/zotero.sqlite"
|
|
||||||
COLUMNS = ("dateAdded", "dateModified", "clientDateModified")
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
def main() -> int:
|
||||||
db = sqlite3.connect(DB_PATH, timeout=10)
|
from zot.ops import fix_dates
|
||||||
db.execute("PRAGMA wal_checkpoint(TRUNCATE)")
|
|
||||||
|
|
||||||
# Fix existing bad dates
|
db = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("zotero/data/zotero.sqlite")
|
||||||
for col in COLUMNS:
|
out = fix_dates(db)
|
||||||
fixed = db.execute(
|
for k, v in out.items():
|
||||||
f"UPDATE items SET {col} = REPLACE({col}, ' ', 'T') || 'Z'"
|
print(f"{k}: {v}")
|
||||||
f" WHERE {col} LIKE '____-__-__ __:__:__'"
|
return 0 if out.get("remaining", 0) == 0 else 1
|
||||||
f" AND {col} NOT LIKE '%T%'",
|
|
||||||
).rowcount
|
|
||||||
if fixed:
|
|
||||||
print(f"Fixed {col}: {fixed}")
|
|
||||||
|
|
||||||
# Install auto-fix triggers
|
|
||||||
for col in COLUMNS:
|
|
||||||
for op in ("INSERT", "UPDATE"):
|
|
||||||
trigger = f"fix_{col}_{op.lower()}"
|
|
||||||
db.execute(f"""
|
|
||||||
CREATE TRIGGER IF NOT EXISTS {trigger}
|
|
||||||
AFTER {op} ON items
|
|
||||||
FOR EACH ROW
|
|
||||||
WHEN NEW.{col} LIKE '____-__-__ __:__:__'
|
|
||||||
AND NEW.{col} NOT LIKE '%T%'
|
|
||||||
BEGIN
|
|
||||||
UPDATE items SET {col} = REPLACE(NEW.{col}, ' ', 'T') || 'Z'
|
|
||||||
WHERE itemID = NEW.itemID;
|
|
||||||
END
|
|
||||||
""")
|
|
||||||
|
|
||||||
db.commit()
|
|
||||||
|
|
||||||
# Verify
|
|
||||||
triggers = [
|
|
||||||
r[0]
|
|
||||||
for r in db.execute(
|
|
||||||
"SELECT name FROM sqlite_master WHERE type='trigger' AND name LIKE 'fix_%'"
|
|
||||||
)
|
|
||||||
]
|
|
||||||
print(f"Triggers installed: {len(triggers)}")
|
|
||||||
for col in COLUMNS:
|
|
||||||
bad = db.execute(
|
|
||||||
f"SELECT COUNT(*) FROM items WHERE {col} NOT LIKE '%T%'"
|
|
||||||
).fetchone()[0]
|
|
||||||
if bad:
|
|
||||||
print(f"WARNING: {col} still has {bad} bad values")
|
|
||||||
db.close()
|
|
||||||
return 1
|
|
||||||
|
|
||||||
db.close()
|
|
||||||
print("All dates valid.")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -1,136 +1,22 @@
|
|||||||
"""Fix invalid Zotero object keys.
|
"""Thin shim — logic moved to :mod:`zot.ops`.
|
||||||
|
|
||||||
Zotero 8 requires keys to be exactly 8 characters from
|
Prefer: ``uv run stack zot fix-keys``.
|
||||||
[23456789ABCDEFGHIJKLMNPQRSTUVWXYZ]. Bulk-imported items may have
|
|
||||||
keys containing 0, 1, or O which cause "key is not valid" errors
|
|
||||||
when Zotero tries to create annotations or other child items.
|
|
||||||
|
|
||||||
This script:
|
|
||||||
1. Finds all items and collections with invalid keys
|
|
||||||
2. Generates new valid keys (no collisions)
|
|
||||||
3. Updates the items/collections tables
|
|
||||||
4. Renames storage folders to match new keys
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
# Stop Zotero first, then:
|
|
||||||
uv run python dev/scripts/fix_zotero_keys.py
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import os
|
|
||||||
import random
|
|
||||||
import re
|
|
||||||
import shutil
|
|
||||||
import sqlite3
|
|
||||||
import sys
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
ALLOWED = "23456789ABCDEFGHIJKLMNPQRSTUVWXYZ"
|
|
||||||
KEY_RE = re.compile(rf"^[{ALLOWED}]{{8}}$")
|
|
||||||
DB_PATH = "zotero/data/zotero.sqlite"
|
|
||||||
STORAGE_DIR = "zotero/data/storage"
|
|
||||||
|
|
||||||
|
|
||||||
def generate_key(existing: set[str]) -> str:
|
|
||||||
"""Generate a valid 8-char key not in *existing*."""
|
|
||||||
while True:
|
|
||||||
key = "".join(random.choices(ALLOWED, k=8))
|
|
||||||
if key not in existing:
|
|
||||||
existing.add(key)
|
|
||||||
return key
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
def main() -> int:
|
||||||
if not os.path.exists(DB_PATH):
|
from zot.ops import fix_keys
|
||||||
print(f"ERROR: {DB_PATH} not found", file=sys.stderr)
|
|
||||||
return 1
|
|
||||||
|
|
||||||
# Backup
|
db = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("zotero/data/zotero.sqlite")
|
||||||
backup = DB_PATH + ".pre-keyfix.bak"
|
out = fix_keys(db)
|
||||||
if not os.path.exists(backup):
|
for k, v in out.items():
|
||||||
shutil.copy2(DB_PATH, backup)
|
print(f"{k}: {v}")
|
||||||
print(f"Backed up to {backup}")
|
return 0 if out.get("remaining", 0) == 0 else 1
|
||||||
|
|
||||||
db = sqlite3.connect(DB_PATH, timeout=10)
|
|
||||||
|
|
||||||
# Checkpoint WAL first
|
|
||||||
db.execute("PRAGMA wal_checkpoint(TRUNCATE)")
|
|
||||||
|
|
||||||
# Collect all existing keys
|
|
||||||
existing_keys: set[str] = set()
|
|
||||||
for (k,) in db.execute("SELECT key FROM items"):
|
|
||||||
existing_keys.add(k)
|
|
||||||
for (k,) in db.execute("SELECT key FROM collections"):
|
|
||||||
existing_keys.add(k)
|
|
||||||
|
|
||||||
# Find invalid item keys
|
|
||||||
invalid_items: list[tuple[int, str]] = []
|
|
||||||
for row in db.execute("SELECT itemID, key FROM items"):
|
|
||||||
if not KEY_RE.match(row[1]):
|
|
||||||
invalid_items.append(row)
|
|
||||||
|
|
||||||
# Find invalid collection keys
|
|
||||||
invalid_colls: list[tuple[int, str]] = []
|
|
||||||
for row in db.execute("SELECT collectionID, key FROM collections"):
|
|
||||||
if not KEY_RE.match(row[1]):
|
|
||||||
invalid_colls.append(row)
|
|
||||||
|
|
||||||
print(f"Invalid item keys: {len(invalid_items)}")
|
|
||||||
print(f"Invalid collection keys: {len(invalid_colls)}")
|
|
||||||
|
|
||||||
if not invalid_items and not invalid_colls:
|
|
||||||
print("Nothing to fix!")
|
|
||||||
db.close()
|
|
||||||
return 0
|
|
||||||
|
|
||||||
# Fix items
|
|
||||||
renames: list[tuple[str, str]] = [] # (old_key, new_key) for storage
|
|
||||||
for item_id, old_key in invalid_items:
|
|
||||||
new_key = generate_key(existing_keys)
|
|
||||||
db.execute("UPDATE items SET key = ? WHERE itemID = ?", (new_key, item_id))
|
|
||||||
old_dir = os.path.join(STORAGE_DIR, old_key)
|
|
||||||
if os.path.isdir(old_dir):
|
|
||||||
renames.append((old_key, new_key))
|
|
||||||
|
|
||||||
# Fix collections
|
|
||||||
for coll_id, old_key in invalid_colls:
|
|
||||||
new_key = generate_key(existing_keys)
|
|
||||||
db.execute(
|
|
||||||
"UPDATE collections SET key = ? WHERE collectionID = ?",
|
|
||||||
(new_key, coll_id),
|
|
||||||
)
|
|
||||||
|
|
||||||
db.commit()
|
|
||||||
db.close()
|
|
||||||
print(f"Updated {len(invalid_items)} items + {len(invalid_colls)} collections")
|
|
||||||
|
|
||||||
# Rename storage folders
|
|
||||||
renamed = 0
|
|
||||||
for old_key, new_key in renames:
|
|
||||||
old_dir = os.path.join(STORAGE_DIR, old_key)
|
|
||||||
new_dir = os.path.join(STORAGE_DIR, new_key)
|
|
||||||
if os.path.isdir(old_dir) and not os.path.exists(new_dir):
|
|
||||||
os.rename(old_dir, new_dir)
|
|
||||||
renamed += 1
|
|
||||||
print(f"Renamed {renamed} storage folders")
|
|
||||||
|
|
||||||
# Verify
|
|
||||||
db = sqlite3.connect(DB_PATH, timeout=10)
|
|
||||||
remaining = 0
|
|
||||||
for (k,) in db.execute("SELECT key FROM items"):
|
|
||||||
if not KEY_RE.match(k):
|
|
||||||
remaining += 1
|
|
||||||
for (k,) in db.execute("SELECT key FROM collections"):
|
|
||||||
if not KEY_RE.match(k):
|
|
||||||
remaining += 1
|
|
||||||
db.close()
|
|
||||||
|
|
||||||
if remaining:
|
|
||||||
print(f"WARNING: {remaining} invalid keys remaining!")
|
|
||||||
return 1
|
|
||||||
|
|
||||||
print("All keys valid. Restart Zotero to apply.")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -291,7 +291,9 @@ def main() -> None:
|
|||||||
& ~all_df["hcpcs_code"].isin(skin_codes)
|
& ~all_df["hcpcs_code"].isin(skin_codes)
|
||||||
]["hcpcs_code"].unique()
|
]["hcpcs_code"].unique()
|
||||||
if len(q4_but_not_skin) > 0:
|
if len(q4_but_not_skin) > 0:
|
||||||
print(f" Excluded Q4 codes NOT in skin sub universe: {sorted(q4_but_not_skin)}")
|
print(
|
||||||
|
f" Excluded Q4 codes NOT in skin sub universe: {sorted(q4_but_not_skin)}"
|
||||||
|
)
|
||||||
|
|
||||||
if not skin_df.empty:
|
if not skin_df.empty:
|
||||||
skin_df["payment_limit"] = clean_numeric(skin_df["payment_limit"])
|
skin_df["payment_limit"] = clean_numeric(skin_df["payment_limit"])
|
||||||
@@ -338,13 +340,15 @@ def main() -> None:
|
|||||||
""",
|
""",
|
||||||
[str(OUTPUT_CSV)],
|
[str(OUTPUT_CSV)],
|
||||||
)
|
)
|
||||||
count = con.execute(
|
count = con.execute("SELECT count(*) FROM skin_subs.asp_quarterly").fetchone()[
|
||||||
"SELECT count(*) FROM skin_subs.asp_quarterly"
|
0
|
||||||
).fetchone()[0]
|
]
|
||||||
print(f" Loaded {count} rows into skin_subs.asp_quarterly")
|
print(f" Loaded {count} rows into skin_subs.asp_quarterly")
|
||||||
else:
|
else:
|
||||||
print(" WARNING: No skin substitute data found in ASP files")
|
print(" WARNING: No skin substitute data found in ASP files")
|
||||||
print(" This may be expected — skin subs may not appear in main ASP pricing files")
|
print(
|
||||||
|
" This may be expected — skin subs may not appear in main ASP pricing files"
|
||||||
|
)
|
||||||
print(" They may be in separate NOC or tissue coding files")
|
print(" They may be in separate NOC or tissue coding files")
|
||||||
con.close()
|
con.close()
|
||||||
|
|
||||||
|
|||||||
@@ -19,7 +19,6 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
import io
|
import io
|
||||||
import re
|
|
||||||
import zipfile
|
import zipfile
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
@@ -133,14 +132,17 @@ def _find_header_row(df: pd.DataFrame, col_map: dict[str, str]) -> int:
|
|||||||
vals = [str(v).strip().lower() for v in row if pd.notna(v) and str(v).strip()]
|
vals = [str(v).strip().lower() for v in row if pd.notna(v) and str(v).strip()]
|
||||||
if len(vals) < 3:
|
if len(vals) < 3:
|
||||||
continue
|
continue
|
||||||
matches = sum(1 for v in vals for k in target_cols if v == k or (len(k) > 3 and k in v))
|
matches = sum(
|
||||||
|
1 for v in vals for k in target_cols if v == k or (len(k) > 3 and k in v)
|
||||||
|
)
|
||||||
if matches >= 3:
|
if matches >= 3:
|
||||||
return i
|
return i
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def read_from_zip(zpath: Path, col_map: dict[str, str],
|
def read_from_zip(
|
||||||
pattern: str = "") -> pd.DataFrame:
|
zpath: Path, col_map: dict[str, str], pattern: str = ""
|
||||||
|
) -> pd.DataFrame:
|
||||||
"""Read and normalise a data file from inside a ZIP.
|
"""Read and normalise a data file from inside a ZIP.
|
||||||
|
|
||||||
Handles CMS preamble rows in both CSV and Excel formats.
|
Handles CMS preamble rows in both CSV and Excel formats.
|
||||||
@@ -168,7 +170,9 @@ def read_from_zip(zpath: Path, col_map: dict[str, str],
|
|||||||
df = pd.read_csv(io.StringIO(buf), dtype=str, on_bad_lines="skip")
|
df = pd.read_csv(io.StringIO(buf), dtype=str, on_bad_lines="skip")
|
||||||
else:
|
else:
|
||||||
# Excel: read without header to find preamble extent
|
# Excel: read without header to find preamble extent
|
||||||
df_raw = pd.read_excel(io.BytesIO(data), dtype=str, header=None, nrows=20)
|
df_raw = pd.read_excel(
|
||||||
|
io.BytesIO(data), dtype=str, header=None, nrows=20
|
||||||
|
)
|
||||||
hdr_row = _find_header_row(df_raw, col_map)
|
hdr_row = _find_header_row(df_raw, col_map)
|
||||||
df = pd.read_excel(io.BytesIO(data), dtype=str, header=hdr_row)
|
df = pd.read_excel(io.BytesIO(data), dtype=str, header=hdr_row)
|
||||||
|
|
||||||
@@ -224,7 +228,11 @@ def ingest_all(con: duckdb.DuckDBPyConnection, year_filter: str = "") -> None:
|
|||||||
df = read_from_zip(addenda_zip, ADDENDUM_A_COL_MAP, pattern="addendum a")
|
df = read_from_zip(addenda_zip, ADDENDUM_A_COL_MAP, pattern="addendum a")
|
||||||
if not df.empty:
|
if not df.empty:
|
||||||
df["year"] = int(year)
|
df["year"] = int(year)
|
||||||
for col in ["relative_weight", "payment_rate", "minimum_unadjusted_copayment"]:
|
for col in [
|
||||||
|
"relative_weight",
|
||||||
|
"payment_rate",
|
||||||
|
"minimum_unadjusted_copayment",
|
||||||
|
]:
|
||||||
if col in df.columns:
|
if col in df.columns:
|
||||||
df[col] = clean_numeric(df[col])
|
df[col] = clean_numeric(df[col])
|
||||||
apc_frames.append(df)
|
apc_frames.append(df)
|
||||||
@@ -234,7 +242,11 @@ def ingest_all(con: duckdb.DuckDBPyConnection, year_filter: str = "") -> None:
|
|||||||
df = read_from_zip(addenda_zip, ADDENDUM_B_COL_MAP, pattern="addendum b")
|
df = read_from_zip(addenda_zip, ADDENDUM_B_COL_MAP, pattern="addendum b")
|
||||||
if not df.empty:
|
if not df.empty:
|
||||||
df["year"] = int(year)
|
df["year"] = int(year)
|
||||||
for col in ["relative_weight", "payment_rate", "minimum_unadjusted_copayment"]:
|
for col in [
|
||||||
|
"relative_weight",
|
||||||
|
"payment_rate",
|
||||||
|
"minimum_unadjusted_copayment",
|
||||||
|
]:
|
||||||
if col in df.columns:
|
if col in df.columns:
|
||||||
df[col] = clean_numeric(df[col])
|
df[col] = clean_numeric(df[col])
|
||||||
addb_frames.append(df)
|
addb_frames.append(df)
|
||||||
@@ -247,22 +259,28 @@ def ingest_all(con: duckdb.DuckDBPyConnection, year_filter: str = "") -> None:
|
|||||||
all_apc = pd.concat(apc_frames, ignore_index=True)
|
all_apc = pd.concat(apc_frames, ignore_index=True)
|
||||||
con.execute("DROP TABLE IF EXISTS opps.apc_weight")
|
con.execute("DROP TABLE IF EXISTS opps.apc_weight")
|
||||||
con.execute("CREATE TABLE opps.apc_weight AS SELECT * FROM all_apc")
|
con.execute("CREATE TABLE opps.apc_weight AS SELECT * FROM all_apc")
|
||||||
print(f" opps.apc_weight: {len(all_apc)} rows "
|
print(
|
||||||
f"({all_apc['year'].nunique()} years)")
|
f" opps.apc_weight: {len(all_apc)} rows "
|
||||||
|
f"({all_apc['year'].nunique()} years)"
|
||||||
|
)
|
||||||
|
|
||||||
if addb_frames:
|
if addb_frames:
|
||||||
all_addb = pd.concat(addb_frames, ignore_index=True)
|
all_addb = pd.concat(addb_frames, ignore_index=True)
|
||||||
con.execute("DROP TABLE IF EXISTS opps.addendum_b")
|
con.execute("DROP TABLE IF EXISTS opps.addendum_b")
|
||||||
con.execute("CREATE TABLE opps.addendum_b AS SELECT * FROM all_addb")
|
con.execute("CREATE TABLE opps.addendum_b AS SELECT * FROM all_addb")
|
||||||
print(f" opps.addendum_b: {len(all_addb)} rows "
|
print(
|
||||||
f"({all_addb['year'].nunique()} years)")
|
f" opps.addendum_b: {len(all_addb)} rows "
|
||||||
|
f"({all_addb['year'].nunique()} years)"
|
||||||
|
)
|
||||||
|
|
||||||
if wage_frames:
|
if wage_frames:
|
||||||
all_wage = pd.concat(wage_frames, ignore_index=True)
|
all_wage = pd.concat(wage_frames, ignore_index=True)
|
||||||
con.execute("DROP TABLE IF EXISTS opps.wage_index")
|
con.execute("DROP TABLE IF EXISTS opps.wage_index")
|
||||||
con.execute("CREATE TABLE opps.wage_index AS SELECT * FROM all_wage")
|
con.execute("CREATE TABLE opps.wage_index AS SELECT * FROM all_wage")
|
||||||
print(f" opps.wage_index: {len(all_wage)} rows "
|
print(
|
||||||
f"({all_wage['year'].nunique()} years)")
|
f" opps.wage_index: {len(all_wage)} rows "
|
||||||
|
f"({all_wage['year'].nunique()} years)"
|
||||||
|
)
|
||||||
|
|
||||||
# Skin sub specific: extract skin sub codes from Addendum B history
|
# Skin sub specific: extract skin sub codes from Addendum B history
|
||||||
if addb_frames:
|
if addb_frames:
|
||||||
@@ -270,14 +288,20 @@ def ingest_all(con: duckdb.DuckDBPyConnection, year_filter: str = "") -> None:
|
|||||||
all_addb = pd.concat(addb_frames, ignore_index=True)
|
all_addb = pd.concat(addb_frames, ignore_index=True)
|
||||||
if "hcpcs" in all_addb.columns:
|
if "hcpcs" in all_addb.columns:
|
||||||
skin_codes = all_addb[
|
skin_codes = all_addb[
|
||||||
all_addb["hcpcs"].astype(str).str.match(r"^Q4\d{2,3}$|^C527[1-8]$", na=False)
|
all_addb["hcpcs"]
|
||||||
|
.astype(str)
|
||||||
|
.str.match(r"^Q4\d{2,3}$|^C527[1-8]$", na=False)
|
||||||
]
|
]
|
||||||
if not skin_codes.empty:
|
if not skin_codes.empty:
|
||||||
con.execute("DROP TABLE IF EXISTS opps.skin_sub_addendum_b")
|
con.execute("DROP TABLE IF EXISTS opps.skin_sub_addendum_b")
|
||||||
con.execute("CREATE TABLE opps.skin_sub_addendum_b AS SELECT * FROM skin_codes")
|
con.execute(
|
||||||
print(f" opps.skin_sub_addendum_b: {len(skin_codes)} rows "
|
"CREATE TABLE opps.skin_sub_addendum_b AS SELECT * FROM skin_codes"
|
||||||
f"({skin_codes['year'].nunique()} years, "
|
)
|
||||||
f"{skin_codes['hcpcs'].nunique()} unique codes)")
|
print(
|
||||||
|
f" opps.skin_sub_addendum_b: {len(skin_codes)} rows "
|
||||||
|
f"({skin_codes['year'].nunique()} years, "
|
||||||
|
f"{skin_codes['hcpcs'].nunique()} unique codes)"
|
||||||
|
)
|
||||||
|
|
||||||
# SI distribution for skin subs
|
# SI distribution for skin subs
|
||||||
if "status_indicator" in skin_codes.columns:
|
if "status_indicator" in skin_codes.columns:
|
||||||
|
|||||||
@@ -14,10 +14,7 @@ Usage::
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
|
||||||
import shutil
|
import shutil
|
||||||
import sqlite3
|
|
||||||
|
|
||||||
import subprocess
|
import subprocess
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -294,10 +291,6 @@ SEED_REGISTRY: tuple[SeedSpec, ...] = (
|
|||||||
# ── Helpers ───────────────────────────────────────────────────────
|
# ── Helpers ───────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def _content_type(path: Path) -> str:
|
def _content_type(path: Path) -> str:
|
||||||
ext = path.suffix.lower()
|
ext = path.suffix.lower()
|
||||||
types = {
|
types = {
|
||||||
@@ -317,9 +310,7 @@ def _content_type(path: Path) -> str:
|
|||||||
# ── Phase A: tag existing Zotero attachments ──────────────────────
|
# ── Phase A: tag existing Zotero attachments ──────────────────────
|
||||||
|
|
||||||
|
|
||||||
def tag_existing_zotero(
|
def tag_existing_zotero(db: Db, spec: SeedSpec, *, dry_run: bool = False) -> bool:
|
||||||
db: Db, spec: SeedSpec, *, dry_run: bool = False
|
|
||||||
) -> bool:
|
|
||||||
"""Find a Zotero attachment by filename and tag its parent."""
|
"""Find a Zotero attachment by filename and tag its parent."""
|
||||||
filename = Path(spec.path).name
|
filename = Path(spec.path).name
|
||||||
row = db.con.execute(
|
row = db.con.execute(
|
||||||
@@ -344,9 +335,7 @@ def tag_existing_zotero(
|
|||||||
# ── Phase B: create missing Zotero items ──────────────────────────
|
# ── Phase B: create missing Zotero items ──────────────────────────
|
||||||
|
|
||||||
|
|
||||||
def register_in_zotero(
|
def register_in_zotero(db: Db, spec: SeedSpec, *, dry_run: bool = False) -> None:
|
||||||
db: Db, spec: SeedSpec, *, dry_run: bool = False
|
|
||||||
) -> None:
|
|
||||||
"""Create a Zotero webpage item + file attachment for a seed."""
|
"""Create a Zotero webpage item + file attachment for a seed."""
|
||||||
if not spec.url:
|
if not spec.url:
|
||||||
if dry_run:
|
if dry_run:
|
||||||
@@ -361,14 +350,17 @@ def register_in_zotero(
|
|||||||
|
|
||||||
# Create parent webpage item
|
# Create parent webpage item
|
||||||
parent_id = db.create_item(TYPE_MAP["webpage"], now=now)
|
parent_id = db.create_item(TYPE_MAP["webpage"], now=now)
|
||||||
db.set_fields(parent_id, {
|
db.set_fields(
|
||||||
"title": spec.title,
|
parent_id,
|
||||||
"url": spec.url,
|
{
|
||||||
"date": now[:10],
|
"title": spec.title,
|
||||||
"accessDate": now,
|
"url": spec.url,
|
||||||
"websiteType": "Government Data Portal",
|
"date": now[:10],
|
||||||
"websiteTitle": "Centers for Medicare & Medicaid Services",
|
"accessDate": now,
|
||||||
})
|
"websiteType": "Government Data Portal",
|
||||||
|
"websiteTitle": "Centers for Medicare & Medicaid Services",
|
||||||
|
},
|
||||||
|
)
|
||||||
db.sync_tags(parent_id, [SEED_TAG] + spec.tags)
|
db.sync_tags(parent_id, [SEED_TAG] + spec.tags)
|
||||||
|
|
||||||
# Attach file (skip directories)
|
# Attach file (skip directories)
|
||||||
|
|||||||
@@ -185,15 +185,21 @@ def efetch_articles(
|
|||||||
except (httpx.RemoteProtocolError, httpx.ReadTimeout) as exc:
|
except (httpx.RemoteProtocolError, httpx.ReadTimeout) as exc:
|
||||||
if attempt < max_retries - 1:
|
if attempt < max_retries - 1:
|
||||||
wait = 2 ** (attempt + 1)
|
wait = 2 ** (attempt + 1)
|
||||||
print(f" RETRY batch {i // batch_size + 1} "
|
print(
|
||||||
f"(attempt {attempt + 2}/{max_retries}, "
|
f" RETRY batch {i // batch_size + 1} "
|
||||||
f"wait {wait}s): {exc}")
|
f"(attempt {attempt + 2}/{max_retries}, "
|
||||||
|
f"wait {wait}s): {exc}"
|
||||||
|
)
|
||||||
time.sleep(wait)
|
time.sleep(wait)
|
||||||
else:
|
else:
|
||||||
print(f" SKIP batch {i // batch_size + 1} after "
|
print(
|
||||||
f"{max_retries} attempts: {exc}")
|
f" SKIP batch {i // batch_size + 1} after "
|
||||||
print(f" efetch: batch {i // batch_size + 1}/{len(pmids) // batch_size + 1}, "
|
f"{max_retries} attempts: {exc}"
|
||||||
f"got {len(articles)} articles so far")
|
)
|
||||||
|
print(
|
||||||
|
f" efetch: batch {i // batch_size + 1}/{len(pmids) // batch_size + 1}, "
|
||||||
|
f"got {len(articles)} articles so far"
|
||||||
|
)
|
||||||
return articles
|
return articles
|
||||||
|
|
||||||
|
|
||||||
@@ -379,8 +385,9 @@ def article_to_source(article: Article, domain_tags: list[str]) -> Source:
|
|||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
parser = argparse.ArgumentParser(description="PubMed skin substitutes search")
|
parser = argparse.ArgumentParser(description="PubMed skin substitutes search")
|
||||||
parser.add_argument("--dry-run", action="store_true",
|
parser.add_argument(
|
||||||
help="Search only, don't write to bib.sqlite")
|
"--dry-run", action="store_true", help="Search only, don't write to bib.sqlite"
|
||||||
|
)
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
print("=" * 70)
|
print("=" * 70)
|
||||||
|
|||||||
@@ -63,9 +63,7 @@ def elink_cited_by(pmids: list[str], batch_size: int = 50) -> dict[str, list[str
|
|||||||
)
|
)
|
||||||
time.sleep(RATE_LIMIT)
|
time.sleep(RATE_LIMIT)
|
||||||
try:
|
try:
|
||||||
resp = httpx.get(
|
resp = httpx.get(f"{EUTILS_BASE}/elink.fcgi", params=params, timeout=60)
|
||||||
f"{EUTILS_BASE}/elink.fcgi", params=params, timeout=60
|
|
||||||
)
|
|
||||||
resp.raise_for_status()
|
resp.raise_for_status()
|
||||||
root = ET.fromstring(resp.text) # noqa: S314
|
root = ET.fromstring(resp.text) # noqa: S314
|
||||||
for linkset in root.findall(".//LinkSet"):
|
for linkset in root.findall(".//LinkSet"):
|
||||||
@@ -111,13 +109,18 @@ def efetch_basic(pmids: list[str], batch_size: int = 100) -> list[dict]:
|
|||||||
for i in range(0, len(pmids), batch_size):
|
for i in range(0, len(pmids), batch_size):
|
||||||
batch = pmids[i : i + batch_size]
|
batch = pmids[i : i + batch_size]
|
||||||
params = _params(
|
params = _params(
|
||||||
db="pubmed", id=",".join(batch), rettype="xml", retmode="xml",
|
db="pubmed",
|
||||||
|
id=",".join(batch),
|
||||||
|
rettype="xml",
|
||||||
|
retmode="xml",
|
||||||
)
|
)
|
||||||
for attempt in range(3):
|
for attempt in range(3):
|
||||||
time.sleep(RATE_LIMIT * (attempt + 1))
|
time.sleep(RATE_LIMIT * (attempt + 1))
|
||||||
try:
|
try:
|
||||||
resp = httpx.get(
|
resp = httpx.get(
|
||||||
f"{EUTILS_BASE}/efetch.fcgi", params=params, timeout=120,
|
f"{EUTILS_BASE}/efetch.fcgi",
|
||||||
|
params=params,
|
||||||
|
timeout=120,
|
||||||
)
|
)
|
||||||
resp.raise_for_status()
|
resp.raise_for_status()
|
||||||
root = ET.fromstring(resp.text) # noqa: S314
|
root = ET.fromstring(resp.text) # noqa: S314
|
||||||
@@ -155,7 +158,9 @@ def efetch_basic(pmids: list[str], batch_size: int = 100) -> list[dict]:
|
|||||||
|
|
||||||
# Journal + year
|
# Journal + year
|
||||||
journal_el = article_el.find("Journal")
|
journal_el = article_el.find("Journal")
|
||||||
journal = _text(journal_el, "Title") if journal_el is not None else ""
|
journal = (
|
||||||
|
_text(journal_el, "Title") if journal_el is not None else ""
|
||||||
|
)
|
||||||
year = ""
|
year = ""
|
||||||
pub_date = article_el.find(".//PubDate")
|
pub_date = article_el.find(".//PubDate")
|
||||||
if pub_date is not None:
|
if pub_date is not None:
|
||||||
@@ -178,16 +183,18 @@ def efetch_basic(pmids: list[str], batch_size: int = 100) -> list[dict]:
|
|||||||
if pt.text:
|
if pt.text:
|
||||||
pub_types.append(pt.text)
|
pub_types.append(pt.text)
|
||||||
|
|
||||||
articles.append({
|
articles.append(
|
||||||
"pmid": pmid,
|
{
|
||||||
"title": title,
|
"pmid": pmid,
|
||||||
"abstract": "\n\n".join(abstract_parts),
|
"title": title,
|
||||||
"authors": authors,
|
"abstract": "\n\n".join(abstract_parts),
|
||||||
"journal": journal,
|
"authors": authors,
|
||||||
"year": year,
|
"journal": journal,
|
||||||
"doi": doi,
|
"year": year,
|
||||||
"pub_types": pub_types,
|
"doi": doi,
|
||||||
})
|
"pub_types": pub_types,
|
||||||
|
}
|
||||||
|
)
|
||||||
break
|
break
|
||||||
except (httpx.RemoteProtocolError, httpx.ReadTimeout) as exc:
|
except (httpx.RemoteProtocolError, httpx.ReadTimeout) as exc:
|
||||||
if attempt < 2:
|
if attempt < 2:
|
||||||
@@ -209,8 +216,12 @@ def efetch_basic(pmids: list[str], batch_size: int = 100) -> list[dict]:
|
|||||||
def main() -> None:
|
def main() -> None:
|
||||||
parser = argparse.ArgumentParser(description="Snowball citation chasing")
|
parser = argparse.ArgumentParser(description="Snowball citation chasing")
|
||||||
parser.add_argument("--dry-run", action="store_true")
|
parser.add_argument("--dry-run", action="store_true")
|
||||||
parser.add_argument("--seed-limit", type=int, default=200,
|
parser.add_argument(
|
||||||
help="Max seed articles for forward snowball")
|
"--seed-limit",
|
||||||
|
type=int,
|
||||||
|
default=200,
|
||||||
|
help="Max seed articles for forward snowball",
|
||||||
|
)
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
print("=" * 70)
|
print("=" * 70)
|
||||||
@@ -298,9 +309,16 @@ def main() -> None:
|
|||||||
|
|
||||||
# --- Relevance filter: must mention skin/wound in title or abstract ---
|
# --- Relevance filter: must mention skin/wound in title or abstract ---
|
||||||
skin_keywords = [
|
skin_keywords = [
|
||||||
"skin substitute", "skin substitutes", "wound", "ulcer",
|
"skin substitute",
|
||||||
"biological dressing", "tissue product", "graft",
|
"skin substitutes",
|
||||||
"dermal", "epidermal", "bioengineered",
|
"wound",
|
||||||
|
"ulcer",
|
||||||
|
"biological dressing",
|
||||||
|
"tissue product",
|
||||||
|
"graft",
|
||||||
|
"dermal",
|
||||||
|
"epidermal",
|
||||||
|
"bioengineered",
|
||||||
]
|
]
|
||||||
relevant = []
|
relevant = []
|
||||||
for art in new_articles:
|
for art in new_articles:
|
||||||
|
|||||||
@@ -41,7 +41,7 @@ def main() -> int:
|
|||||||
mapping = {item["env"]: item["key"] for item in mapping_list}
|
mapping = {item["env"]: item["key"] for item in mapping_list}
|
||||||
|
|
||||||
if not mapping:
|
if not mapping:
|
||||||
print(f"No secret mappings defined in [databricks.secrets]")
|
print("No secret mappings defined in [databricks.secrets]")
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
from aco.lake.unity import UnityClient
|
from aco.lake.unity import UnityClient
|
||||||
|
|||||||
@@ -12,9 +12,17 @@ from pathlib import Path
|
|||||||
from zot.db import generate_key
|
from zot.db import generate_key
|
||||||
|
|
||||||
HOST_DB = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/tmp/host-zotero.sqlite")
|
HOST_DB = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/tmp/host-zotero.sqlite")
|
||||||
CONTAINER_DB = Path(sys.argv[2]) if len(sys.argv) > 2 else Path("/home/ubuntu/Zotero/zotero.sqlite")
|
CONTAINER_DB = (
|
||||||
HOST_STORAGE = Path(sys.argv[3]) if len(sys.argv) > 3 else Path("/home/ubuntu/data/Zotero/storage")
|
Path(sys.argv[2])
|
||||||
CONTAINER_STORAGE = Path(sys.argv[4]) if len(sys.argv) > 4 else Path("/home/ubuntu/Zotero/storage")
|
if len(sys.argv) > 2
|
||||||
|
else Path("/home/ubuntu/Zotero/zotero.sqlite")
|
||||||
|
)
|
||||||
|
HOST_STORAGE = (
|
||||||
|
Path(sys.argv[3]) if len(sys.argv) > 3 else Path("/home/ubuntu/data/Zotero/storage")
|
||||||
|
)
|
||||||
|
CONTAINER_STORAGE = (
|
||||||
|
Path(sys.argv[4]) if len(sys.argv) > 4 else Path("/home/ubuntu/Zotero/storage")
|
||||||
|
)
|
||||||
|
|
||||||
LIBRARY_ID = 1 # user library
|
LIBRARY_ID = 1 # user library
|
||||||
|
|
||||||
@@ -41,16 +49,32 @@ def main():
|
|||||||
dst.execute("PRAGMA foreign_keys=OFF") # we handle ordering ourselves
|
dst.execute("PRAGMA foreign_keys=OFF") # we handle ordering ourselves
|
||||||
|
|
||||||
# Collect existing keys in destination
|
# Collect existing keys in destination
|
||||||
existing_keys = {r[0] for r in dst.execute("SELECT key FROM items WHERE libraryID=?", (LIBRARY_ID,))}
|
existing_keys = {
|
||||||
existing_keys |= {r[0] for r in dst.execute("SELECT key FROM collections WHERE libraryID=?", (LIBRARY_ID,))}
|
r[0]
|
||||||
|
for r in dst.execute("SELECT key FROM items WHERE libraryID=?", (LIBRARY_ID,))
|
||||||
|
}
|
||||||
|
existing_keys |= {
|
||||||
|
r[0]
|
||||||
|
for r in dst.execute(
|
||||||
|
"SELECT key FROM collections WHERE libraryID=?", (LIBRARY_ID,)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
# --- Max IDs in destination ---
|
# --- Max IDs in destination ---
|
||||||
max_item_id = dst.execute("SELECT COALESCE(MAX(itemID),0) FROM items").fetchone()[0]
|
max_item_id = dst.execute("SELECT COALESCE(MAX(itemID),0) FROM items").fetchone()[0]
|
||||||
max_coll_id = dst.execute("SELECT COALESCE(MAX(collectionID),0) FROM collections").fetchone()[0]
|
max_coll_id = dst.execute(
|
||||||
max_value_id = dst.execute("SELECT COALESCE(MAX(valueID),0) FROM itemDataValues").fetchone()[0]
|
"SELECT COALESCE(MAX(collectionID),0) FROM collections"
|
||||||
max_creator_id = dst.execute("SELECT COALESCE(MAX(creatorID),0) FROM creators").fetchone()[0]
|
).fetchone()[0]
|
||||||
|
max_value_id = dst.execute(
|
||||||
|
"SELECT COALESCE(MAX(valueID),0) FROM itemDataValues"
|
||||||
|
).fetchone()[0]
|
||||||
|
max_creator_id = dst.execute(
|
||||||
|
"SELECT COALESCE(MAX(creatorID),0) FROM creators"
|
||||||
|
).fetchone()[0]
|
||||||
max_tag_id = dst.execute("SELECT COALESCE(MAX(tagID),0) FROM tags").fetchone()[0]
|
max_tag_id = dst.execute("SELECT COALESCE(MAX(tagID),0) FROM tags").fetchone()[0]
|
||||||
max_word_id = dst.execute("SELECT COALESCE(MAX(wordID),0) FROM fulltextWords").fetchone()[0]
|
max_word_id = dst.execute(
|
||||||
|
"SELECT COALESCE(MAX(wordID),0) FROM fulltextWords"
|
||||||
|
).fetchone()[0]
|
||||||
|
|
||||||
# ============================================================
|
# ============================================================
|
||||||
# 1. Create "archive" collection + mirror host collection tree
|
# 1. Create "archive" collection + mirror host collection tree
|
||||||
@@ -103,7 +127,10 @@ def main():
|
|||||||
src_value_map[r["valueID"]] = dst_values[val]
|
src_value_map[r["valueID"]] = dst_values[val]
|
||||||
else:
|
else:
|
||||||
max_value_id += 1
|
max_value_id += 1
|
||||||
dst.execute("INSERT INTO itemDataValues (valueID, value) VALUES (?,?)", (max_value_id, r["value"]))
|
dst.execute(
|
||||||
|
"INSERT INTO itemDataValues (valueID, value) VALUES (?,?)",
|
||||||
|
(max_value_id, r["value"]),
|
||||||
|
)
|
||||||
dst_values[val] = max_value_id
|
dst_values[val] = max_value_id
|
||||||
src_value_map[r["valueID"]] = max_value_id
|
src_value_map[r["valueID"]] = max_value_id
|
||||||
print(f"Mapped {len(src_value_map)} itemDataValues")
|
print(f"Mapped {len(src_value_map)} itemDataValues")
|
||||||
@@ -112,11 +139,15 @@ def main():
|
|||||||
# 3. Migrate creators (deduplicated by lastName+firstName+fieldMode)
|
# 3. Migrate creators (deduplicated by lastName+firstName+fieldMode)
|
||||||
# ============================================================
|
# ============================================================
|
||||||
dst_creators: dict[tuple, int] = {}
|
dst_creators: dict[tuple, int] = {}
|
||||||
for r in dst.execute("SELECT creatorID, firstName, lastName, fieldMode FROM creators"):
|
for r in dst.execute(
|
||||||
|
"SELECT creatorID, firstName, lastName, fieldMode FROM creators"
|
||||||
|
):
|
||||||
dst_creators[(r["lastName"], r["firstName"], r["fieldMode"])] = r["creatorID"]
|
dst_creators[(r["lastName"], r["firstName"], r["fieldMode"])] = r["creatorID"]
|
||||||
|
|
||||||
src_creator_map: dict[int, int] = {}
|
src_creator_map: dict[int, int] = {}
|
||||||
for r in src.execute("SELECT creatorID, firstName, lastName, fieldMode FROM creators"):
|
for r in src.execute(
|
||||||
|
"SELECT creatorID, firstName, lastName, fieldMode FROM creators"
|
||||||
|
):
|
||||||
ck = (r["lastName"], r["firstName"], r["fieldMode"])
|
ck = (r["lastName"], r["firstName"], r["fieldMode"])
|
||||||
if ck in dst_creators:
|
if ck in dst_creators:
|
||||||
src_creator_map[r["creatorID"]] = dst_creators[ck]
|
src_creator_map[r["creatorID"]] = dst_creators[ck]
|
||||||
@@ -143,7 +174,9 @@ def main():
|
|||||||
src_tag_map[r["tagID"]] = dst_tags[r["name"]]
|
src_tag_map[r["tagID"]] = dst_tags[r["name"]]
|
||||||
else:
|
else:
|
||||||
max_tag_id += 1
|
max_tag_id += 1
|
||||||
dst.execute("INSERT INTO tags (tagID, name) VALUES (?,?)", (max_tag_id, r["name"]))
|
dst.execute(
|
||||||
|
"INSERT INTO tags (tagID, name) VALUES (?,?)", (max_tag_id, r["name"])
|
||||||
|
)
|
||||||
dst_tags[r["name"]] = max_tag_id
|
dst_tags[r["name"]] = max_tag_id
|
||||||
src_tag_map[r["tagID"]] = max_tag_id
|
src_tag_map[r["tagID"]] = max_tag_id
|
||||||
print(f"Mapped {len(src_tag_map)} tags")
|
print(f"Mapped {len(src_tag_map)} tags")
|
||||||
@@ -161,7 +194,10 @@ def main():
|
|||||||
src_word_map[r["wordID"]] = dst_words[r["word"]]
|
src_word_map[r["wordID"]] = dst_words[r["word"]]
|
||||||
else:
|
else:
|
||||||
max_word_id += 1
|
max_word_id += 1
|
||||||
dst.execute("INSERT INTO fulltextWords (wordID, word) VALUES (?,?)", (max_word_id, r["word"]))
|
dst.execute(
|
||||||
|
"INSERT INTO fulltextWords (wordID, word) VALUES (?,?)",
|
||||||
|
(max_word_id, r["word"]),
|
||||||
|
)
|
||||||
dst_words[r["word"]] = max_word_id
|
dst_words[r["word"]] = max_word_id
|
||||||
src_word_map[r["wordID"]] = max_word_id
|
src_word_map[r["wordID"]] = max_word_id
|
||||||
print(f"Mapped {len(src_word_map)} fulltextWords")
|
print(f"Mapped {len(src_word_map)} fulltextWords")
|
||||||
@@ -187,7 +223,15 @@ def main():
|
|||||||
dst.execute(
|
dst.execute(
|
||||||
"INSERT INTO items (itemID, itemTypeID, dateAdded, dateModified, clientDateModified, libraryID, key, version, synced) "
|
"INSERT INTO items (itemID, itemTypeID, dateAdded, dateModified, clientDateModified, libraryID, key, version, synced) "
|
||||||
"VALUES (?,?,?,?,?,?,?,0,0)",
|
"VALUES (?,?,?,?,?,?,?,0,0)",
|
||||||
(max_item_id, hi["itemTypeID"], hi["dateAdded"], hi["dateModified"], hi["clientDateModified"], LIBRARY_ID, new_key),
|
(
|
||||||
|
max_item_id,
|
||||||
|
hi["itemTypeID"],
|
||||||
|
hi["dateAdded"],
|
||||||
|
hi["dateModified"],
|
||||||
|
hi["clientDateModified"],
|
||||||
|
LIBRARY_ID,
|
||||||
|
new_key,
|
||||||
|
),
|
||||||
)
|
)
|
||||||
print(f"Migrated {len(item_id_map)} items")
|
print(f"Migrated {len(item_id_map)} items")
|
||||||
|
|
||||||
@@ -209,12 +253,19 @@ def main():
|
|||||||
# 8. Migrate itemCreators
|
# 8. Migrate itemCreators
|
||||||
# ============================================================
|
# ============================================================
|
||||||
count = 0
|
count = 0
|
||||||
for r in src.execute("SELECT itemID, creatorID, creatorTypeID, orderIndex FROM itemCreators"):
|
for r in src.execute(
|
||||||
|
"SELECT itemID, creatorID, creatorTypeID, orderIndex FROM itemCreators"
|
||||||
|
):
|
||||||
if r["itemID"] not in item_id_map:
|
if r["itemID"] not in item_id_map:
|
||||||
continue
|
continue
|
||||||
dst.execute(
|
dst.execute(
|
||||||
"INSERT INTO itemCreators (itemID, creatorID, creatorTypeID, orderIndex) VALUES (?,?,?,?)",
|
"INSERT INTO itemCreators (itemID, creatorID, creatorTypeID, orderIndex) VALUES (?,?,?,?)",
|
||||||
(item_id_map[r["itemID"]], src_creator_map[r["creatorID"]], r["creatorTypeID"], r["orderIndex"]),
|
(
|
||||||
|
item_id_map[r["itemID"]],
|
||||||
|
src_creator_map[r["creatorID"]],
|
||||||
|
r["creatorTypeID"],
|
||||||
|
r["orderIndex"],
|
||||||
|
),
|
||||||
)
|
)
|
||||||
count += 1
|
count += 1
|
||||||
print(f"Migrated {count} itemCreators rows")
|
print(f"Migrated {count} itemCreators rows")
|
||||||
@@ -223,14 +274,30 @@ def main():
|
|||||||
# 9. Migrate itemAttachments
|
# 9. Migrate itemAttachments
|
||||||
# ============================================================
|
# ============================================================
|
||||||
count = 0
|
count = 0
|
||||||
for r in src.execute("SELECT itemID, parentItemID, linkMode, contentType, charsetID, path, syncState, storageModTime, storageHash FROM itemAttachments"):
|
for r in src.execute(
|
||||||
|
"SELECT itemID, parentItemID, linkMode, contentType, charsetID, path, syncState, storageModTime, storageHash FROM itemAttachments"
|
||||||
|
):
|
||||||
if r["itemID"] not in item_id_map:
|
if r["itemID"] not in item_id_map:
|
||||||
continue
|
continue
|
||||||
parent = item_id_map.get(r["parentItemID"]) if r["parentItemID"] is not None else None
|
parent = (
|
||||||
|
item_id_map.get(r["parentItemID"])
|
||||||
|
if r["parentItemID"] is not None
|
||||||
|
else None
|
||||||
|
)
|
||||||
dst.execute(
|
dst.execute(
|
||||||
"INSERT INTO itemAttachments (itemID, parentItemID, linkMode, contentType, charsetID, path, syncState, storageModTime, storageHash) "
|
"INSERT INTO itemAttachments (itemID, parentItemID, linkMode, contentType, charsetID, path, syncState, storageModTime, storageHash) "
|
||||||
"VALUES (?,?,?,?,?,?,?,?,?)",
|
"VALUES (?,?,?,?,?,?,?,?,?)",
|
||||||
(item_id_map[r["itemID"]], parent, r["linkMode"], r["contentType"], r["charsetID"], r["path"], r["syncState"], r["storageModTime"], r["storageHash"]),
|
(
|
||||||
|
item_id_map[r["itemID"]],
|
||||||
|
parent,
|
||||||
|
r["linkMode"],
|
||||||
|
r["contentType"],
|
||||||
|
r["charsetID"],
|
||||||
|
r["path"],
|
||||||
|
r["syncState"],
|
||||||
|
r["storageModTime"],
|
||||||
|
r["storageHash"],
|
||||||
|
),
|
||||||
)
|
)
|
||||||
count += 1
|
count += 1
|
||||||
print(f"Migrated {count} itemAttachments rows")
|
print(f"Migrated {count} itemAttachments rows")
|
||||||
@@ -242,7 +309,11 @@ def main():
|
|||||||
for r in src.execute("SELECT itemID, parentItemID, note, title FROM itemNotes"):
|
for r in src.execute("SELECT itemID, parentItemID, note, title FROM itemNotes"):
|
||||||
if r["itemID"] not in item_id_map:
|
if r["itemID"] not in item_id_map:
|
||||||
continue
|
continue
|
||||||
parent = item_id_map.get(r["parentItemID"]) if r["parentItemID"] is not None else None
|
parent = (
|
||||||
|
item_id_map.get(r["parentItemID"])
|
||||||
|
if r["parentItemID"] is not None
|
||||||
|
else None
|
||||||
|
)
|
||||||
dst.execute(
|
dst.execute(
|
||||||
"INSERT INTO itemNotes (itemID, parentItemID, note, title) VALUES (?,?,?,?)",
|
"INSERT INTO itemNotes (itemID, parentItemID, note, title) VALUES (?,?,?,?)",
|
||||||
(item_id_map[r["itemID"]], parent, r["note"], r["title"]),
|
(item_id_map[r["itemID"]], parent, r["note"], r["title"]),
|
||||||
@@ -283,13 +354,21 @@ def main():
|
|||||||
# 13. Migrate fulltextItems + fulltextItemWords
|
# 13. Migrate fulltextItems + fulltextItemWords
|
||||||
# ============================================================
|
# ============================================================
|
||||||
count = 0
|
count = 0
|
||||||
for r in src.execute("SELECT itemID, indexedPages, totalPages, indexedChars, totalChars FROM fulltextItems"):
|
for r in src.execute(
|
||||||
|
"SELECT itemID, indexedPages, totalPages, indexedChars, totalChars FROM fulltextItems"
|
||||||
|
):
|
||||||
if r["itemID"] not in item_id_map:
|
if r["itemID"] not in item_id_map:
|
||||||
continue
|
continue
|
||||||
dst.execute(
|
dst.execute(
|
||||||
"INSERT INTO fulltextItems (itemID, indexedPages, totalPages, indexedChars, totalChars, version, synced) "
|
"INSERT INTO fulltextItems (itemID, indexedPages, totalPages, indexedChars, totalChars, version, synced) "
|
||||||
"VALUES (?,?,?,?,?,0,0)",
|
"VALUES (?,?,?,?,?,0,0)",
|
||||||
(item_id_map[r["itemID"]], r["indexedPages"], r["totalPages"], r["indexedChars"], r["totalChars"]),
|
(
|
||||||
|
item_id_map[r["itemID"]],
|
||||||
|
r["indexedPages"],
|
||||||
|
r["totalPages"],
|
||||||
|
r["indexedChars"],
|
||||||
|
r["totalChars"],
|
||||||
|
),
|
||||||
)
|
)
|
||||||
count += 1
|
count += 1
|
||||||
print(f"Migrated {count} fulltextItems rows")
|
print(f"Migrated {count} fulltextItems rows")
|
||||||
@@ -312,13 +391,17 @@ def main():
|
|||||||
# ============================================================
|
# ============================================================
|
||||||
# Zotero triggers prevent adding child attachments/notes to collections
|
# Zotero triggers prevent adding child attachments/notes to collections
|
||||||
child_item_ids = set()
|
child_item_ids = set()
|
||||||
for r in src.execute("SELECT itemID FROM itemAttachments WHERE parentItemID IS NOT NULL"):
|
for r in src.execute(
|
||||||
|
"SELECT itemID FROM itemAttachments WHERE parentItemID IS NOT NULL"
|
||||||
|
):
|
||||||
child_item_ids.add(r["itemID"])
|
child_item_ids.add(r["itemID"])
|
||||||
for r in src.execute("SELECT itemID FROM itemNotes WHERE parentItemID IS NOT NULL"):
|
for r in src.execute("SELECT itemID FROM itemNotes WHERE parentItemID IS NOT NULL"):
|
||||||
child_item_ids.add(r["itemID"])
|
child_item_ids.add(r["itemID"])
|
||||||
|
|
||||||
count = 0
|
count = 0
|
||||||
for r in src.execute("SELECT collectionID, itemID, orderIndex FROM collectionItems"):
|
for r in src.execute(
|
||||||
|
"SELECT collectionID, itemID, orderIndex FROM collectionItems"
|
||||||
|
):
|
||||||
if r["itemID"] not in item_id_map:
|
if r["itemID"] not in item_id_map:
|
||||||
continue
|
continue
|
||||||
if r["collectionID"] not in coll_id_map:
|
if r["collectionID"] not in coll_id_map:
|
||||||
|
|||||||
@@ -1,100 +1,27 @@
|
|||||||
"""Fix Zotero items that have data stored under base field IDs instead of mapped field IDs.
|
"""Thin shim — logic moved to :mod:`zot.ops`.
|
||||||
|
|
||||||
Zotero uses type-specific fields (e.g. caseName for case items) that map to base fields
|
Prefer: ``uv run stack zot fix-fields``.
|
||||||
(e.g. title). When items have data under the base field ID but their item type requires the
|
|
||||||
mapped field ID, Zotero logs "is not a valid field" errors.
|
|
||||||
|
|
||||||
This script remaps base field IDs to the correct type-specific field IDs, and removes
|
|
||||||
truly invalid field data (fields with no mapping for that type).
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import sqlite3
|
from __future__ import annotations
|
||||||
|
|
||||||
import sys
|
import sys
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
DB = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/home/ubuntu/Zotero/zotero.sqlite")
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
from zot.ops import fix_fields
|
||||||
|
|
||||||
def main():
|
db = (
|
||||||
db = sqlite3.connect(str(DB))
|
Path(sys.argv[1])
|
||||||
db.row_factory = sqlite3.Row
|
if len(sys.argv) > 1
|
||||||
db.execute("PRAGMA journal_mode=WAL")
|
else Path("/home/ubuntu/Zotero/zotero.sqlite")
|
||||||
db.execute("PRAGMA foreign_keys=OFF")
|
)
|
||||||
|
out = fix_fields(db)
|
||||||
# Build set of valid (itemTypeID, fieldID) pairs
|
for k, v in out.items():
|
||||||
valid_fields: set[tuple[int, int]] = set()
|
print(f"{k}: {v}")
|
||||||
for r in db.execute("SELECT itemTypeID, fieldID FROM itemTypeFieldsCombined"):
|
return 0
|
||||||
valid_fields.add((r["itemTypeID"], r["fieldID"]))
|
|
||||||
|
|
||||||
# Build base field mapping: (itemTypeID, baseFieldID) -> mappedFieldID
|
|
||||||
field_map: dict[tuple[int, int], int] = {}
|
|
||||||
for r in db.execute("SELECT itemTypeID, baseFieldID, fieldID FROM baseFieldMappingsCombined"):
|
|
||||||
field_map[(r["itemTypeID"], r["baseFieldID"])] = r["fieldID"]
|
|
||||||
|
|
||||||
# Find all invalid itemData rows
|
|
||||||
# An itemData row is invalid if (itemTypeID, fieldID) is not in valid_fields
|
|
||||||
invalid = db.execute("""
|
|
||||||
SELECT id.itemID, id.fieldID, id.valueID, i.itemTypeID
|
|
||||||
FROM itemData id
|
|
||||||
JOIN items i ON id.itemID = i.itemID
|
|
||||||
WHERE i.libraryID = 1
|
|
||||||
AND (i.itemTypeID, id.fieldID) NOT IN (
|
|
||||||
SELECT itemTypeID, fieldID FROM itemTypeFieldsCombined
|
|
||||||
)
|
|
||||||
""").fetchall()
|
|
||||||
|
|
||||||
print(f"Found {len(invalid)} invalid itemData rows")
|
|
||||||
|
|
||||||
remapped = 0
|
|
||||||
deleted = 0
|
|
||||||
conflicts = 0
|
|
||||||
|
|
||||||
for row in invalid:
|
|
||||||
item_id = row["itemID"]
|
|
||||||
old_field = row["fieldID"]
|
|
||||||
value_id = row["valueID"]
|
|
||||||
item_type = row["itemTypeID"]
|
|
||||||
|
|
||||||
# Check if there's a mapping for this base field to a type-specific field
|
|
||||||
mapped_field = field_map.get((item_type, old_field))
|
|
||||||
|
|
||||||
if mapped_field and (item_type, mapped_field) in valid_fields:
|
|
||||||
# Check if the mapped field already has data for this item
|
|
||||||
existing = db.execute(
|
|
||||||
"SELECT valueID FROM itemData WHERE itemID=? AND fieldID=?",
|
|
||||||
(item_id, mapped_field),
|
|
||||||
).fetchone()
|
|
||||||
|
|
||||||
if existing:
|
|
||||||
# Mapped field already has data — delete the base field entry
|
|
||||||
db.execute(
|
|
||||||
"DELETE FROM itemData WHERE itemID=? AND fieldID=?",
|
|
||||||
(item_id, old_field),
|
|
||||||
)
|
|
||||||
conflicts += 1
|
|
||||||
else:
|
|
||||||
# Remap: update fieldID from base to mapped
|
|
||||||
db.execute(
|
|
||||||
"UPDATE itemData SET fieldID=? WHERE itemID=? AND fieldID=?",
|
|
||||||
(mapped_field, item_id, old_field),
|
|
||||||
)
|
|
||||||
remapped += 1
|
|
||||||
else:
|
|
||||||
# No valid mapping exists — delete the orphan data
|
|
||||||
db.execute(
|
|
||||||
"DELETE FROM itemData WHERE itemID=? AND fieldID=?",
|
|
||||||
(item_id, old_field),
|
|
||||||
)
|
|
||||||
deleted += 1
|
|
||||||
|
|
||||||
db.commit()
|
|
||||||
db.close()
|
|
||||||
|
|
||||||
print(f"Remapped: {remapped}")
|
|
||||||
print(f"Deleted (no mapping): {deleted}")
|
|
||||||
print(f"Deleted (conflict): {conflicts}")
|
|
||||||
print(f"Total fixed: {remapped + deleted + conflicts}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
main()
|
sys.exit(main())
|
||||||
|
|||||||
145
infra/droplets/mail-setup.sh
Executable file
@@ -0,0 +1,145 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Provision a DigitalOcean Maddy mail droplet.
|
||||||
|
#
|
||||||
|
# Runs under cloud-init with these env vars already exported by the
|
||||||
|
# builder (see src/cli/mail.py::_cloud_init):
|
||||||
|
# HOSTNAME fqdn of this mail server, e.g. mail.corwins.media
|
||||||
|
# PRIMARY_DOMAIN bare domain, e.g. corwins.media
|
||||||
|
# POSTMASTER_PASSWORD initial password for postmaster@<domain>
|
||||||
|
# GITEA_SMTP_PASSWORD initial password for gitea@<domain> (app-only account)
|
||||||
|
#
|
||||||
|
# Idempotent — safe to re-run. Installs Docker + Maddy, lays down config,
|
||||||
|
# provisions the two seed accounts, and opens the firewall for SMTP/IMAP.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
export DEBIAN_FRONTEND=noninteractive
|
||||||
|
|
||||||
|
apt-get update -qq
|
||||||
|
apt-get install -y -qq curl gettext-base ca-certificates ufw
|
||||||
|
|
||||||
|
# ── Docker Engine (upstream, not the ancient distro package) ──────
|
||||||
|
if ! command -v docker >/dev/null 2>&1; then
|
||||||
|
install -m 0755 -d /etc/apt/keyrings
|
||||||
|
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
|
||||||
|
gpg --dearmor -o /etc/apt/keyrings/docker.gpg
|
||||||
|
chmod a+r /etc/apt/keyrings/docker.gpg
|
||||||
|
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
|
||||||
|
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
|
||||||
|
> /etc/apt/sources.list.d/docker.list
|
||||||
|
apt-get update -qq
|
||||||
|
apt-get install -y -qq docker-ce docker-ce-cli containerd.io docker-compose-plugin
|
||||||
|
systemctl enable --now docker
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Maddy config + data dir ───────────────────────────────────────
|
||||||
|
install -d -m 0750 /srv/mail /srv/mail/tls /srv/mail/data
|
||||||
|
|
||||||
|
cat > /srv/mail/maddy.conf.tpl <<'MADDY_CONF_EOF'
|
||||||
|
__MADDY_CONF_TPL_PLACEHOLDER__
|
||||||
|
MADDY_CONF_EOF
|
||||||
|
|
||||||
|
# Render config atomically and detect changes so we only restart maddy
|
||||||
|
# when the rendered file actually differs.
|
||||||
|
envsubst '$HOSTNAME $PRIMARY_DOMAIN' \
|
||||||
|
< /srv/mail/maddy.conf.tpl > /srv/mail/data/maddy.conf.new
|
||||||
|
if ! cmp -s /srv/mail/data/maddy.conf /srv/mail/data/maddy.conf.new 2>/dev/null; then
|
||||||
|
mv /srv/mail/data/maddy.conf.new /srv/mail/data/maddy.conf
|
||||||
|
MADDY_CONF_CHANGED=1
|
||||||
|
else
|
||||||
|
rm -f /srv/mail/data/maddy.conf.new
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Empty aliases file so the local_aliases chain doesn't error on first boot
|
||||||
|
[ -f /srv/mail/data/aliases ] || : > /srv/mail/data/aliases
|
||||||
|
|
||||||
|
# ── docker-compose.yml ────────────────────────────────────────────
|
||||||
|
cat > /srv/mail/docker-compose.yml <<COMPOSE_EOF
|
||||||
|
services:
|
||||||
|
mail:
|
||||||
|
image: foxcpp/maddy:latest
|
||||||
|
container_name: mail
|
||||||
|
restart: unless-stopped
|
||||||
|
hostname: ${HOSTNAME}
|
||||||
|
environment:
|
||||||
|
MADDY_HOSTNAME: ${HOSTNAME}
|
||||||
|
MADDY_DOMAIN: ${PRIMARY_DOMAIN}
|
||||||
|
ports:
|
||||||
|
- "25:25"
|
||||||
|
- "143:143"
|
||||||
|
- "465:465"
|
||||||
|
- "587:587"
|
||||||
|
- "993:993"
|
||||||
|
volumes:
|
||||||
|
- ./data:/data
|
||||||
|
- ./tls:/data/tls:ro
|
||||||
|
COMPOSE_EOF
|
||||||
|
|
||||||
|
# ── Let's Encrypt via DNS-01 (Cloudflare plugin) ──────────────────
|
||||||
|
# DNS-01 not HTTP-01 because:
|
||||||
|
# 1. The mail.$(domain) A record doesn't yet point at this droplet
|
||||||
|
# when cloud-init runs (the CLI writes DNS *after* the droplet is
|
||||||
|
# up), so HTTP-01 would race and fail.
|
||||||
|
# 2. Port 80 doesn't need to be internet-facing on a mail server.
|
||||||
|
# 3. DNS-01 works even when the droplet has no inbound HTTP at all.
|
||||||
|
#
|
||||||
|
# Requires CLOUDFLARE_API_TOKEN to be exported by the user_data wrapper.
|
||||||
|
apt-get install -y -qq certbot python3-certbot-dns-cloudflare
|
||||||
|
|
||||||
|
mkdir -p /etc/letsencrypt/cloudflare
|
||||||
|
cat > /etc/letsencrypt/cloudflare/credentials.ini <<CF_EOF
|
||||||
|
dns_cloudflare_api_token = ${CLOUDFLARE_API_TOKEN}
|
||||||
|
CF_EOF
|
||||||
|
chmod 600 /etc/letsencrypt/cloudflare/credentials.ini
|
||||||
|
|
||||||
|
# --keep-until-expiring makes this safe to re-run.
|
||||||
|
certbot certonly --dns-cloudflare \
|
||||||
|
--dns-cloudflare-credentials /etc/letsencrypt/cloudflare/credentials.ini \
|
||||||
|
--dns-cloudflare-propagation-seconds 20 \
|
||||||
|
--non-interactive --agree-tos --keep-until-expiring \
|
||||||
|
-m "postmaster@${PRIMARY_DOMAIN}" -d "${HOSTNAME}" || true
|
||||||
|
|
||||||
|
if [ -f "/etc/letsencrypt/live/${HOSTNAME}/fullchain.pem" ]; then
|
||||||
|
cp /etc/letsencrypt/live/${HOSTNAME}/fullchain.pem /srv/mail/tls/fullchain.pem
|
||||||
|
cp /etc/letsencrypt/live/${HOSTNAME}/privkey.pem /srv/mail/tls/privkey.pem
|
||||||
|
chmod 644 /srv/mail/tls/fullchain.pem
|
||||||
|
chmod 600 /srv/mail/tls/privkey.pem
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Daily renew hook — DNS-01 doesn't need port 80.
|
||||||
|
cat > /etc/cron.daily/maddy-cert-renew <<'CRON_EOF'
|
||||||
|
#!/bin/bash
|
||||||
|
set -e
|
||||||
|
certbot renew --quiet --deploy-hook "cp /etc/letsencrypt/live/${HOSTNAME}/fullchain.pem /srv/mail/tls/ && \
|
||||||
|
cp /etc/letsencrypt/live/${HOSTNAME}/privkey.pem /srv/mail/tls/ && \
|
||||||
|
docker restart mail"
|
||||||
|
CRON_EOF
|
||||||
|
chmod +x /etc/cron.daily/maddy-cert-renew
|
||||||
|
|
||||||
|
# ── Firewall ──────────────────────────────────────────────────────
|
||||||
|
ufw default deny incoming
|
||||||
|
ufw default allow outgoing
|
||||||
|
ufw allow 22/tcp
|
||||||
|
ufw allow 25/tcp
|
||||||
|
ufw allow 143/tcp
|
||||||
|
ufw allow 465/tcp
|
||||||
|
ufw allow 587/tcp
|
||||||
|
ufw allow 993/tcp
|
||||||
|
ufw --force enable
|
||||||
|
|
||||||
|
# ── Bring up maddy ────────────────────────────────────────────────
|
||||||
|
cd /srv/mail
|
||||||
|
docker compose pull
|
||||||
|
docker compose up -d
|
||||||
|
# Only restart when config actually drifted — keeps re-runs cheap.
|
||||||
|
if [ "${MADDY_CONF_CHANGED:-0}" = "1" ]; then
|
||||||
|
docker restart mail
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Mailbox seeding is owned by the orchestrator (`stack mail` /
|
||||||
|
# `corwins mail`), not this script. The orchestrator SSHes in after
|
||||||
|
# Maddy is up and runs `maddy creds create` per mailbox. Keeping
|
||||||
|
# seeding out of cloud-init means adding a new mailbox is one CLI
|
||||||
|
# call (`stack mail rotate-creds <name>`) — no droplet rebuild, no
|
||||||
|
# template edit, no env-var threading.
|
||||||
|
|
||||||
|
echo "==> Maddy provisioning complete for ${HOSTNAME}"
|
||||||
2
infra/gitea/.dockerignore
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
custom/
|
||||||
|
templates/
|
||||||
725
infra/gitea/CROSSWALK.md
Normal file
@@ -0,0 +1,725 @@
|
|||||||
|
# Fhirworx ⇄ Gitea v1.25.4 CSS Variable Crosswalk
|
||||||
|
|
||||||
|
This report cross-references every CSS variable Gitea expects against
|
||||||
|
what fhirworx provides. Format:
|
||||||
|
|
||||||
|
- ✅ defined and value matches upstream intent
|
||||||
|
- ⚠️ defined with different value (intentional remap or risk)
|
||||||
|
- ❌ NOT defined → resolves to `initial` (transparent bg, black text)
|
||||||
|
- 🔍 used in source but not declared anywhere
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
| Set | Count |
|
||||||
|
|---|---|
|
||||||
|
| Upstream theme-gitea-light.css declares | 259 |
|
||||||
|
| Fhirworx light declares | 253 |
|
||||||
|
| Fhirworx dark declares | 253 |
|
||||||
|
| Total distinct --vars referenced in src | 186 |
|
||||||
|
| Vars used in src but NOT declared anywhere | 6 |
|
||||||
|
|
||||||
|
## Coverage verdict
|
||||||
|
|
||||||
|
**Light variant: ✅ defines every `--color-*` variable upstream uses (full coverage)**
|
||||||
|
**Dark variant: ✅ defines every `--color-*` variable upstream uses (full coverage)**
|
||||||
|
|
||||||
|
Structural vars (`--font-*`, `--border-radius`, `--gap-*`, etc.) come from `base.css`'s
|
||||||
|
|
||||||
|
`:root` and are inherited automatically; themes don't need to override them.
|
||||||
|
|
||||||
|
## 🔍 Variables used in source but NOT declared in upstream OR fhirworx
|
||||||
|
|
||||||
|
- `--fonts-default-override-ja` — referenced by 1 rules; first: `font_i18n.css::root :lang(ja)`
|
||||||
|
- `--fonts-default-override-ko` — referenced by 1 rules; first: `font_i18n.css::root :lang(ko)`
|
||||||
|
- `--fonts-default-override-zh-cn` — referenced by 1 rules; first: `font_i18n.css::root :lang(zh-CN)`
|
||||||
|
- `--fonts-default-override-zh-hk` — referenced by 1 rules; first: `font_i18n.css::root :lang(zh-HK)`
|
||||||
|
- `--fonts-default-override-zh-tw` — referenced by 1 rules; first: `font_i18n.css::root :lang(zh-TW)`
|
||||||
|
- `--fonts-override` — referenced by 1 rules; first: `base.css:} :root *`
|
||||||
|
|
||||||
|
## Extra vars defined by fhirworx LIGHT (fhirworx-only conveniences)
|
||||||
|
|
||||||
|
- `--color-error` = `var(--color-red)`
|
||||||
|
- `--color-info` = `var(--color-blue)`
|
||||||
|
- `--color-primary-foreground` = `var(--fw-primary-foreground)`
|
||||||
|
- `--color-success` = `var(--color-green)`
|
||||||
|
- `--color-warning` = `#7F5E08`
|
||||||
|
- `--fw-background` = `#F7F5F0`
|
||||||
|
- `--fw-border` = `#B8B3A4`
|
||||||
|
- `--fw-card` = `#FFFFFF`
|
||||||
|
- `--fw-card-hover` = `#F3F0EA`
|
||||||
|
- `--fw-chart-3` = `#A88B5C`
|
||||||
|
- `--fw-chart-5` = `#C8702A`
|
||||||
|
- `--fw-destructive` = `#C0392B`
|
||||||
|
- `--fw-foreground` = `#1A1A18`
|
||||||
|
- `--fw-link` = `#2D5F7F`
|
||||||
|
- `--fw-muted-foreground` = `#6B6B68`
|
||||||
|
- `--fw-nav-active-bg` = `#2E4359`
|
||||||
|
- `--fw-nav-bg` = `#1C2B3A`
|
||||||
|
- `--fw-nav-hover-bg` = `#253748`
|
||||||
|
- `--fw-nav-text` = `#F0F4F8`
|
||||||
|
- `--fw-primary` = `#1C2B3A`
|
||||||
|
- `--fw-primary-foreground` = `#F7F5F0`
|
||||||
|
- `--fw-secondary` = `#EDEBE6`
|
||||||
|
|
||||||
|
## Extra vars defined by fhirworx DARK
|
||||||
|
|
||||||
|
- `--color-error` = `var(--color-red)`
|
||||||
|
- `--color-info` = `var(--color-blue)`
|
||||||
|
- `--color-primary-foreground` = `var(--fw-primary-foreground)`
|
||||||
|
- `--color-success` = `var(--color-green)`
|
||||||
|
- `--color-warning` = `var(--color-yellow)`
|
||||||
|
- `--fonts-override` = `"Source Serif 4", Georgia, serif`
|
||||||
|
- `--fw-background` = `#0F1419`
|
||||||
|
- `--fw-border` = `#3A4452`
|
||||||
|
- `--fw-card` = `#1A2028`
|
||||||
|
- `--fw-card-hover` = `#232A34`
|
||||||
|
- `--fw-chart-3` = `#B8A078`
|
||||||
|
- `--fw-chart-5` = `#E08A4A`
|
||||||
|
- `--fw-destructive` = `#E07A6E`
|
||||||
|
- `--fw-foreground` = `#E8E5DD`
|
||||||
|
- `--fw-link` = `#5C9CC0`
|
||||||
|
- `--fw-muted-foreground` = `#8A8A87`
|
||||||
|
- `--fw-nav-active-bg` = `#25384C`
|
||||||
|
- `--fw-nav-bg` = `#0A0E13`
|
||||||
|
- `--fw-nav-hover-bg` = `#1C2B3A`
|
||||||
|
- `--fw-nav-text` = `#E8F0F8`
|
||||||
|
- `--fw-primary` = `#5684A3`
|
||||||
|
- `--fw-primary-foreground` = `#0F1419`
|
||||||
|
- `--fw-secondary` = `#1F262E`
|
||||||
|
|
||||||
|
## Variable crosswalk — every var referenced in source
|
||||||
|
|
||||||
|
| Var | Upstream value | Fhirworx light | Fhirworx dark | Source consumers (top 3) |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| ❌ `--background-view-image` | `url("data:image/png` | `—` | `—` | features/imagediff.css, repo/file-view.css |
|
||||||
|
| ❌ `--border-radius` | `4px` | `—` | `—` | base.css, codemirror/base.css, features/codeedito… |
|
||||||
|
| ❌ `--border-radius-full` | `99999px` | `—` | `—` | base.css, modules/animations.css, repo.css |
|
||||||
|
| ❌ `--border-radius-medium` | `6px` | `—` | `—` | features/codeeditor.css, features/dropzone.css, r… |
|
||||||
|
| ❌ `--checkbox-mask-checked` | `url('data:image/svg+xml` | `—` | `—` | markup/content.css |
|
||||||
|
| ❌ `--checkbox-mask-indeterminate` | `url('data:image/svg+xml` | `—` | `—` | markup/content.css |
|
||||||
|
| ❌ `--checkbox-size` | `15px` | `—` | `—` | modules/checkbox.css |
|
||||||
|
| ⚠️ `--color-accent` | `var(--color-primary-light-1)` | `var(--fw-primary)` | `var(--color-primary-dark-1)` | base.css, review.css |
|
||||||
|
| ⚠️ `--color-active` | `#00001714` | `#0000171F` | `#E8F3FF24` | base.css, editor/fileeditor.css, helpers.css |
|
||||||
|
| ⚠️ `--color-ansi-black` | `#1e2327` | `#1E2327` | `#1E2327` | features/console.css |
|
||||||
|
| ⚠️ `--color-ansi-blue` | `#3a8ac6` | `#3A8AC6` | `#3A8AC6` | features/console.css |
|
||||||
|
| ⚠️ `--color-ansi-bright-black` | `#46494d` | `#46494D` | `#424851` | features/console.css |
|
||||||
|
| ⚠️ `--color-ansi-bright-blue` | `#4e96cc` | `#4E96CC` | `#4E96CC` | features/console.css |
|
||||||
|
| ⚠️ `--color-ansi-bright-cyan` | `#00b6ad` | `#00B6AD` | `#00B6AD` | features/console.css |
|
||||||
|
| ⚠️ `--color-ansi-bright-green` | `#93b373` | `#93B373` | `#93B373` | features/console.css |
|
||||||
|
| ⚠️ `--color-ansi-bright-magenta` | `#d74397` | `#D74397` | `#D74397` | features/console.css |
|
||||||
|
| ⚠️ `--color-ansi-bright-red` | `#d15a5a` | `#D15A5A` | `#D15A5A` | features/console.css |
|
||||||
|
| ✅ `--color-ansi-bright-white` | `var(--color-console-fg)` | `var(--color-console-fg)` | `var(--color-console-fg)` | features/console.css |
|
||||||
|
| ⚠️ `--color-ansi-bright-yellow` | `#eaaf03` | `#EAAF03` | `#EAAF03` | features/console.css |
|
||||||
|
| ⚠️ `--color-ansi-cyan` | `#00918a` | `#00918A` | `#00918A` | features/console.css |
|
||||||
|
| ⚠️ `--color-ansi-green` | `#87ab63` | `#87AB63` | `#87AB63` | features/console.css |
|
||||||
|
| ⚠️ `--color-ansi-magenta` | `#d22e8b` | `#D22E8B` | `#D22E8B` | features/console.css |
|
||||||
|
| ⚠️ `--color-ansi-red` | `#cc4848` | `#CC4848` | `#CC4848` | features/console.css |
|
||||||
|
| ✅ `--color-ansi-white` | `var(--color-console-fg-subtle)` | `var(--color-console-fg-subtle)` | `var(--color-console-fg-subtle)` | features/console.css |
|
||||||
|
| ⚠️ `--color-ansi-yellow` | `#cc9903` | `#CC9903` | `#CC9903` | features/console.css |
|
||||||
|
| ⚠️ `--color-blue` | `#2185d0` | `var(--fw-link)` | `var(--fw-link)` | actions.css, base.css, modules/message.css |
|
||||||
|
| ⚠️ `--color-blue-dark-1` | `#1e78bb` | `#244B63` | `#487FA1` | base.css |
|
||||||
|
| ⚠️ `--color-body` | `#ffffff` | `var(--fw-background) …` | `var(--fw-background) !importa…` | base.css, editor/fileeditor.css, modules/menu.css |
|
||||||
|
| ⚠️ `--color-box-body` | `#ffffff` | `var(--fw-card) …` | `var(--fw-card) !importa…` | markup/content.css, modules/menu.css, modules/mes… |
|
||||||
|
| ⚠️ `--color-box-body-highlight` | `#ecf5fd` | `var(--color-primary-light-7)` | `#1E2630` | repo.css |
|
||||||
|
| ⚠️ `--color-box-header` | `#f1f3f5` | `var(--fw-secondary)` | `var(--fw-secondary)` | modules/header.css, modules/menu.css, modules/tab… |
|
||||||
|
| ⚠️ `--color-button` | `#f8f9fb` | `var(--fw-card)` | `var(--fw-card)` | modules/button.css, modules/label.css |
|
||||||
|
| ⚠️ `--color-card` | `#f8f9fb` | `var(--fw-card) …` | `var(--fw-card) !importa…` | modules/card.css, repo/issue-card.css, user.css |
|
||||||
|
| ✅ `--color-caret` | `var(--color-text-dark)` | `var(--color-text-dark)` | `var(--color-text)` | base.css, codemirror/base.css |
|
||||||
|
| ⚠️ `--color-code-bg` | `#fafdff` | `var(--fw-card)` | `#0B0F14` | repo.css, repo/file-view.css |
|
||||||
|
| ⚠️ `--color-console-bg` | `#171b1e` | `#0F1419` | `#050910` | features/console.css |
|
||||||
|
| ⚠️ `--color-console-fg` | `#f7f8f9` | `#F7F5F0` | `#F7F8F9` | features/console.css |
|
||||||
|
| ⚠️ `--color-diff-added-linenum-bg` | `#d1f8d9` | `#CDEED6` | `#1E3C2A` | repo.css |
|
||||||
|
| ⚠️ `--color-diff-added-row-bg` | `#e6ffed` | `#E2F5E8` | `#18301E` | repo.css |
|
||||||
|
| ⚠️ `--color-diff-added-row-border` | `#e6ffed` | `#C2E0CB` | `#2A4A32` | repo.css |
|
||||||
|
| ⚠️ `--color-diff-added-word-bg` | `#acf2bd` | `#A8E0B7` | `#2E5C3E` | repo.css |
|
||||||
|
| ⚠️ `--color-diff-inactive` | `#f0f2f4` | `#EEECE7` | `#1A1F26` | repo.css |
|
||||||
|
| ⚠️ `--color-diff-moved-row-bg` | `#f1f8d1` | `#EEF4D8` | `#423E20` | repo.css |
|
||||||
|
| ⚠️ `--color-diff-removed-linenum-bg` | `#ffcecb` | `#F5C8C5` | `#402020` | repo.css |
|
||||||
|
| ⚠️ `--color-diff-removed-row-bg` | `#ffeef0` | `#FAE8E8` | `#2C1818` | repo.css |
|
||||||
|
| ⚠️ `--color-diff-removed-row-border` | `#f1c0c0` | `#EBBFBF` | `#5A3434` | repo.css |
|
||||||
|
| ⚠️ `--color-diff-removed-word-bg` | `#fdb8c0` | `#F3B5BC` | `#5C2D2D` | repo.css |
|
||||||
|
| ⚠️ `--color-error-bg` | `#fff6f6` | `#F6E5E4` | `#3C1F1F` | form.css, modules/header.css, modules/input.css |
|
||||||
|
| ⚠️ `--color-error-bg-active` | `#fbb` | `#E8ADAD` | `#5C2E2E` | form.css |
|
||||||
|
| ⚠️ `--color-error-bg-hover` | `#fdd` | `#F0C8C8` | `#4A2525` | form.css |
|
||||||
|
| ⚠️ `--color-error-border` | `#e0b4b4` | `#D48B8B` | `#8A3B3B` | form.css, modules/header.css, modules/input.css |
|
||||||
|
| ⚠️ `--color-error-text` | `#9f3a38` | `#8E2E2E` | `#F0B0B0` | form.css, modules/header.css, modules/input.css |
|
||||||
|
| ⚠️ `--color-expand-button` | `#cfe8fa` | `var(--color-primary-light-6)` | `#2A3845` | review.css |
|
||||||
|
| ✅ `--color-footer` | `var(--color-nav-bg)` | `var(--color-nav-bg)` | `var(--color-nav-bg)` | home.css |
|
||||||
|
| ⚠️ `--color-git` | `#f05133` | `#F05133` | `#F05133` | base.css |
|
||||||
|
| ⚠️ `--color-gold` | `#a1882b` | `var(--fw-chart-3)` | `var(--fw-chart-3)` | base.css |
|
||||||
|
| ⚠️ `--color-green` | `#21ba45` | `#247058` | `#4CB28A` | actions.css, base.css, features/imagediff.css |
|
||||||
|
| ⚠️ `--color-green-badge` | `#21ba45` | `var(--color-green)` | `var(--color-green)` | repo/commit-sign.css |
|
||||||
|
| ⚠️ `--color-green-badge-bg` | `#21ba451a` | `#2470581A` | `#4CB28A1A` | repo/commit-sign.css |
|
||||||
|
| ⚠️ `--color-green-badge-hover-bg` | `#21ba454d` | `#2470584D` | `#4CB28A4D` | repo/commit-sign.css |
|
||||||
|
| ⚠️ `--color-green-dark-1` | `#1ea73e` | `#1E5E49` | `#3A9876` | modules/button.css, modules/label.css |
|
||||||
|
| ⚠️ `--color-green-dark-2` | `#1a9537` | `#17483A` | `#2D7C5F` | modules/button.css |
|
||||||
|
| ⚠️ `--color-grey` | `#697077` | `var(--fw-muted-foreground)` | `var(--fw-muted-foreground)` | repo.css |
|
||||||
|
| ⚠️ `--color-grey-light` | `#7c838a` | `#8A8A87` | `#98948C` | base.css |
|
||||||
|
| ⚠️ `--color-highlight-bg` | `#fffbdd` | `#FFF3C4` | `#3A2E1A` | repo/file-view.css |
|
||||||
|
| ⚠️ `--color-highlight-fg` | `#eed200` | `var(--fw-chart-3)` | `#C9A347` | repo/file-view.css |
|
||||||
|
| ⚠️ `--color-hover` | `#00001708` | `#00001710` | `#E8F3FF14` | base.css, editor/fileeditor.css, helpers.css |
|
||||||
|
| ⚠️ `--color-hover-opaque` | `#f1f3f5` | `var(--fw-card-hover)` | `var(--fw-card-hover)` | repo/home-file-list.css |
|
||||||
|
| ⚠️ `--color-info-bg` | `#f8ffff` | `#E8F0F6` | `#1A2E44` | modules/message.css |
|
||||||
|
| ⚠️ `--color-info-border` | `#a9d5de` | `#A5BCCD` | `#3A6A8F` | modules/message.css |
|
||||||
|
| ⚠️ `--color-info-text` | `#276f86` | `var(--color-blue)` | `var(--color-blue)` | modules/message.css |
|
||||||
|
| ⚠️ `--color-input-background` | `#fff` | `var(--fw-card)` | `#12171E` | codemirror/base.css, form.css, markup/content.css |
|
||||||
|
| ✅ `--color-input-border` | `var(--color-secondary)` | `var(--color-secondary)` | `var(--color-secondary)` | form.css, modules/input.css |
|
||||||
|
| ⚠️ `--color-input-border-hover` | `var(--color-secondary-dark-1)` | `var(--color-secondary-dark-2)` | `var(--color-secondary-dark-1)` | form.css |
|
||||||
|
| ✅ `--color-input-text` | `var(--color-text-dark)` | `var(--color-text-dark)` | `var(--color-text-dark)` | codemirror/base.css, form.css, modules/input.css |
|
||||||
|
| ⚠️ `--color-input-toggle-background` | `#d0d7de` | `var(--fw-border)` | `#2A323D` | modules/checkbox.css |
|
||||||
|
| ⚠️ `--color-label-bg` | `#949da64b` | `#94908648` | `#6A7A8E4B` | base.css, modules/label.css, modules/menu.css |
|
||||||
|
| ⚠️ `--color-label-hover-bg` | `#949da6a0` | `#9490869E` | `#6A7A8EA0` | modules/label.css |
|
||||||
|
| ⚠️ `--color-label-text` | `var(--color-text)` | `var(--fw-foreground)` | `var(--fw-foreground)` | modules/label.css, modules/menu.css |
|
||||||
|
| ⚠️ `--color-light` | `#00001706` | `#EFECE5` | `#00001728` | base.css, modules/table.css, repo.css |
|
||||||
|
| ⚠️ `--color-light-border` | `#0000171d` | `#00001728` | `#E8F3FF28` | base.css, modules/button.css, modules/label.css |
|
||||||
|
| ⚠️ `--color-logo` | `#609926` | `var(--fw-primary)` | `var(--fw-primary)` | home.css |
|
||||||
|
| ✅ `--color-markup-code-block` | `#00306010` | `#00306010` | `#E8F3FF12` | markup/content.css, repo.css |
|
||||||
|
| ⚠️ `--color-markup-code-inline` | `#00306012` | `#00306014` | `#E8F3FF20` | markup/content.css |
|
||||||
|
| ⚠️ `--color-markup-table-row` | `#0030600a` | `#00306008` | `#E8F3FF0F` | markup/content.css |
|
||||||
|
| ⚠️ `--color-menu` | `#f8f9fb` | `var(--fw-card) …` | `var(--fw-card) !importa…` | base.css, features/expander.css, modules/menu.css |
|
||||||
|
| ⚠️ `--color-nav-bg` | `#f6f7fa` | `var(--fw-nav-bg) …` | `var(--fw-nav-bg) !importa…` | modules/navbar.css |
|
||||||
|
| ⚠️ `--color-nav-hover-bg` | `var(--color-secondary-light-1)` | `var(--fw-nav-hover-bg)` | `var(--fw-nav-hover-bg)` | modules/navbar.css |
|
||||||
|
| ⚠️ `--color-nav-text` | `var(--color-text)` | `var(--fw-nav-text) …` | `var(--fw-nav-text) !importa…` | modules/navbar.css |
|
||||||
|
| ⚠️ `--color-olive` | `#b5cc18` | `#8B9A4C` | `#ABC075` | modules/label.css |
|
||||||
|
| ⚠️ `--color-orange` | `#f2711c` | `var(--fw-chart-5)` | `var(--fw-chart-5)` | base.css, modules/label.css |
|
||||||
|
| ⚠️ `--color-orange-badge` | `#f2711c` | `var(--color-orange)` | `var(--color-orange)` | repo/commit-sign.css |
|
||||||
|
| ⚠️ `--color-orange-badge-bg` | `#f2711c1a` | `#C8702A1A` | `#E08A4A1A` | repo/commit-sign.css |
|
||||||
|
| ⚠️ `--color-orange-badge-hover-bg` | `#f2711c4d` | `#C8702A4D` | `#E08A4A4D` | repo/commit-sign.css |
|
||||||
|
| ⚠️ `--color-orange-dark-1` | `#e6630d` | `#B4621F` | `#C87239` | modules/label.css |
|
||||||
|
| ⚠️ `--color-overlay-backdrop` | `#080808c0` | `#000017C0` | `#000B17D0` | modules/dimmer.css |
|
||||||
|
| ✅ `--color-placeholder-text` | `var(--color-text-light-3)` | `var(--color-text-light-3)` | `var(--color-text-light-3)` | base.css, codemirror/base.css |
|
||||||
|
| ⚠️ `--color-primary` | `#4183c4` | `var(--fw-primary)` | `var(--fw-primary)` | base.css, codemirror/base.css, editor/combomarkdo… |
|
||||||
|
| ⚠️ `--color-primary-active` | `var(--color-primary-dark-2)` | `var(--color-primary-dark-1)` | `var(--color-primary-dark-2)` | helpers.css, modules/button.css |
|
||||||
|
| ⚠️ `--color-primary-alpha-30` | `#4183c44b` | `#1C2B3A4B` | `#5684A34B` | repo.css |
|
||||||
|
| ⚠️ `--color-primary-contrast` | `#ffffff` | `var(--fw-primary-foreground)` | `var(--fw-primary-foreground)` | base.css, features/expander.css, modules/button.c… |
|
||||||
|
| ⚠️ `--color-primary-dark-1` | `#3876b3` | `#172530` | `#6D97B3` | modules/label.css |
|
||||||
|
| ⚠️ `--color-primary-dark-2` | `#31699f` | `#121F28` | `#85ABC2` | modules/label.css |
|
||||||
|
| ⚠️ `--color-primary-dark-3` | `#2b5c8b` | `#0E1920` | `#9CBED1` | modules/label.css |
|
||||||
|
| ⚠️ `--color-primary-hover` | `var(--color-primary-dark-1)` | `var(--color-primary-light-1)` | `var(--color-primary-dark-1)` | modules/button.css |
|
||||||
|
| ⚠️ `--color-primary-light-1` | `#548fca` | `#2E4A63` | `#4A7590` | base.css, codemirror/base.css |
|
||||||
|
| ⚠️ `--color-primary-light-4` | `#8db5dc` | `#7BA0BC` | `#294557` | form.css, review.css |
|
||||||
|
| ⚠️ `--color-primary-light-5` | `#b3cde7` | `#A5BCCD` | `#203644` | review.css |
|
||||||
|
| ⚠️ `--color-primary-light-6` | `#d9e6f3` | `#D2DDE6` | `#162531` | form.css |
|
||||||
|
| ⚠️ `--color-primary-light-7` | `#f4f8fb` | `#EEF2F6` | `#0D1620` | base.css, features/imagediff.css |
|
||||||
|
| ✅ `--color-project-column-bg` | `var(--color-secondary-light-4)` | `var(--color-secondary-light-4)` | `var(--color-secondary-light-2)` | features/projects.css |
|
||||||
|
| ⚠️ `--color-purple` | `#a333c8` | `#8A4BA8` | `#B478D4` | base.css, modules/label.css |
|
||||||
|
| ⚠️ `--color-purple-dark-1` | `#932eb4` | `#6E388A` | `#9860B7` | modules/label.css |
|
||||||
|
| ⚠️ `--color-reaction-active-bg` | `var(--color-primary-light-6)` | `var(--color-primary-light-5)` | `var(--color-primary-light-5)` | repo/reactions.css |
|
||||||
|
| ⚠️ `--color-reaction-hover-bg` | `var(--color-primary-light-5)` | `var(--color-primary-light-6)` | `var(--color-primary-light-4)` | repo/reactions.css |
|
||||||
|
| ⚠️ `--color-red` | `#db2828` | `var(--fw-destructive)` | `var(--fw-destructive)` | base.css, dashboard.css, features/imagediff.css |
|
||||||
|
| ⚠️ `--color-red-badge` | `#db2828` | `var(--color-red)` | `var(--color-red)` | repo/commit-sign.css |
|
||||||
|
| ⚠️ `--color-red-badge-bg` | `#db28281a` | `#C0392B1A` | `#E07A6E1A` | repo/commit-sign.css |
|
||||||
|
| ⚠️ `--color-red-badge-hover-bg` | `#db28284d` | `#C0392B4D` | `#E07A6E4D` | repo/commit-sign.css |
|
||||||
|
| ⚠️ `--color-red-dark-1` | `#c82121` | `#A93024` | `#C86559` | base.css, modules/button.css, modules/label.css |
|
||||||
|
| ⚠️ `--color-red-dark-2` | `#b11e1e` | `#8D2720` | `#A74D42` | modules/button.css |
|
||||||
|
| ⚠️ `--color-red-light` | `#e45e5e` | `#D66A5C` | `#E89891` | actions.css |
|
||||||
|
| ⚠️ `--color-secondary` | `#d0d7de` | `var(--fw-border)` | `var(--fw-border)` | admin.css, base.css, codemirror/base.css |
|
||||||
|
| ⚠️ `--color-secondary-alpha-20` | `#d0d7de33` | `#B8B3A433` | `#3A445233` | repo.css |
|
||||||
|
| ⚠️ `--color-secondary-alpha-50` | `#d0d7de80` | `#B8B3A480` | `#3A445280` | modules/table.css, repo/issue-list.css |
|
||||||
|
| ⚠️ `--color-secondary-bg` | `#f2f5f8` | `var(--fw-secondary)` | `var(--fw-secondary)` | modules/modal.css, modules/segment.css |
|
||||||
|
| ⚠️ `--color-secondary-dark-1` | `#c7ced5` | `#A49F90` | `#45505E` | base.css, home.css, markup/codecopy.css |
|
||||||
|
| ⚠️ `--color-secondary-dark-2` | `#b9c0c7` | `#908B7D` | `#505C6A` | base.css, modules/button.css, modules/label.css |
|
||||||
|
| ⚠️ `--color-secondary-dark-4` | `#899097` | `#6A6558` | `#6E7A89` | features/projects.css, repo/issue-list.css |
|
||||||
|
| ⚠️ `--color-secondary-dark-5` | `#7a8188` | `#5A554A` | `#7E8A99` | features/gitgraph.css, form.css |
|
||||||
|
| ⚠️ `--color-secondary-dark-7` | `#5b6269` | `#3A372F` | `#9CA7B4` | base.css |
|
||||||
|
| ⚠️ `--color-secondary-dark-8` | `#4b5259` | `#2C2924` | `#ABB5C0` | features/imagediff.css, modules/animations.css |
|
||||||
|
| ⚠️ `--color-secondary-light-1` | `#dee5ec` | `#C7C3B6` | `#2F3845` | modules/card.css |
|
||||||
|
| ⚠️ `--color-secondary-nav-bg` | `#f9fafb` | `var(--color-secondary-light-4)` | `var(--color-secondary-light-3)` | modules/navbar.css |
|
||||||
|
| ✅ `--color-shadow` | `#00001726` | `#00001726` | `#00001758` | base.css, features/expander.css, modules/checkbox… |
|
||||||
|
| ✅ `--color-small-accent` | `var(--color-primary-light-6)` | `var(--color-primary-light-6)` | `var(--color-primary-light-4)` | review.css |
|
||||||
|
| ⚠️ `--color-success-bg` | `#fcfff5` | `#E8F2DD` | `#1F3A2A` | modules/message.css |
|
||||||
|
| ⚠️ `--color-success-border` | `#a3c293` | `#A3C293` | `#3F7D4F` | modules/message.css |
|
||||||
|
| ⚠️ `--color-success-text` | `#2c662d` | `#264F26` | `#7EC89A` | base.css, modules/message.css |
|
||||||
|
| ⚠️ `--color-teal` | `#00b5ad` | `#3A9A94` | `#66C7C0` | repo.css |
|
||||||
|
| ⚠️ `--color-text` | `#181c21` | `var(--fw-foreground) …` | `var(--fw-foreground) !importa…` | base.css, editor/combomarkdowneditor.css, feature… |
|
||||||
|
| ⚠️ `--color-text-dark` | `#01050a` | `#0D0D0C` | `#F5F2EA` | base.css, modules/list.css, modules/menu.css |
|
||||||
|
| ⚠️ `--color-text-light` | `#30363b` | `#2E2E2C` | `#CFCCC5` | actions.css, base.css, editor/fileeditor.css |
|
||||||
|
| ⚠️ `--color-text-light-1` | `#40474d` | `#454542` | `#B4B0A8` | base.css, features/expander.css, form.css |
|
||||||
|
| ⚠️ `--color-text-light-2` | `#5b6167` | `#595955` | `#9A968E` | base.css, markup/content.css, modules/breadcrumb.… |
|
||||||
|
| ⚠️ `--color-text-light-3` | `#747c84` | `#6E6E6A` | `#807C74` | base.css, features/gitgraph.css, repo.css |
|
||||||
|
| ⚠️ `--color-timeline` | `#d0d7de` | `var(--fw-border)` | `var(--fw-border)` | repo.css |
|
||||||
|
| ⚠️ `--color-tooltip-bg` | `#000017f0` | `#111B25F0` | `#000B17F0` | modules/tippy.css |
|
||||||
|
| ⚠️ `--color-tooltip-text` | `#fbfdff` | `var(--fw-primary-foreground)` | `#F5F2EA` | modules/tippy.css |
|
||||||
|
| ⚠️ `--color-violet-dark-1` | `#5a30b5` | `#55418A` | `#7E65BD` | base.css |
|
||||||
|
| ⚠️ `--color-warning-bg` | `#fffaf3` | `#F8F0DC` | `#3A3220` | modules/header.css, modules/message.css |
|
||||||
|
| ⚠️ `--color-warning-border` | `#c9ba9b` | `#C9BA9B` | `#8A7A30` | modules/header.css, modules/message.css, modules/… |
|
||||||
|
| ⚠️ `--color-warning-text` | `#573a08` | `#5A3D10` | `#E5BE5A` | base.css, modules/header.css, modules/message.css |
|
||||||
|
| ⚠️ `--color-white` | `#ffffff` | `#FFFFFF` | `#FFFFFF` | actions.css, base.css, codemirror/base.css |
|
||||||
|
| ⚠️ `--color-yellow` | `#fbbd08` | `#D4A017` | `#E5BE5A` | actions.css, base.css, modules/label.css |
|
||||||
|
| ⚠️ `--color-yellow-badge` | `#fbbd08` | `var(--color-yellow)` | `var(--color-yellow)` | repo/commit-sign.css |
|
||||||
|
| ⚠️ `--color-yellow-badge-bg` | `#fbbd081a` | `#D4A0171A` | `#E5BE5A1A` | repo/commit-sign.css |
|
||||||
|
| ⚠️ `--color-yellow-badge-hover-bg` | `#fbbd084d` | `#D4A0174D` | `#E5BE5A4D` | repo/commit-sign.css |
|
||||||
|
| ⚠️ `--color-yellow-dark-1` | `#e5ac04` | `#B38712` | `#C9A347` | modules/label.css |
|
||||||
|
| ❌ `--font-size-label` | `12px` | `—` | `—` | modules/label.css |
|
||||||
|
| ❌ `--font-weight-bold` | `700` | `—` | `—` | base.css, modules/navbar.css |
|
||||||
|
| ❌ `--font-weight-medium` | `500` | `—` | `—` | base.css, features/expander.css, modules/card.css |
|
||||||
|
| ❌ `--font-weight-normal` | `400` | `—` | `—` | base.css, features/expander.css, modules/button.c… |
|
||||||
|
| ❌ `--font-weight-semibold` | `600` | `—` | `—` | admin.css, base.css, chroma/base.css |
|
||||||
|
| ❌ `--fonts-default-override-ja` | `—` | `—` | `—` | font_i18n.css |
|
||||||
|
| ❌ `--fonts-default-override-ko` | `—` | `—` | `—` | font_i18n.css |
|
||||||
|
| ❌ `--fonts-default-override-zh-cn` | `—` | `—` | `—` | font_i18n.css |
|
||||||
|
| ❌ `--fonts-default-override-zh-hk` | `—` | `—` | `—` | font_i18n.css |
|
||||||
|
| ❌ `--fonts-default-override-zh-tw` | `—` | `—` | `—` | font_i18n.css |
|
||||||
|
| ❌ `--fonts-emoji` | `-emoji-fallback` | `—` | `—` | base.css, review.css |
|
||||||
|
| ⚠️ `--fonts-monospace` | `ui-monospace, SFMono-Regular,…` | `"JetBrains Mono", "Fira Code"…` | `"JetBrains Mono", "Fira Code"…` | base.css, codemirror/base.css, features/console.c… |
|
||||||
|
| ❌ `--fonts-override` | `—` | `—` | `"Source Serif 4", Georgia, se…` | base.css |
|
||||||
|
| ❌ `--fonts-proportional` | `-apple-system, "Segoe UI", sy…` | `—` | `—` | base.css, font_i18n.css |
|
||||||
|
| ⚠️ `--fonts-regular` | `var(--fonts-override, var(--f…` | `var(--fonts-override, var(--f…` | `—` | base.css, font_i18n.css, modules/button.css |
|
||||||
|
| ❌ `--gap-block` | `0.5rem` | `—` | `—` | base.css |
|
||||||
|
| ❌ `--gap-inline` | `0.25rem` | `—` | `—` | base.css, modules/label.css, repo.css |
|
||||||
|
| ❌ `--height-loading` | `16rem` | `—` | `—` | markup/content.css, modules/animations.css |
|
||||||
|
| ❌ `--line-height-default` | `normal` | `—` | `—` | admin.css, base.css, features/gitgraph.css |
|
||||||
|
| ❌ `--min-height-textarea` | `132px` | `—` | `—` | editor/combomarkdowneditor.css, form.css |
|
||||||
|
| ❌ `--octicon-chevron-right` | `url('data:image/svg+xml` | `—` | `—` | shared/settings.css |
|
||||||
|
| ❌ `--opacity-disabled` | `0.55` | `—` | `—` | form.css, modules/button.css, modules/input.css |
|
||||||
|
| ❌ `--page-margin-x` | `8px` | `—` | `—` | modules/container.css |
|
||||||
|
| ❌ `--page-space-bottom` | `64px` | `—` | `—` | base.css |
|
||||||
|
| ❌ `--page-spacing` | `16px` | `—` | `—` | base.css, modules/flexcontainer.css, repo/home.css |
|
||||||
|
| ❌ `--tab-size` | `4` | `—` | `—` | base.css |
|
||||||
|
| ❌ `--z-index-toast` | `1002` | `—` | `—` | modules/toast.css |
|
||||||
|
|
||||||
|
|
||||||
|
## Top selectors by var-reference count
|
||||||
|
|
||||||
|
- `modules/label.css` `/* based on Fomantic UI label module, with just the parts extracted th` — 6 vars: --border-radius, --color-label-bg, --color-label-text, --font-size-label, --font-weight-medium...
|
||||||
|
- `modules/button.css` `.ui.button` — 6 vars: --border-radius, --color-button, --color-light-border, --color-text, --font-weight-normal...
|
||||||
|
- `repo.css` `} .repository.file.editor .commit-form-wrapper .commit-form .quick-pul` — 4 vars: --border-radius, --color-secondary, --color-text, --fonts-monospace
|
||||||
|
- `codemirror/base.css` `.EasyMDEContainer .CodeMirror` — 4 vars: --color-input-background, --color-input-text, --color-secondary, --fonts-monospace
|
||||||
|
- `markup/content.css` `.markup kbd` — 4 vars: --border-radius, --color-markup-code-inline, --color-secondary, --color-text-light
|
||||||
|
- `repo/issue-card.css` `.issue-card` — 4 vars: --border-radius, --color-card, --color-secondary, --color-text
|
||||||
|
- `features/expander.css` `text-expander .suggestions, .tribute-container` — 4 vars: --border-radius, --color-menu, --color-secondary, --color-shadow
|
||||||
|
- `modules/menu.css` `.ui.menu` — 4 vars: --color-menu, --color-secondary, --font-weight-normal, --fonts-regular
|
||||||
|
- `modules/message.css` `/* based on Fomantic UI message module, with just the parts extracted ` — 4 vars: --border-radius, --color-box-body, --color-secondary, --color-text
|
||||||
|
- `modules/modal.css` `.ui.modal > .header` — 4 vars: --border-radius, --color-body, --color-secondary, --color-text-dark
|
||||||
|
- `modules/tippy.css` `.tippy-box` — 4 vars: --border-radius, --color-menu, --color-secondary, --color-text
|
||||||
|
- `modules/table.css` `.ui.table > thead > tr > th` — 4 vars: --color-box-header, --color-secondary, --color-text, --font-weight-normal
|
||||||
|
- `modules/table.css` `.ui.table > tfoot > tr > th, .ui.table > tfoot > tr > td` — 4 vars: --color-box-body, --color-secondary, --color-text, --font-weight-normal
|
||||||
|
- `base.css` `::file-selector-button` — 4 vars: --border-radius, --color-light, --color-light-border, --color-text-light
|
||||||
|
- `features/console.css` `/* Based on https://github.com/buildkite/terminal-to-html/blob/697ff23` — 4 vars: --border-radius, --color-console-bg, --color-console-fg, --fonts-monospace
|
||||||
|
- `modules/toast.css` `.toastify` — 4 vars: --border-radius, --color-shadow, --color-white, --z-index-toast
|
||||||
|
- `base.css` `body` — 4 vars: --color-body, --color-text, --fonts-regular, --tab-size
|
||||||
|
- `modules/input.css` `.ui.input > input` — 4 vars: --color-input-border, --color-input-text, --fonts-regular, --line-height-default
|
||||||
|
- `shared/flex-list.css` `.flex-item .flex-item-title` — 3 vars: --color-text, --font-weight-semibold, --gap-inline
|
||||||
|
- `user.css` `#readme_profile` — 3 vars: --border-radius, --color-card, --color-secondary
|
||||||
|
- `user.css` `#notification_table` — 3 vars: --border-radius, --color-box-body, --color-secondary
|
||||||
|
- `repo.css` `.repository.file.editor .commit-form-wrapper .commit-form` — 3 vars: --border-radius, --color-box-body, --color-secondary
|
||||||
|
- `repo.css` `.repository.view.issue .comment-list .comment .comment-container` — 3 vars: --border-radius, --color-box-body, --color-secondary
|
||||||
|
- `repo.css` `.comment-header` — 3 vars: --color-box-header, --color-secondary, --color-text
|
||||||
|
- `repo.css` `.resolved-placeholder` — 3 vars: --border-radius, --color-box-header, --color-secondary
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Per-file source CSS consumption summary
|
||||||
|
|
||||||
|
For each Gitea source CSS file, the `--color-*` variables it consumes
|
||||||
|
(top 10 by reference count). Use this to know which file to read when a
|
||||||
|
specific variable misbehaves.
|
||||||
|
|
||||||
|
|
||||||
|
### `actions.css` (11 refs across 6 unique vars)
|
||||||
|
- `--color-white` × 6
|
||||||
|
- `--color-green` × 1
|
||||||
|
- `--color-red-light` × 1
|
||||||
|
- `--color-blue` × 1
|
||||||
|
- `--color-yellow` × 1
|
||||||
|
- `--color-text-light` × 1
|
||||||
|
|
||||||
|
### `admin.css` (1 refs across 1 unique vars)
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
|
||||||
|
### `base.css` (104 refs across 40 unique vars)
|
||||||
|
- `--color-text` × 14
|
||||||
|
- `--color-secondary` × 13
|
||||||
|
- `--color-text-light-2` × 10
|
||||||
|
- `--color-primary` × 7
|
||||||
|
- `--color-hover` × 6
|
||||||
|
- `--color-text-light` × 4
|
||||||
|
- `--color-body` × 3
|
||||||
|
- `--color-label-bg` × 2
|
||||||
|
- `--color-secondary-dark-1` × 2
|
||||||
|
- `--color-accent` × 2
|
||||||
|
|
||||||
|
### `codemirror/base.css` (9 refs across 8 unique vars)
|
||||||
|
- `--color-primary` × 2
|
||||||
|
- `--color-input-text` × 1
|
||||||
|
- `--color-input-background` × 1
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
- `--color-caret` × 1
|
||||||
|
- `--color-primary-light-1` × 1
|
||||||
|
- `--color-white` × 1
|
||||||
|
- `--color-placeholder-text` × 1
|
||||||
|
|
||||||
|
### `dashboard.css` (1 refs across 1 unique vars)
|
||||||
|
- `--color-red` × 1
|
||||||
|
|
||||||
|
### `editor/combomarkdowneditor.css` (4 refs across 3 unique vars)
|
||||||
|
- `--color-secondary` × 2
|
||||||
|
- `--color-text` × 1
|
||||||
|
- `--color-primary` × 1
|
||||||
|
|
||||||
|
### `editor/fileeditor.css` (9 refs across 5 unique vars)
|
||||||
|
- `--color-secondary` × 3
|
||||||
|
- `--color-body` × 2
|
||||||
|
- `--color-text-light` × 2
|
||||||
|
- `--color-hover` × 1
|
||||||
|
- `--color-active` × 1
|
||||||
|
|
||||||
|
### `explore.css` (1 refs across 1 unique vars)
|
||||||
|
- `--color-text-light` × 1
|
||||||
|
|
||||||
|
### `features/codeeditor.css` (1 refs across 1 unique vars)
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
|
||||||
|
### `features/console.css` (55 refs across 19 unique vars)
|
||||||
|
- `--color-ansi-bright-black` × 5
|
||||||
|
- `--color-ansi-red` × 4
|
||||||
|
- `--color-ansi-green` × 4
|
||||||
|
- `--color-ansi-bright-red` × 4
|
||||||
|
- `--color-ansi-bright-green` × 4
|
||||||
|
- `--color-ansi-black` × 3
|
||||||
|
- `--color-ansi-yellow` × 3
|
||||||
|
- `--color-ansi-blue` × 3
|
||||||
|
- `--color-ansi-magenta` × 3
|
||||||
|
- `--color-ansi-cyan` × 3
|
||||||
|
|
||||||
|
### `features/dropzone.css` (3 refs across 3 unique vars)
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
- `--color-text-light` × 1
|
||||||
|
- `--color-text` × 1
|
||||||
|
|
||||||
|
### `features/expander.css` (7 refs across 6 unique vars)
|
||||||
|
- `--color-secondary` × 2
|
||||||
|
- `--color-menu` × 1
|
||||||
|
- `--color-shadow` × 1
|
||||||
|
- `--color-text-light-1` × 1
|
||||||
|
- `--color-primary` × 1
|
||||||
|
- `--color-primary-contrast` × 1
|
||||||
|
|
||||||
|
### `features/gitgraph.css` (4 refs across 3 unique vars)
|
||||||
|
- `--color-secondary-dark-5` × 2
|
||||||
|
- `--color-text-light` × 1
|
||||||
|
- `--color-text-light-3` × 1
|
||||||
|
|
||||||
|
### `features/heatmap.css` (1 refs across 1 unique vars)
|
||||||
|
- `--color-text` × 1
|
||||||
|
|
||||||
|
### `features/imagediff.css` (5 refs across 4 unique vars)
|
||||||
|
- `--color-secondary-dark-8` × 2
|
||||||
|
- `--color-primary-light-7` × 1
|
||||||
|
- `--color-red` × 1
|
||||||
|
- `--color-green` × 1
|
||||||
|
|
||||||
|
### `features/projects.css` (3 refs across 3 unique vars)
|
||||||
|
- `--color-project-column-bg` × 1
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
- `--color-secondary-dark-4` × 1
|
||||||
|
|
||||||
|
### `form.css` (24 refs across 16 unique vars)
|
||||||
|
- `--color-input-text` × 4
|
||||||
|
- `--color-error-border` × 3
|
||||||
|
- `--color-input-background` × 3
|
||||||
|
- `--color-text` × 2
|
||||||
|
- `--color-error-bg` × 1
|
||||||
|
- `--color-error-text` × 1
|
||||||
|
- `--color-error-bg-hover` × 1
|
||||||
|
- `--color-error-bg-active` × 1
|
||||||
|
- `--color-primary-light-6` × 1
|
||||||
|
- `--color-primary-light-4` × 1
|
||||||
|
|
||||||
|
### `helpers.css` (4 refs across 4 unique vars)
|
||||||
|
- `--color-primary` × 1
|
||||||
|
- `--color-primary-active` × 1
|
||||||
|
- `--color-hover` × 1
|
||||||
|
- `--color-active` × 1
|
||||||
|
|
||||||
|
### `home.css` (5 refs across 4 unique vars)
|
||||||
|
- `--color-logo` × 2
|
||||||
|
- `--color-footer` × 1
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
- `--color-secondary-dark-1` × 1
|
||||||
|
|
||||||
|
### `install.css` (2 refs across 2 unique vars)
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
- `--color-red` × 1
|
||||||
|
|
||||||
|
### `markup/codecopy.css` (2 refs across 2 unique vars)
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
- `--color-secondary-dark-1` × 1
|
||||||
|
|
||||||
|
### `markup/codepreview.css` (3 refs across 2 unique vars)
|
||||||
|
- `--color-secondary` × 2
|
||||||
|
- `--color-text-light-1` × 1
|
||||||
|
|
||||||
|
### `markup/content.css` (23 refs across 11 unique vars)
|
||||||
|
- `--color-secondary` × 10
|
||||||
|
- `--color-text-light-2` × 2
|
||||||
|
- `--color-text` × 2
|
||||||
|
- `--color-markup-code-inline` × 2
|
||||||
|
- `--color-red` × 1
|
||||||
|
- `--color-input-background` × 1
|
||||||
|
- `--color-primary` × 1
|
||||||
|
- `--color-markup-table-row` × 1
|
||||||
|
- `--color-box-body` × 1
|
||||||
|
- `--color-markup-code-block` × 1
|
||||||
|
|
||||||
|
### `modules/animations.css` (4 refs across 2 unique vars)
|
||||||
|
- `--color-secondary` × 2
|
||||||
|
- `--color-secondary-dark-8` × 2
|
||||||
|
|
||||||
|
### `modules/breadcrumb.css` (1 refs across 1 unique vars)
|
||||||
|
- `--color-text-light-2` × 1
|
||||||
|
|
||||||
|
### `modules/button.css` (61 refs across 19 unique vars)
|
||||||
|
- `--color-light-border` × 6
|
||||||
|
- `--color-text` × 5
|
||||||
|
- `--color-secondary-dark-2` × 4
|
||||||
|
- `--color-green` × 4
|
||||||
|
- `--color-white` × 4
|
||||||
|
- `--color-green-dark-1` × 4
|
||||||
|
- `--color-red` × 4
|
||||||
|
- `--color-hover` × 3
|
||||||
|
- `--color-primary` × 3
|
||||||
|
- `--color-primary-hover` × 3
|
||||||
|
|
||||||
|
### `modules/card.css` (13 refs across 6 unique vars)
|
||||||
|
- `--color-text` × 5
|
||||||
|
- `--color-secondary` × 3
|
||||||
|
- `--color-card` × 2
|
||||||
|
- `--color-primary` × 1
|
||||||
|
- `--color-text-light-2` × 1
|
||||||
|
- `--color-secondary-light-1` × 1
|
||||||
|
|
||||||
|
### `modules/checkbox.css` (5 refs across 5 unique vars)
|
||||||
|
- `--color-white` × 1
|
||||||
|
- `--color-shadow` × 1
|
||||||
|
- `--color-input-toggle-background` × 1
|
||||||
|
- `--color-text` × 1
|
||||||
|
- `--color-primary` × 1
|
||||||
|
|
||||||
|
### `modules/dimmer.css` (1 refs across 1 unique vars)
|
||||||
|
- `--color-overlay-backdrop` × 1
|
||||||
|
|
||||||
|
### `modules/divider.css` (4 refs across 2 unique vars)
|
||||||
|
- `--color-secondary` × 3
|
||||||
|
- `--color-text` × 1
|
||||||
|
|
||||||
|
### `modules/header.css` (12 refs across 10 unique vars)
|
||||||
|
- `--color-text` × 2
|
||||||
|
- `--color-secondary` × 2
|
||||||
|
- `--color-text-light-1` × 1
|
||||||
|
- `--color-box-header` × 1
|
||||||
|
- `--color-error-bg` × 1
|
||||||
|
- `--color-error-text` × 1
|
||||||
|
- `--color-error-border` × 1
|
||||||
|
- `--color-warning-bg` × 1
|
||||||
|
- `--color-warning-text` × 1
|
||||||
|
- `--color-warning-border` × 1
|
||||||
|
|
||||||
|
### `modules/input.css` (13 refs across 6 unique vars)
|
||||||
|
- `--color-input-border` × 3
|
||||||
|
- `--color-primary` × 3
|
||||||
|
- `--color-error-border` × 3
|
||||||
|
- `--color-input-text` × 2
|
||||||
|
- `--color-error-bg` × 1
|
||||||
|
- `--color-error-text` × 1
|
||||||
|
|
||||||
|
### `modules/label.css` (85 refs across 26 unique vars)
|
||||||
|
- `--color-white` × 12
|
||||||
|
- `--color-label-hover-bg` × 6
|
||||||
|
- `--color-label-bg` × 4
|
||||||
|
- `--color-label-text` × 4
|
||||||
|
- `--color-red` × 4
|
||||||
|
- `--color-red-dark-1` × 4
|
||||||
|
- `--color-orange` × 4
|
||||||
|
- `--color-orange-dark-1` × 4
|
||||||
|
- `--color-yellow` × 4
|
||||||
|
- `--color-yellow-dark-1` × 4
|
||||||
|
|
||||||
|
### `modules/list.css` (5 refs across 3 unique vars)
|
||||||
|
- `--color-text` × 2
|
||||||
|
- `--color-text-dark` × 2
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
|
||||||
|
### `modules/menu.css` (49 refs across 13 unique vars)
|
||||||
|
- `--color-text` × 13
|
||||||
|
- `--color-secondary` × 9
|
||||||
|
- `--color-active` × 7
|
||||||
|
- `--color-hover` × 4
|
||||||
|
- `--color-menu` × 3
|
||||||
|
- `--color-text-light-2` × 3
|
||||||
|
- `--color-text-dark` × 3
|
||||||
|
- `--color-body` × 2
|
||||||
|
- `--color-box-header` × 1
|
||||||
|
- `--color-label-bg` × 1
|
||||||
|
|
||||||
|
### `modules/message.css` (19 refs across 19 unique vars)
|
||||||
|
- `--color-box-body` × 1
|
||||||
|
- `--color-text` × 1
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
- `--color-blue` × 1
|
||||||
|
- `--color-info-bg` × 1
|
||||||
|
- `--color-info-text` × 1
|
||||||
|
- `--color-info-border` × 1
|
||||||
|
- `--color-green` × 1
|
||||||
|
- `--color-success-bg` × 1
|
||||||
|
- `--color-success-text` × 1
|
||||||
|
|
||||||
|
### `modules/modal.css` (10 refs across 5 unique vars)
|
||||||
|
- `--color-body` × 3
|
||||||
|
- `--color-text-dark` × 2
|
||||||
|
- `--color-secondary` × 2
|
||||||
|
- `--color-shadow` × 2
|
||||||
|
- `--color-secondary-bg` × 1
|
||||||
|
|
||||||
|
### `modules/navbar.css` (10 refs across 7 unique vars)
|
||||||
|
- `--color-nav-bg` × 3
|
||||||
|
- `--color-nav-hover-bg` × 2
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
- `--color-nav-text` × 1
|
||||||
|
- `--color-active` × 1
|
||||||
|
- `--color-primary` × 1
|
||||||
|
- `--color-secondary-nav-bg` × 1
|
||||||
|
|
||||||
|
### `modules/segment.css` (17 refs across 7 unique vars)
|
||||||
|
- `--color-secondary` × 7
|
||||||
|
- `--color-box-body` × 3
|
||||||
|
- `--color-text` × 3
|
||||||
|
- `--color-secondary-bg` × 1
|
||||||
|
- `--color-text-light` × 1
|
||||||
|
- `--color-error-border` × 1
|
||||||
|
- `--color-warning-border` × 1
|
||||||
|
|
||||||
|
### `modules/table.css` (19 refs across 7 unique vars)
|
||||||
|
- `--color-secondary` × 7
|
||||||
|
- `--color-text` × 3
|
||||||
|
- `--color-box-body` × 2
|
||||||
|
- `--color-secondary-alpha-50` × 2
|
||||||
|
- `--color-hover` × 2
|
||||||
|
- `--color-light` × 2
|
||||||
|
- `--color-box-header` × 1
|
||||||
|
|
||||||
|
### `modules/tippy.css` (21 refs across 10 unique vars)
|
||||||
|
- `--color-menu` × 4
|
||||||
|
- `--color-shadow` × 4
|
||||||
|
- `--color-text` × 2
|
||||||
|
- `--color-secondary` × 2
|
||||||
|
- `--color-tooltip-bg` × 2
|
||||||
|
- `--color-hover` × 2
|
||||||
|
- `--color-box-body` × 2
|
||||||
|
- `--color-tooltip-text` × 1
|
||||||
|
- `--color-active` × 1
|
||||||
|
- `--color-box-header` × 1
|
||||||
|
|
||||||
|
### `modules/toast.css` (4 refs across 4 unique vars)
|
||||||
|
- `--color-white` × 1
|
||||||
|
- `--color-shadow` × 1
|
||||||
|
- `--color-hover` × 1
|
||||||
|
- `--color-active` × 1
|
||||||
|
|
||||||
|
### `org.css` (2 refs across 2 unique vars)
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
- `--color-box-body` × 1
|
||||||
|
|
||||||
|
### `repo/clone.css` (2 refs across 2 unique vars)
|
||||||
|
- `--color-text-light-2` × 1
|
||||||
|
- `--color-text-dark` × 1
|
||||||
|
|
||||||
|
### `repo/commit-sign.css` (13 refs across 13 unique vars)
|
||||||
|
- `--color-light-border` × 1
|
||||||
|
- `--color-green-badge` × 1
|
||||||
|
- `--color-green-badge-bg` × 1
|
||||||
|
- `--color-green-badge-hover-bg` × 1
|
||||||
|
- `--color-yellow-badge` × 1
|
||||||
|
- `--color-yellow-badge-bg` × 1
|
||||||
|
- `--color-yellow-badge-hover-bg` × 1
|
||||||
|
- `--color-orange-badge` × 1
|
||||||
|
- `--color-orange-badge-bg` × 1
|
||||||
|
- `--color-orange-badge-hover-bg` × 1
|
||||||
|
|
||||||
|
### `repo/file-view.css` (6 refs across 5 unique vars)
|
||||||
|
- `--color-secondary` × 2
|
||||||
|
- `--color-highlight-bg` × 1
|
||||||
|
- `--color-highlight-fg` × 1
|
||||||
|
- `--color-code-bg` × 1
|
||||||
|
- `--color-text-dark` × 1
|
||||||
|
|
||||||
|
### `repo/home-file-list.css` (7 refs across 5 unique vars)
|
||||||
|
- `--color-secondary` × 2
|
||||||
|
- `--color-text-light-1` × 2
|
||||||
|
- `--color-box-body` × 1
|
||||||
|
- `--color-hover-opaque` × 1
|
||||||
|
- `--color-box-header` × 1
|
||||||
|
|
||||||
|
### `repo/home.css` (1 refs across 1 unique vars)
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
|
||||||
|
### `repo/issue-card.css` (3 refs across 3 unique vars)
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
- `--color-card` × 1
|
||||||
|
- `--color-text` × 1
|
||||||
|
|
||||||
|
### `repo/issue-label.css` (1 refs across 1 unique vars)
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
|
||||||
|
### `repo/issue-list.css` (3 refs across 2 unique vars)
|
||||||
|
- `--color-secondary-dark-4` × 2
|
||||||
|
- `--color-secondary-alpha-50` × 1
|
||||||
|
|
||||||
|
### `repo/reactions.css` (2 refs across 2 unique vars)
|
||||||
|
- `--color-reaction-active-bg` × 1
|
||||||
|
- `--color-reaction-hover-bg` × 1
|
||||||
|
|
||||||
|
### `repo/release-tag.css` (2 refs across 2 unique vars)
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
- `--color-text-light-1` × 1
|
||||||
|
|
||||||
|
### `repo/wiki.css` (3 refs across 2 unique vars)
|
||||||
|
- `--color-secondary` × 2
|
||||||
|
- `--color-hover` × 1
|
||||||
|
|
||||||
|
### `repo.css` (98 refs across 37 unique vars)
|
||||||
|
- `--color-secondary` × 18
|
||||||
|
- `--color-text` × 8
|
||||||
|
- `--color-red` × 5
|
||||||
|
- `--color-primary` × 5
|
||||||
|
- `--color-box-body` × 5
|
||||||
|
- `--color-box-header` × 5
|
||||||
|
- `--color-body` × 4
|
||||||
|
- `--color-yellow` × 3
|
||||||
|
- `--color-text-light-2` × 3
|
||||||
|
- `--color-timeline` × 3
|
||||||
|
|
||||||
|
### `review.css` (14 refs across 11 unique vars)
|
||||||
|
- `--color-primary-contrast` × 2
|
||||||
|
- `--color-accent` × 2
|
||||||
|
- `--color-small-accent` × 2
|
||||||
|
- `--color-red` × 1
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
- `--color-text` × 1
|
||||||
|
- `--color-text-light` × 1
|
||||||
|
- `--color-expand-button` × 1
|
||||||
|
- `--color-primary` × 1
|
||||||
|
- `--color-primary-light-4` × 1
|
||||||
|
|
||||||
|
### `shared/flex-list.css` (5 refs across 4 unique vars)
|
||||||
|
- `--color-text` × 2
|
||||||
|
- `--color-primary` × 1
|
||||||
|
- `--color-text-light-2` × 1
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
|
||||||
|
### `shared/milestone.css` (4 refs across 3 unique vars)
|
||||||
|
- `--color-text-light-2` × 2
|
||||||
|
- `--color-secondary` × 1
|
||||||
|
- `--color-text` × 1
|
||||||
|
|
||||||
|
### `shared/settings.css` (1 refs across 1 unique vars)
|
||||||
|
- `--color-body` × 1
|
||||||
|
|
||||||
|
### `user.css` (6 refs across 4 unique vars)
|
||||||
|
- `--color-secondary` × 3
|
||||||
|
- `--color-card` × 1
|
||||||
|
- `--color-box-body` × 1
|
||||||
|
- `--color-hover` × 1
|
||||||
96
infra/gitea/Dockerfile
Normal file
@@ -0,0 +1,96 @@
|
|||||||
|
# syntax=docker/dockerfile:1.7
|
||||||
|
# =============================================================================
|
||||||
|
# Gitea with fhirworx theme compiled in.
|
||||||
|
#
|
||||||
|
# Clones go-gitea/gitea at the pinned tag, drops our theme CSS into
|
||||||
|
# web_src/css/themes/, then runs Gitea's own webpack+go build so the theme is
|
||||||
|
# baked into `bindata`. Runtime stage is the same alpine+dumb-init layout as
|
||||||
|
# the upstream rootless image.
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
ARG GITEA_VERSION=v1.25.4
|
||||||
|
# BUILD_TAG must be unique per build; defaults to current epoch when unset.
|
||||||
|
# Gitea uses main.Version to construct the `?v=...` query string on every
|
||||||
|
# asset URL. Without changing it, browsers cache theme-fhirworx.css for 6h
|
||||||
|
# (Cache-Control: max-age=21600) — so theme edits never reach the user
|
||||||
|
# until the URL key changes. Setting it per-build forces every cache layer
|
||||||
|
# (browser, CDN, Cloudflare) to refetch.
|
||||||
|
ARG BUILD_TAG
|
||||||
|
|
||||||
|
# ---- build stage ------------------------------------------------------------
|
||||||
|
FROM docker.io/library/golang:1.25-alpine3.22 AS build
|
||||||
|
ARG GITEA_VERSION
|
||||||
|
ARG BUILD_TAG
|
||||||
|
ENV GOPROXY=https://proxy.golang.org,direct \
|
||||||
|
GOSUMDB=sum.golang.org \
|
||||||
|
TAGS="bindata timetzdata sqlite sqlite_unlock_notify" \
|
||||||
|
CGO_ENABLED=1
|
||||||
|
|
||||||
|
RUN apk add --no-cache build-base git nodejs npm \
|
||||||
|
&& npm install -g pnpm@10
|
||||||
|
|
||||||
|
WORKDIR /src
|
||||||
|
RUN git clone --depth 1 --branch ${GITEA_VERSION} \
|
||||||
|
https://github.com/go-gitea/gitea.git . \
|
||||||
|
&& git log -1 --format='%H %s'
|
||||||
|
|
||||||
|
# Drop fhirworx theme into Gitea's theme dir before the frontend build so
|
||||||
|
# webpack/bindata pick it up.
|
||||||
|
COPY theme/theme-fhirworx.css web_src/css/themes/theme-fhirworx.css
|
||||||
|
COPY theme/theme-fhirworx-dark.css web_src/css/themes/theme-fhirworx-dark.css
|
||||||
|
|
||||||
|
# Replace Gitea's source SVGs with fhirworx branding. tools/generate-images.ts
|
||||||
|
# reads these to produce logo.svg/.png, favicon.svg/.png, apple-touch-icon.png,
|
||||||
|
# avatar_default.png — all baked into bindata.
|
||||||
|
COPY brand/logo.svg assets/logo.svg
|
||||||
|
COPY brand/favicon.svg assets/favicon.svg
|
||||||
|
|
||||||
|
# Full build. Order matters: deps → generate-images (uses our brand SVGs) →
|
||||||
|
# webpack frontend → go binary with bindata embedding everything in public/.
|
||||||
|
RUN --mount=type=cache,target=/root/.cache/go-build \
|
||||||
|
--mount=type=cache,target=/go/pkg/mod \
|
||||||
|
--mount=type=cache,target=/root/.local/share/pnpm/store \
|
||||||
|
make deps-frontend \
|
||||||
|
&& make generate-images \
|
||||||
|
&& BUILD_TAG="${BUILD_TAG:-$(date +%s)}" \
|
||||||
|
&& export LDFLAGS="-X 'main.Version=${GITEA_VERSION}-fhirworx.${BUILD_TAG}'" \
|
||||||
|
&& echo "Building with version: ${GITEA_VERSION}-fhirworx.${BUILD_TAG}" \
|
||||||
|
&& make clean-all build LDFLAGS="${LDFLAGS}" \
|
||||||
|
&& go build contrib/environment-to-ini/environment-to-ini.go
|
||||||
|
|
||||||
|
# Upstream rootless overlay files (entrypoint, setup, gitea wrapper).
|
||||||
|
RUN chmod 755 docker/rootless/usr/local/bin/docker-entrypoint.sh \
|
||||||
|
docker/rootless/usr/local/bin/docker-setup.sh \
|
||||||
|
docker/rootless/usr/local/bin/gitea \
|
||||||
|
/src/gitea \
|
||||||
|
/src/environment-to-ini
|
||||||
|
|
||||||
|
# ---- runtime stage ----------------------------------------------------------
|
||||||
|
FROM docker.io/library/alpine:3.22
|
||||||
|
LABEL org.opencontainers.image.source="https://github.com/go-gitea/gitea"
|
||||||
|
LABEL fhirworx.theme="fhirworx"
|
||||||
|
|
||||||
|
EXPOSE 2222 3000
|
||||||
|
|
||||||
|
RUN apk add --no-cache bash ca-certificates dumb-init gettext git curl gnupg openssh-keygen \
|
||||||
|
&& addgroup -S -g 1000 git \
|
||||||
|
&& adduser -S -H -D -h /var/lib/gitea/git -s /bin/bash -u 1000 -G git git \
|
||||||
|
&& mkdir -p /var/lib/gitea /etc/gitea \
|
||||||
|
&& chown git:git /var/lib/gitea /etc/gitea
|
||||||
|
|
||||||
|
COPY --from=build /src/docker/rootless /
|
||||||
|
COPY --from=build --chown=root:root /src/gitea /app/gitea/gitea
|
||||||
|
COPY --from=build --chown=root:root /src/environment-to-ini /usr/local/bin/environment-to-ini
|
||||||
|
|
||||||
|
USER 1000:1000
|
||||||
|
ENV GITEA_WORK_DIR=/var/lib/gitea \
|
||||||
|
GITEA_CUSTOM=/var/lib/gitea/custom \
|
||||||
|
GITEA_TEMP=/tmp/gitea \
|
||||||
|
TMPDIR=/tmp/gitea \
|
||||||
|
GITEA_APP_INI=/etc/gitea/app.ini \
|
||||||
|
HOME=/var/lib/gitea/git
|
||||||
|
|
||||||
|
VOLUME ["/var/lib/gitea", "/etc/gitea"]
|
||||||
|
WORKDIR /var/lib/gitea
|
||||||
|
ENTRYPOINT ["/usr/bin/dumb-init", "--", "/usr/local/bin/docker-entrypoint.sh"]
|
||||||
|
CMD []
|
||||||
132
infra/gitea/README.md
Normal file
@@ -0,0 +1,132 @@
|
|||||||
|
# Fhirworx Gitea image
|
||||||
|
|
||||||
|
This directory builds `fhirworx/gitea:<tag>` — upstream `go-gitea/gitea` at a
|
||||||
|
pinned tag with the fhirworx theme baked in via `bindata`. **No source
|
||||||
|
changes to upstream**; we only inject one CSS file and two brand SVGs into
|
||||||
|
the upstream tree before its own build runs.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
infra/gitea/
|
||||||
|
├── Dockerfile # multi-stage build, see below
|
||||||
|
├── .dockerignore
|
||||||
|
├── theme/
|
||||||
|
│ └── theme-fhirworx.css # ONE css file = the entire theme
|
||||||
|
├── brand/
|
||||||
|
│ ├── logo.svg # → assets/logo.svg → make generate-images
|
||||||
|
│ └── favicon.svg # → assets/favicon.svg
|
||||||
|
├── custom/ # bind-mounted at runtime as /var/lib/gitea/custom
|
||||||
|
│ └── templates/
|
||||||
|
│ └── home.tmpl # custom anonymous landing page
|
||||||
|
└── README.md # this file
|
||||||
|
```
|
||||||
|
|
||||||
|
## Staying downstream from upstream Gitea
|
||||||
|
|
||||||
|
Goal: track upstream cleanly, never fork the source.
|
||||||
|
|
||||||
|
The build does only three things to the upstream tree:
|
||||||
|
|
||||||
|
1. `COPY theme/theme-fhirworx.css web_src/css/themes/theme-fhirworx.css`
|
||||||
|
2. `COPY brand/logo.svg assets/logo.svg`
|
||||||
|
3. `COPY brand/favicon.svg assets/favicon.svg`
|
||||||
|
|
||||||
|
After these, the upstream `make clean-all build` runs unmodified — webpack
|
||||||
|
processes our theme into `public/assets/css/theme-fhirworx.css`, the image
|
||||||
|
generator regenerates every PNG/SVG variant from our SVGs, and `bindata`
|
||||||
|
embeds the whole `public/` tree into the Go binary.
|
||||||
|
|
||||||
|
### Bumping Gitea
|
||||||
|
|
||||||
|
```
|
||||||
|
# 1. Update the pinned tag
|
||||||
|
sed -i 's/GITEA_VERSION=v1\.[0-9.]\+/GITEA_VERSION=v1.NEW.VER/' Dockerfile
|
||||||
|
# 2. Diff our theme against upstream's reference
|
||||||
|
diff theme/theme-fhirworx.css \
|
||||||
|
<(curl -sL https://raw.githubusercontent.com/go-gitea/gitea/v1.NEW.VER/web_src/css/themes/theme-gitea-light.css)
|
||||||
|
# 3. Add any new --color-* vars upstream introduced
|
||||||
|
# 4. Rebuild
|
||||||
|
docker compose build gitea && docker compose up -d gitea
|
||||||
|
```
|
||||||
|
|
||||||
|
The theme file's variable order **mirrors upstream's `theme-gitea-light.css`
|
||||||
|
1:1**, on purpose, so step 2 produces a clean readable diff. Add variables
|
||||||
|
upstream added; refresh values you've remapped.
|
||||||
|
|
||||||
|
## Theme contract
|
||||||
|
|
||||||
|
The theme is a complete drop-in replacement for upstream's
|
||||||
|
`theme-gitea-light.css`. It defines:
|
||||||
|
|
||||||
|
- **All ~140 `--color-*` variables** Gitea references. Any var left undefined
|
||||||
|
resolves to CSS `initial` (transparent bg / black text), which breaks
|
||||||
|
surfaces like the navbar, secondary-nav, footer, clone panel, menu hover
|
||||||
|
states. Historical bug: an early version defined ~30 vars and many
|
||||||
|
components broke.
|
||||||
|
|
||||||
|
- **`--fonts-override`** — Gitea's `base.css` composes
|
||||||
|
`--fonts-regular: var(--fonts-override, var(--fonts-proportional)), ...`.
|
||||||
|
Setting `--fonts-override` propagates the editorial type (Source Serif 4)
|
||||||
|
through every Fomantic UI component (menus, buttons, tabs, inputs).
|
||||||
|
|
||||||
|
- **A defensive `#navbar` color sweep**. Fhirworx is the only design
|
||||||
|
(compared against awesome-gitea's full theme list — Catppuccin, Rainnny
|
||||||
|
GitHub, lutinglt, Earl Grey, Dark Arc, etc.) that puts a *dark* navbar
|
||||||
|
over a *light* body. Every reference theme keeps both surfaces in the
|
||||||
|
same luminance class. Because of that, Gitea's base CSS doesn't anticipate
|
||||||
|
the inversion: any Fomantic class with `color: var(--color-text)` (e.g.
|
||||||
|
`.ui.button` for the hamburger `#navbar-expand-toggle`) leaks near-black
|
||||||
|
text into the dark navbar. The sweep at the end of theme-fhirworx.css
|
||||||
|
forces nav-text on every text/icon element inside `#navbar`, and flips
|
||||||
|
dropdown popouts back to the light body palette since they float over
|
||||||
|
the page, not the bar.
|
||||||
|
|
||||||
|
## Custom templates (still bind-mounted)
|
||||||
|
|
||||||
|
`custom/templates/home.tmpl` is the anonymous landing page. It overrides
|
||||||
|
upstream's stock dashboard for unauthenticated visitors and renders the
|
||||||
|
homelab service grid. It's bind-mounted via `compose.yml`, not baked in,
|
||||||
|
because it's content not theme — easier to edit without rebuilding.
|
||||||
|
|
||||||
|
## Compose wiring
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
gitea:
|
||||||
|
build:
|
||||||
|
context: ./infra/gitea
|
||||||
|
args:
|
||||||
|
GITEA_VERSION: v1.25.4
|
||||||
|
image: fhirworx/gitea:v1.25.4
|
||||||
|
environment:
|
||||||
|
- GITEA__ui__THEMES=fhirworx
|
||||||
|
- GITEA__ui__DEFAULT_THEME=fhirworx
|
||||||
|
volumes:
|
||||||
|
- gitea_data:/var/lib/gitea
|
||||||
|
- gitea_config:/etc/gitea
|
||||||
|
- ./infra/gitea/custom:/var/lib/gitea/custom
|
||||||
|
```
|
||||||
|
|
||||||
|
`THEMES=fhirworx` (single option) — no gitea-auto/light/dark/protanopia
|
||||||
|
variants are exposed in the user appearance dropdown. fhirworx is the only
|
||||||
|
choice and it's the default. Per-user `theme` column in the postgres
|
||||||
|
`"user"` table should be set to `'fhirworx'`.
|
||||||
|
|
||||||
|
## What goes wrong if you bypass the build
|
||||||
|
|
||||||
|
Earlier iterations bind-mounted raw CSS files into stock
|
||||||
|
`gitea/gitea:1.25.4-rootless`. Two persistent failures:
|
||||||
|
|
||||||
|
- **Inode drift**: Docker's single-file bind mount tracks by inode. The
|
||||||
|
`Edit`/`Write` tools rewrite atomically, replacing the inode. The container
|
||||||
|
keeps pointing at the now-orphaned old inode and sees nothing change.
|
||||||
|
Required `docker restart` after every edit.
|
||||||
|
|
||||||
|
- **Cache + version-pinning**: The asset URL embeds Gitea's version
|
||||||
|
(`?v=1.25.4`). Browsers cache aggressively for 6h. Edits to the CSS file
|
||||||
|
don't change the URL → cache hit serves stale CSS forever (or until the
|
||||||
|
user knows to hard-refresh). On Cloudflare it's even longer.
|
||||||
|
|
||||||
|
Baking the theme into bindata sidesteps both. The CSS only changes when you
|
||||||
|
rebuild the image, which means the version string actually changes in the
|
||||||
|
binary's metadata, and there's no inode tracking to drift.
|
||||||
BIN
infra/gitea/brand/favicon-180.png
Normal file
|
After Width: | Height: | Size: 18 KiB |
1
infra/gitea/brand/favicon.svg
Normal file
|
After Width: | Height: | Size: 58 KiB |
BIN
infra/gitea/brand/logo-512.png
Normal file
|
After Width: | Height: | Size: 74 KiB |
1
infra/gitea/brand/logo.svg
Normal file
|
After Width: | Height: | Size: 58 KiB |
12
infra/gitea/custom/templates/custom/header.tmpl
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
{{/*
|
||||||
|
Fhirworx: declare the page color-scheme as light-only so client-side
|
||||||
|
dark-mode features (Chrome Auto Dark Mode for Web Contents, Firefox
|
||||||
|
`layout.css.prefers-color-scheme.content-override`, Safari Reader dark)
|
||||||
|
refuse to recolor the served palette.
|
||||||
|
|
||||||
|
This is in addition to `color-scheme: light only` in theme-fhirworx.css.
|
||||||
|
The <meta> form is honored by browsers BEFORE CSS evaluates, blocking
|
||||||
|
the auto-dark heuristic at the earliest possible point.
|
||||||
|
*/}}
|
||||||
|
<meta name="color-scheme" content="light dark">
|
||||||
|
<meta name="supported-color-schemes" content="light dark">
|
||||||
@@ -33,6 +33,17 @@
|
|||||||
gap: 0.5rem;
|
gap: 0.5rem;
|
||||||
margin-bottom: 1rem;
|
margin-bottom: 1rem;
|
||||||
}
|
}
|
||||||
|
.homelab-dashboard .masthead-brand {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 1rem;
|
||||||
|
}
|
||||||
|
.homelab-dashboard .masthead-logo {
|
||||||
|
width: clamp(48px, 6vw, 80px);
|
||||||
|
height: clamp(48px, 6vw, 80px);
|
||||||
|
display: block;
|
||||||
|
flex: 0 0 auto;
|
||||||
|
}
|
||||||
.homelab-dashboard .masthead-row h1 {
|
.homelab-dashboard .masthead-row h1 {
|
||||||
font-family: "Playfair Display", Georgia, serif !important;
|
font-family: "Playfair Display", Georgia, serif !important;
|
||||||
font-size: clamp(2.5rem, 5vw, 3.5rem) !important;
|
font-size: clamp(2.5rem, 5vw, 3.5rem) !important;
|
||||||
@@ -41,6 +52,7 @@
|
|||||||
letter-spacing: -0.02em !important;
|
letter-spacing: -0.02em !important;
|
||||||
line-height: 1.1 !important;
|
line-height: 1.1 !important;
|
||||||
border: none !important;
|
border: none !important;
|
||||||
|
margin: 0 !important;
|
||||||
}
|
}
|
||||||
.homelab-dashboard .masthead-eyebrow {
|
.homelab-dashboard .masthead-eyebrow {
|
||||||
font-family: "JetBrains Mono", "Fira Code", monospace;
|
font-family: "JetBrains Mono", "Fira Code", monospace;
|
||||||
@@ -232,7 +244,10 @@
|
|||||||
<div class="masthead">
|
<div class="masthead">
|
||||||
<div class="rule-line"></div>
|
<div class="rule-line"></div>
|
||||||
<div class="masthead-row">
|
<div class="masthead-row">
|
||||||
<h1>Homelab</h1>
|
<div class="masthead-brand">
|
||||||
|
<img src="/assets/img/logo.svg" alt="fhirworx" class="masthead-logo">
|
||||||
|
<h1>Homelab</h1>
|
||||||
|
</div>
|
||||||
<span class="masthead-eyebrow">git.fhirworx.io</span>
|
<span class="masthead-eyebrow">git.fhirworx.io</span>
|
||||||
</div>
|
</div>
|
||||||
<p class="tagline">
|
<p class="tagline">
|
||||||
|
|||||||
564
infra/gitea/theme/theme-fhirworx-dark.css
Normal file
@@ -0,0 +1,564 @@
|
|||||||
|
/* =============================================================================
|
||||||
|
Fhirworx — Gitea theme (dark variant)
|
||||||
|
=============================================================================
|
||||||
|
Structural reference: Gitea v1.25.4 web_src/css/themes/theme-gitea-dark.css.
|
||||||
|
Variable order mirrors upstream line-for-line (clean-diff rule). Only
|
||||||
|
VALUES differ from upstream. Shares variable NAMES with theme-fhirworx.css.
|
||||||
|
========================================================================== */
|
||||||
|
|
||||||
|
@import url('https://fonts.googleapis.com/css2?family=Playfair+Display:wght@400;600;700;800&family=Source+Serif+4:ital,wght@0,300;0,400;0,600;1,400&family=JetBrains+Mono:wght@400;500;600&display=swap');
|
||||||
|
|
||||||
|
gitea-theme-meta-info {
|
||||||
|
--theme-display-name: "Fhirworx Dark";
|
||||||
|
}
|
||||||
|
|
||||||
|
:root {
|
||||||
|
/* --- Fhirworx dark palette (source of truth) ---------------------------- */
|
||||||
|
--fw-background: #0F1419; /* deep dark body */
|
||||||
|
--fw-foreground: #E8E5DD; /* warm off-white body text */
|
||||||
|
--fw-card: #1A2028; /* card/panel surface, lifted from body */
|
||||||
|
--fw-card-hover: #232A34;
|
||||||
|
--fw-primary: #5684A3; /* muted blue; lifted so it beats --card contrast */
|
||||||
|
--fw-primary-foreground: #0F1419;
|
||||||
|
--fw-secondary: #1F262E;
|
||||||
|
--fw-muted-foreground: #8A8A87;
|
||||||
|
--fw-border: #3A4452;
|
||||||
|
--fw-nav-bg: #0A0E13; /* even darker than body (visible separation) */
|
||||||
|
--fw-nav-text: #E8F0F8;
|
||||||
|
--fw-nav-hover-bg: #1C2B3A;
|
||||||
|
--fw-nav-active-bg: #25384C;
|
||||||
|
--fw-destructive: #E07A6E;
|
||||||
|
--fw-link: #5C9CC0;
|
||||||
|
--fw-chart-3: #B8A078;
|
||||||
|
--fw-chart-5: #E08A4A;
|
||||||
|
|
||||||
|
--fonts-override: "Source Serif 4", Georgia, serif;
|
||||||
|
--fonts-monospace: "JetBrains Mono", "Fira Code", ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace;
|
||||||
|
|
||||||
|
/* ======================================================================= */
|
||||||
|
/* BELOW: Gitea variables in upstream order (theme-gitea-dark.css). */
|
||||||
|
/* ======================================================================= */
|
||||||
|
|
||||||
|
--is-dark-theme: true;
|
||||||
|
--color-primary: var(--fw-primary);
|
||||||
|
--color-primary-contrast: var(--fw-primary-foreground);
|
||||||
|
--color-primary-dark-1: #6D97B3;
|
||||||
|
--color-primary-dark-2: #85ABC2;
|
||||||
|
--color-primary-dark-3: #9CBED1;
|
||||||
|
--color-primary-dark-4: #B3D0DF;
|
||||||
|
--color-primary-dark-5: #C8DDE8;
|
||||||
|
--color-primary-dark-6: #DEEAF1;
|
||||||
|
--color-primary-dark-7: #F1F6F9;
|
||||||
|
--color-primary-light-1: #4A7590;
|
||||||
|
--color-primary-light-2: #3E657D;
|
||||||
|
--color-primary-light-3: #33556A;
|
||||||
|
--color-primary-light-4: #294557;
|
||||||
|
--color-primary-light-5: #203644;
|
||||||
|
--color-primary-light-6: #162531;
|
||||||
|
--color-primary-light-7: #0D1620;
|
||||||
|
--color-primary-alpha-10: #5684A319;
|
||||||
|
--color-primary-alpha-20: #5684A333;
|
||||||
|
--color-primary-alpha-30: #5684A34B;
|
||||||
|
--color-primary-alpha-40: #5684A366;
|
||||||
|
--color-primary-alpha-50: #5684A380;
|
||||||
|
--color-primary-alpha-60: #5684A399;
|
||||||
|
--color-primary-alpha-70: #5684A3B3;
|
||||||
|
--color-primary-alpha-80: #5684A3CC;
|
||||||
|
--color-primary-alpha-90: #5684A3E1;
|
||||||
|
--color-primary-hover: var(--color-primary-dark-1);
|
||||||
|
--color-primary-active: var(--color-primary-dark-2);
|
||||||
|
|
||||||
|
--color-secondary: var(--fw-border);
|
||||||
|
--color-secondary-dark-1: #45505E;
|
||||||
|
--color-secondary-dark-2: #505C6A;
|
||||||
|
--color-secondary-dark-3: #5F6B7A;
|
||||||
|
--color-secondary-dark-4: #6E7A89;
|
||||||
|
--color-secondary-dark-5: #7E8A99;
|
||||||
|
--color-secondary-dark-6: #8D99A7;
|
||||||
|
--color-secondary-dark-7: #9CA7B4;
|
||||||
|
--color-secondary-dark-8: #ABB5C0;
|
||||||
|
--color-secondary-dark-9: #B7C0CA;
|
||||||
|
--color-secondary-dark-10: #C3CBD4;
|
||||||
|
--color-secondary-dark-11: #CFD6DE;
|
||||||
|
--color-secondary-dark-12: #DAE0E7;
|
||||||
|
--color-secondary-dark-13: #E3E8EE;
|
||||||
|
--color-secondary-light-1: #2F3845;
|
||||||
|
--color-secondary-light-2: #262D38;
|
||||||
|
--color-secondary-light-3: #1C222B;
|
||||||
|
--color-secondary-light-4: #161B22;
|
||||||
|
--color-secondary-alpha-10: #3A445219;
|
||||||
|
--color-secondary-alpha-20: #3A445233;
|
||||||
|
--color-secondary-alpha-30: #3A44524B;
|
||||||
|
--color-secondary-alpha-40: #3A445266;
|
||||||
|
--color-secondary-alpha-50: #3A445280;
|
||||||
|
--color-secondary-alpha-60: #3A445299;
|
||||||
|
--color-secondary-alpha-70: #3A4452B3;
|
||||||
|
--color-secondary-alpha-80: #3A4452CC;
|
||||||
|
--color-secondary-alpha-90: #3A4452E1;
|
||||||
|
--color-secondary-button: var(--color-secondary-dark-4);
|
||||||
|
--color-secondary-hover: var(--color-secondary-dark-3);
|
||||||
|
--color-secondary-active: var(--color-secondary-dark-2);
|
||||||
|
|
||||||
|
/* console */
|
||||||
|
--color-console-fg: #F7F8F9;
|
||||||
|
--color-console-fg-subtle: #BDC4CC;
|
||||||
|
--color-console-bg: #050910;
|
||||||
|
--color-console-border: #2A323A;
|
||||||
|
--color-console-hover-bg: #121820;
|
||||||
|
--color-console-active-bg: #1A2028;
|
||||||
|
--color-console-menu-bg: #161B22;
|
||||||
|
--color-console-menu-border: #2E353D;
|
||||||
|
|
||||||
|
/* named colors — lifted ~15% lightness from the light variant so each hue
|
||||||
|
reads on dark bg. */
|
||||||
|
--color-red: var(--fw-destructive);
|
||||||
|
--color-orange: var(--fw-chart-5);
|
||||||
|
--color-yellow: #E5BE5A;
|
||||||
|
--color-olive: #ABC075;
|
||||||
|
--color-green: #4CB28A;
|
||||||
|
--color-teal: #66C7C0;
|
||||||
|
--color-blue: var(--fw-link);
|
||||||
|
--color-violet: #9B7FDB;
|
||||||
|
--color-purple: #B478D4;
|
||||||
|
--color-pink: #D878B0;
|
||||||
|
--color-brown: var(--fw-chart-3);
|
||||||
|
--color-black: #0A0E13;
|
||||||
|
|
||||||
|
--color-red-light: #E89891;
|
||||||
|
--color-orange-light: #E8A576;
|
||||||
|
--color-yellow-light: #EDD084;
|
||||||
|
--color-olive-light: #BCCF95;
|
||||||
|
--color-green-light: #77C6A5;
|
||||||
|
--color-teal-light: #8BD4CE;
|
||||||
|
--color-blue-light: #7CB0CB;
|
||||||
|
--color-violet-light: #AE97E0;
|
||||||
|
--color-purple-light: #C494DD;
|
||||||
|
--color-pink-light: #DF96C0;
|
||||||
|
--color-brown-light: #C6B28E;
|
||||||
|
--color-black-light: #2C333C;
|
||||||
|
|
||||||
|
--color-red-dark-1: #C86559;
|
||||||
|
--color-orange-dark-1: #C87239;
|
||||||
|
--color-yellow-dark-1: #C9A347;
|
||||||
|
--color-olive-dark-1: #92A65F;
|
||||||
|
--color-green-dark-1: #3A9876;
|
||||||
|
--color-teal-dark-1: #4FA9A2;
|
||||||
|
--color-blue-dark-1: #487FA1;
|
||||||
|
--color-violet-dark-1: #7E65BD;
|
||||||
|
--color-purple-dark-1: #9860B7;
|
||||||
|
--color-pink-dark-1: #BA6097;
|
||||||
|
--color-brown-dark-1: #9A8260;
|
||||||
|
--color-black-dark-1: #05080B;
|
||||||
|
|
||||||
|
--color-red-dark-2: #A74D42;
|
||||||
|
--color-orange-dark-2: #A35824;
|
||||||
|
--color-yellow-dark-2: #A8862F;
|
||||||
|
--color-olive-dark-2: #788B47;
|
||||||
|
--color-green-dark-2: #2D7C5F;
|
||||||
|
--color-teal-dark-2: #3B8C86;
|
||||||
|
--color-blue-dark-2: #3A688A;
|
||||||
|
--color-violet-dark-2: #634E9E;
|
||||||
|
--color-purple-dark-2: #7C4E99;
|
||||||
|
--color-pink-dark-2: #9B4C7E;
|
||||||
|
--color-brown-dark-2: #7B684C;
|
||||||
|
--color-black-dark-2: #020406;
|
||||||
|
|
||||||
|
/* ansi */
|
||||||
|
--color-ansi-black: #1E2327;
|
||||||
|
--color-ansi-red: #CC4848;
|
||||||
|
--color-ansi-green: #87AB63;
|
||||||
|
--color-ansi-yellow: #CC9903;
|
||||||
|
--color-ansi-blue: #3A8AC6;
|
||||||
|
--color-ansi-magenta: #D22E8B;
|
||||||
|
--color-ansi-cyan: #00918A;
|
||||||
|
--color-ansi-white: var(--color-console-fg-subtle);
|
||||||
|
--color-ansi-bright-black: #424851;
|
||||||
|
--color-ansi-bright-red: #D15A5A;
|
||||||
|
--color-ansi-bright-green: #93B373;
|
||||||
|
--color-ansi-bright-yellow: #EAAF03;
|
||||||
|
--color-ansi-bright-blue: #4E96CC;
|
||||||
|
--color-ansi-bright-magenta: #D74397;
|
||||||
|
--color-ansi-bright-cyan: #00B6AD;
|
||||||
|
--color-ansi-bright-white: var(--color-console-fg);
|
||||||
|
|
||||||
|
/* other */
|
||||||
|
--color-grey: var(--fw-muted-foreground);
|
||||||
|
--color-grey-light: #98948C;
|
||||||
|
--color-gold: var(--fw-chart-3);
|
||||||
|
--color-white: #FFFFFF;
|
||||||
|
|
||||||
|
/* diff colors */
|
||||||
|
--color-diff-added-linenum-bg: #1E3C2A;
|
||||||
|
--color-diff-added-row-bg: #18301E;
|
||||||
|
--color-diff-added-row-border: #2A4A32;
|
||||||
|
--color-diff-added-word-bg: #2E5C3E;
|
||||||
|
--color-diff-moved-row-bg: #423E20;
|
||||||
|
--color-diff-moved-row-border: #6A6836;
|
||||||
|
--color-diff-removed-linenum-bg: #402020;
|
||||||
|
--color-diff-removed-row-bg: #2C1818;
|
||||||
|
--color-diff-removed-row-border: #5A3434;
|
||||||
|
--color-diff-removed-word-bg: #5C2D2D;
|
||||||
|
--color-diff-inactive: #1A1F26;
|
||||||
|
|
||||||
|
--color-error-border: #8A3B3B;
|
||||||
|
--color-error-bg: #3C1F1F;
|
||||||
|
--color-error-bg-active:#5C2E2E;
|
||||||
|
--color-error-bg-hover: #4A2525;
|
||||||
|
--color-error-text: #F0B0B0;
|
||||||
|
--color-success-border: #3F7D4F;
|
||||||
|
--color-success-bg: #1F3A2A;
|
||||||
|
--color-success-text: #7EC89A;
|
||||||
|
--color-warning-border: #8A7A30;
|
||||||
|
--color-warning-bg: #3A3220;
|
||||||
|
--color-warning-text: #E5BE5A;
|
||||||
|
--color-info-border: #3A6A8F;
|
||||||
|
--color-info-bg: #1A2E44;
|
||||||
|
--color-info-text: var(--color-blue);
|
||||||
|
|
||||||
|
--color-red-badge: var(--color-red);
|
||||||
|
--color-red-badge-bg: #E07A6E1A;
|
||||||
|
--color-red-badge-hover-bg: #E07A6E4D;
|
||||||
|
--color-green-badge: var(--color-green);
|
||||||
|
--color-green-badge-bg: #4CB28A1A;
|
||||||
|
--color-green-badge-hover-bg:#4CB28A4D;
|
||||||
|
--color-yellow-badge: var(--color-yellow);
|
||||||
|
--color-yellow-badge-bg: #E5BE5A1A;
|
||||||
|
--color-yellow-badge-hover-bg:#E5BE5A4D;
|
||||||
|
--color-orange-badge: var(--color-orange);
|
||||||
|
--color-orange-badge-bg: #E08A4A1A;
|
||||||
|
--color-orange-badge-hover-bg:#E08A4A4D;
|
||||||
|
|
||||||
|
--color-git: #F05133;
|
||||||
|
--color-logo: var(--fw-primary);
|
||||||
|
|
||||||
|
/* target-based */
|
||||||
|
--color-body: var(--fw-background);
|
||||||
|
--color-box-header: var(--fw-secondary);
|
||||||
|
--color-box-body: var(--fw-card);
|
||||||
|
--color-box-body-highlight: #1E2630;
|
||||||
|
--color-text-dark: #F5F2EA;
|
||||||
|
--color-text: var(--fw-foreground);
|
||||||
|
--color-text-light: #CFCCC5;
|
||||||
|
--color-text-light-1: #B4B0A8;
|
||||||
|
--color-text-light-2: #9A968E;
|
||||||
|
--color-text-light-3: #807C74;
|
||||||
|
--color-footer: var(--color-nav-bg);
|
||||||
|
--color-timeline: var(--fw-border);
|
||||||
|
--color-input-text: var(--color-text-dark);
|
||||||
|
--color-input-background: #12171E;
|
||||||
|
--color-input-toggle-background: #2A323D;
|
||||||
|
--color-input-border: var(--color-secondary);
|
||||||
|
--color-input-border-hover: var(--color-secondary-dark-1);
|
||||||
|
|
||||||
|
--color-light: #00001728;
|
||||||
|
--color-light-mimic-enabled: rgba(0, 0, 0, calc(40 / 255 * 222 / 255 / var(--opacity-disabled)));
|
||||||
|
--color-light-border: #E8F3FF28;
|
||||||
|
--color-hover: #E8F3FF14;
|
||||||
|
--color-hover-opaque: var(--fw-card-hover);
|
||||||
|
--color-active: #E8F3FF24;
|
||||||
|
|
||||||
|
/* Dropdown/menu panel bg — must visibly separate from body, even when
|
||||||
|
floating over nav. In dark we use --card (slightly lifted from body). */
|
||||||
|
--color-menu: var(--fw-card);
|
||||||
|
--color-card: var(--fw-card);
|
||||||
|
--color-markup-table-row: #E8F3FF0F;
|
||||||
|
--color-markup-code-block: #E8F3FF12;
|
||||||
|
--color-markup-code-inline:#E8F3FF20;
|
||||||
|
--color-button: var(--fw-card);
|
||||||
|
--color-code-bg: #0B0F14;
|
||||||
|
--color-shadow: #00001758;
|
||||||
|
--color-shadow-opaque: #000017;
|
||||||
|
--color-secondary-bg: var(--fw-secondary);
|
||||||
|
--color-expand-button: #2A3845;
|
||||||
|
--color-placeholder-text: var(--color-text-light-3);
|
||||||
|
--color-editor-line-highlight: var(--color-primary-light-5);
|
||||||
|
--color-project-column-bg: var(--color-secondary-light-2);
|
||||||
|
--color-caret: var(--color-text);
|
||||||
|
--color-reaction-bg: #E8F3FF12;
|
||||||
|
--color-reaction-hover-bg: var(--color-primary-light-4);
|
||||||
|
--color-reaction-active-bg: var(--color-primary-light-5);
|
||||||
|
--color-tooltip-text: #F5F2EA;
|
||||||
|
--color-tooltip-bg: #000B17F0;
|
||||||
|
|
||||||
|
/* NAVBAR — still darker than body for separation */
|
||||||
|
--color-nav-bg: var(--fw-nav-bg);
|
||||||
|
--color-nav-hover-bg: var(--fw-nav-hover-bg);
|
||||||
|
--color-nav-text: var(--fw-nav-text);
|
||||||
|
--color-secondary-nav-bg: var(--color-secondary-light-3);
|
||||||
|
|
||||||
|
--color-label-text: var(--fw-foreground);
|
||||||
|
--color-label-bg: #6A7A8E4B;
|
||||||
|
--color-label-hover-bg: #6A7A8EA0;
|
||||||
|
--color-label-active-bg: #6A7A8EFF;
|
||||||
|
|
||||||
|
--color-accent: var(--color-primary-dark-1);
|
||||||
|
--color-small-accent: var(--color-primary-light-4);
|
||||||
|
--color-highlight-fg: #C9A347;
|
||||||
|
--color-highlight-bg: #3A2E1A;
|
||||||
|
--color-overlay-backdrop: #000B17D0;
|
||||||
|
|
||||||
|
/* Convenience aliases (fhirworx custom templates) */
|
||||||
|
--color-warning: var(--color-yellow);
|
||||||
|
--color-success: var(--color-green);
|
||||||
|
--color-info: var(--color-blue);
|
||||||
|
--color-error: var(--color-red);
|
||||||
|
--color-primary-foreground: var(--fw-primary-foreground);
|
||||||
|
|
||||||
|
accent-color: var(--color-accent);
|
||||||
|
color-scheme: dark only;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ============================================================================
|
||||||
|
Component polish — same structural rules as light variant.
|
||||||
|
============================================================================ */
|
||||||
|
|
||||||
|
h1, h2, h3, h4, h5, h6 {
|
||||||
|
font-family: "Playfair Display", Georgia, serif;
|
||||||
|
letter-spacing: -0.01em;
|
||||||
|
}
|
||||||
|
|
||||||
|
#navbar {
|
||||||
|
border-bottom: 3px solid var(--fw-border);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ---------- #navbar sweep ---------------------------------------------------
|
||||||
|
Same logic as light variant: Fomantic classes leak default text color into
|
||||||
|
nav. In dark theme the leak is from --color-text (warm off-white) onto
|
||||||
|
--color-nav-bg (near-black) — that's actually readable, but we still force
|
||||||
|
consistent nav-text color because some Fomantic state rules flip to white/
|
||||||
|
black directly, not through variables. */
|
||||||
|
|
||||||
|
#navbar,
|
||||||
|
#navbar a,
|
||||||
|
#navbar button,
|
||||||
|
#navbar .item,
|
||||||
|
#navbar .ui.button,
|
||||||
|
#navbar .ui.label,
|
||||||
|
#navbar .ui.menu,
|
||||||
|
#navbar .ui.menu .item,
|
||||||
|
#navbar .ui.dropdown,
|
||||||
|
#navbar .ui.dropdown > .text,
|
||||||
|
#navbar .ui.input,
|
||||||
|
#navbar #navbar-expand-toggle,
|
||||||
|
#navbar .navbar-left > *,
|
||||||
|
#navbar .navbar-right > *,
|
||||||
|
#navbar .navbar-mobile-right > * {
|
||||||
|
color: var(--color-nav-text);
|
||||||
|
}
|
||||||
|
|
||||||
|
#navbar svg,
|
||||||
|
#navbar svg path,
|
||||||
|
#navbar svg circle,
|
||||||
|
#navbar svg rect,
|
||||||
|
#navbar svg polygon {
|
||||||
|
fill: currentcolor;
|
||||||
|
color: currentcolor;
|
||||||
|
}
|
||||||
|
|
||||||
|
#navbar a:hover,
|
||||||
|
#navbar button:hover,
|
||||||
|
#navbar .ui.button:hover,
|
||||||
|
#navbar .item:hover,
|
||||||
|
#navbar .item.active,
|
||||||
|
#navbar #navbar-expand-toggle:hover {
|
||||||
|
background: var(--color-nav-hover-bg);
|
||||||
|
color: var(--color-nav-text);
|
||||||
|
}
|
||||||
|
|
||||||
|
#navbar .ui.button,
|
||||||
|
#navbar .item.button {
|
||||||
|
background: transparent;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Navbar dropdown popouts: panels float over body (--fw-background); use
|
||||||
|
card surface so they visually detach from the nav. */
|
||||||
|
#navbar .ui.dropdown > .menu {
|
||||||
|
background: var(--color-menu);
|
||||||
|
color: var(--color-text);
|
||||||
|
border: 1px solid var(--color-secondary);
|
||||||
|
}
|
||||||
|
#navbar .ui.dropdown > .menu .item,
|
||||||
|
#navbar .ui.dropdown > .menu a.item,
|
||||||
|
#navbar .ui.dropdown > .menu .header.item {
|
||||||
|
color: var(--color-text);
|
||||||
|
background: transparent;
|
||||||
|
}
|
||||||
|
#navbar .ui.dropdown > .menu .item:hover,
|
||||||
|
#navbar .ui.dropdown > .menu .item.selected,
|
||||||
|
#navbar .ui.dropdown > .menu .item.active {
|
||||||
|
background: var(--color-hover);
|
||||||
|
color: var(--color-text);
|
||||||
|
}
|
||||||
|
#navbar .ui.dropdown > .menu .item svg,
|
||||||
|
#navbar .ui.dropdown > .menu .item .svg {
|
||||||
|
color: var(--color-text);
|
||||||
|
fill: currentcolor;
|
||||||
|
}
|
||||||
|
#navbar .ui.dropdown > .menu .divider {
|
||||||
|
border-color: var(--color-secondary);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Notification badge: same override as light — upstream sets
|
||||||
|
color:var(--color-nav-bg), which here is near-black = same as badge bg. */
|
||||||
|
#navbar a.item .notification_count,
|
||||||
|
#navbar a.item .header-stopwatch-dot {
|
||||||
|
color: var(--fw-nav-bg);
|
||||||
|
background: var(--fw-chart-5);
|
||||||
|
border-color: var(--fw-nav-bg);
|
||||||
|
}
|
||||||
|
#navbar a.item:hover .notification_count,
|
||||||
|
#navbar a.item:hover .header-stopwatch-dot {
|
||||||
|
border-color: var(--fw-nav-hover-bg);
|
||||||
|
}
|
||||||
|
|
||||||
|
#navbar input,
|
||||||
|
#navbar .ui.input input {
|
||||||
|
background: var(--fw-nav-hover-bg);
|
||||||
|
color: var(--fw-nav-text);
|
||||||
|
border-color: var(--fw-nav-active-bg);
|
||||||
|
}
|
||||||
|
#navbar input::placeholder,
|
||||||
|
#navbar .ui.input input::placeholder {
|
||||||
|
color: var(--color-text-light-3);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Footer + monospace */
|
||||||
|
.page-footer {
|
||||||
|
border-top: 3px solid var(--fw-border);
|
||||||
|
font-family: var(--fonts-monospace);
|
||||||
|
font-size: 12px;
|
||||||
|
}
|
||||||
|
.commit-sha, .sha, [class*="sha"],
|
||||||
|
.branch-name, .tag-name, .file-name, code.ref {
|
||||||
|
font-family: var(--fonts-monospace);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Yellow/olive/orange labels & buttons: on light-hue fills the default white
|
||||||
|
fg is still okay on dark-theme yellow/orange when the hue is lifted, but
|
||||||
|
to stay consistent across variants we pin to foreground color in both. */
|
||||||
|
.ui.ui.ui.yellow.label,
|
||||||
|
.ui.ui.ui.olive.label,
|
||||||
|
.ui.ui.ui.orange.label,
|
||||||
|
.ui.yellow.button,
|
||||||
|
.ui.olive.button,
|
||||||
|
.ui.orange.button,
|
||||||
|
.ui.yellow.button:hover,
|
||||||
|
.ui.olive.button:hover,
|
||||||
|
.ui.orange.button:hover {
|
||||||
|
color: #0F1419; /* dark text on light-hue fills; explicit for WCAG AA */
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Basic-green label on dark bg — use the bright green directly, it has AA */
|
||||||
|
.ui.basic.green.label,
|
||||||
|
.ui.basic.green.labels .label {
|
||||||
|
color: var(--color-green);
|
||||||
|
border-color: var(--color-green);
|
||||||
|
}
|
||||||
|
|
||||||
|
a.ui.ui.ui.grey.label:hover,
|
||||||
|
a.ui.ui.ui.grey.label:focus {
|
||||||
|
color: var(--color-label-text);
|
||||||
|
background: var(--color-label-hover-bg);
|
||||||
|
}
|
||||||
|
|
||||||
|
.review-comments-counter {
|
||||||
|
background-color: var(--color-primary);
|
||||||
|
color: var(--color-primary-contrast);
|
||||||
|
}
|
||||||
|
|
||||||
|
.ui.card > .extra,
|
||||||
|
.ui.cards .card > .extra {
|
||||||
|
border-top-color: var(--color-secondary);
|
||||||
|
}
|
||||||
|
|
||||||
|
.text.small,
|
||||||
|
.flex-item-body .text.small {
|
||||||
|
font-family: var(--fonts-proportional), sans-serif;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* `.text.COLOR` helpers — on dark bg, use the *light* variants so hue stays
|
||||||
|
saturated and contrast is on the right side of 4.5:1. base.css uses
|
||||||
|
!important, so we must match it. */
|
||||||
|
.text.orange { color: var(--color-orange-light) !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.yellow { color: var(--color-yellow-light) !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.olive { color: var(--color-olive-light) !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.teal { color: var(--color-teal-light) !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.brown { color: var(--color-brown-light) !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.pink { color: var(--color-pink-light) !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.gold { color: var(--color-brown-light) !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.green { color: var(--color-green) !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.red { color: var(--color-red) !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.blue { color: var(--color-blue) !important; /* AA: base.css uses !important */ }
|
||||||
|
|
||||||
|
.ui.positive.message, .ui.success.message { color: var(--color-success-text); }
|
||||||
|
.ui.negative.message, .ui.error.message { color: var(--color-error-text); }
|
||||||
|
.ui.warning.message { color: var(--color-warning-text); }
|
||||||
|
.ui.info.message { color: var(--color-info-text); }
|
||||||
|
|
||||||
|
.secondary-nav {
|
||||||
|
background: var(--color-secondary-nav-bg) !important; /* beats .ui.secondary.menu !important */
|
||||||
|
border-bottom: 1px solid var(--color-secondary);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Link color: upstream `a { color: var(--color-primary) }`; primary in dark
|
||||||
|
is the muted blue which IS our link color already, so this is mostly a
|
||||||
|
no-op — kept for consistency with light variant. Exclude UI atoms. */
|
||||||
|
a:not(.item):not(.button):not(.label):not(.tab) {
|
||||||
|
color: var(--color-blue);
|
||||||
|
}
|
||||||
|
a:not(.item):not(.button):not(.label):not(.tab):hover {
|
||||||
|
color: var(--color-blue-light);
|
||||||
|
}
|
||||||
|
a.muted, a.suppressed, a.silenced, .muted-links a {
|
||||||
|
color: inherit;
|
||||||
|
}
|
||||||
|
|
||||||
|
.repository .diff-detail-box .diff-detail-stats strong:nth-of-type(1) {
|
||||||
|
color: var(--color-yellow);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ============================================================================
|
||||||
|
prefers-color-scheme:light defense — for users who set UA to light and
|
||||||
|
somehow land on this theme; re-assert our dark palette.
|
||||||
|
============================================================================ */
|
||||||
|
|
||||||
|
@media (prefers-color-scheme: light) {
|
||||||
|
:root {
|
||||||
|
color-scheme: dark only !important;
|
||||||
|
--color-body: var(--fw-background) !important;
|
||||||
|
--color-text: var(--fw-foreground) !important;
|
||||||
|
--color-nav-bg: var(--fw-nav-bg) !important;
|
||||||
|
--color-nav-text: var(--fw-nav-text) !important;
|
||||||
|
--color-box-body: var(--fw-card) !important;
|
||||||
|
--color-card: var(--fw-card) !important;
|
||||||
|
--color-menu: var(--fw-card) !important;
|
||||||
|
}
|
||||||
|
body {
|
||||||
|
background: var(--fw-background) !important; /* beats UA light override */
|
||||||
|
color: var(--fw-foreground) !important; /* beats UA light override */
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* invert emojis that upstream identifies as hard-to-read on dark bg */
|
||||||
|
.emoji[aria-label="check mark"],
|
||||||
|
.emoji[aria-label="currency exchange"],
|
||||||
|
.emoji[aria-label="TOP arrow"],
|
||||||
|
.emoji[aria-label="END arrow"],
|
||||||
|
.emoji[aria-label="ON! arrow"],
|
||||||
|
.emoji[aria-label="SOON arrow"],
|
||||||
|
.emoji[aria-label="heavy dollar sign"],
|
||||||
|
.emoji[aria-label="copyright"],
|
||||||
|
.emoji[aria-label="registered"],
|
||||||
|
.emoji[aria-label="trade mark"],
|
||||||
|
.emoji[aria-label="multiply"],
|
||||||
|
.emoji[aria-label="plus"],
|
||||||
|
.emoji[aria-label="minus"],
|
||||||
|
.emoji[aria-label="divide"],
|
||||||
|
.emoji[aria-label="curly loop"],
|
||||||
|
.emoji[aria-label="double curly loop"],
|
||||||
|
.emoji[aria-label="wavy dash"],
|
||||||
|
.emoji[aria-label="paw prints"],
|
||||||
|
.emoji[aria-label="musical note"],
|
||||||
|
.emoji[aria-label="musical notes"] {
|
||||||
|
filter: invert(100%) hue-rotate(180deg);
|
||||||
|
}
|
||||||
606
infra/gitea/theme/theme-fhirworx.css
Normal file
@@ -0,0 +1,606 @@
|
|||||||
|
/* =============================================================================
|
||||||
|
Fhirworx — Gitea theme (light variant)
|
||||||
|
=============================================================================
|
||||||
|
Structural reference: Gitea v1.25.4 web_src/css/themes/theme-gitea-light.css.
|
||||||
|
Variable ORDER in :root below mirrors upstream line-for-line so future Gitea
|
||||||
|
version bumps produce clean 3-way diffs (git merge). Only the VALUES change.
|
||||||
|
Component overrides (below :root) target selectors that can't be reshaped by
|
||||||
|
variable values alone (e.g. #navbar dark-surface sweep).
|
||||||
|
========================================================================== */
|
||||||
|
|
||||||
|
@import url('https://fonts.googleapis.com/css2?family=Playfair+Display:wght@400;600;700;800&family=Source+Serif+4:ital,wght@0,300;0,400;0,600;1,400&family=JetBrains+Mono:wght@400;500;600&display=swap');
|
||||||
|
|
||||||
|
gitea-theme-meta-info {
|
||||||
|
--theme-display-name: "Fhirworx";
|
||||||
|
}
|
||||||
|
|
||||||
|
:root {
|
||||||
|
/* --- Fhirworx source-of-truth palette ----------------------------------- */
|
||||||
|
--fw-background: #F7F5F0; /* cream body */
|
||||||
|
--fw-foreground: #1A1A18; /* near-black body text */
|
||||||
|
--fw-card: #FFFFFF; /* pure-white surfaces (separates from body) */
|
||||||
|
--fw-card-hover: #F3F0EA;
|
||||||
|
--fw-primary: #1C2B3A; /* dark navy; nav + primary button */
|
||||||
|
--fw-primary-foreground: #F7F5F0;
|
||||||
|
--fw-secondary: #EDEBE6; /* warm beige subtle bg */
|
||||||
|
--fw-muted-foreground: #6B6B68;
|
||||||
|
--fw-border: #B8B3A4; /* warm beige-darker (visible on cream) */
|
||||||
|
--fw-nav-bg: #1C2B3A;
|
||||||
|
--fw-nav-text: #F0F4F8;
|
||||||
|
--fw-nav-hover-bg: #253748;
|
||||||
|
--fw-nav-active-bg: #2E4359;
|
||||||
|
--fw-destructive: #C0392B;
|
||||||
|
--fw-link: #2D5F7F; /* editorial blue link */
|
||||||
|
--fw-chart-3: #A88B5C;
|
||||||
|
--fw-chart-5: #C8702A;
|
||||||
|
|
||||||
|
/* Font pipeline: Gitea composes
|
||||||
|
--fonts-regular: var(--fonts-override, var(--fonts-proportional)), ...
|
||||||
|
in base.css:59. Setting --fonts-override here propagates to every .ui.*
|
||||||
|
component, tab, button, dropdown, menu — single hook, whole UI. */
|
||||||
|
--fonts-override: "Source Serif 4", Georgia, serif;
|
||||||
|
--fonts-monospace: "JetBrains Mono", "Fira Code", ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace;
|
||||||
|
|
||||||
|
/* ======================================================================= */
|
||||||
|
/* BELOW: Gitea variables in upstream order (theme-gitea-light.css). */
|
||||||
|
/* Any variable present upstream MUST be present here (clean-diff rule). */
|
||||||
|
/* ======================================================================= */
|
||||||
|
|
||||||
|
--is-dark-theme: false;
|
||||||
|
--color-primary: var(--fw-primary);
|
||||||
|
--color-primary-contrast: var(--fw-primary-foreground);
|
||||||
|
--color-primary-dark-1: #172530;
|
||||||
|
--color-primary-dark-2: #121F28;
|
||||||
|
--color-primary-dark-3: #0E1920;
|
||||||
|
--color-primary-dark-4: #0A1418;
|
||||||
|
--color-primary-dark-5: #060D11;
|
||||||
|
--color-primary-dark-6: #030609;
|
||||||
|
--color-primary-dark-7: #010203;
|
||||||
|
--color-primary-light-1: #2E4A63;
|
||||||
|
--color-primary-light-2: #3F6683;
|
||||||
|
--color-primary-light-3: #5684A3;
|
||||||
|
--color-primary-light-4: #7BA0BC;
|
||||||
|
--color-primary-light-5: #A5BCCD;
|
||||||
|
--color-primary-light-6: #D2DDE6;
|
||||||
|
--color-primary-light-7: #EEF2F6;
|
||||||
|
--color-primary-alpha-10: #1C2B3A19;
|
||||||
|
--color-primary-alpha-20: #1C2B3A33;
|
||||||
|
--color-primary-alpha-30: #1C2B3A4B;
|
||||||
|
--color-primary-alpha-40: #1C2B3A66;
|
||||||
|
--color-primary-alpha-50: #1C2B3A80;
|
||||||
|
--color-primary-alpha-60: #1C2B3A99;
|
||||||
|
--color-primary-alpha-70: #1C2B3AB3;
|
||||||
|
--color-primary-alpha-80: #1C2B3ACC;
|
||||||
|
--color-primary-alpha-90: #1C2B3AE1;
|
||||||
|
--color-primary-hover: var(--color-primary-light-1);
|
||||||
|
--color-primary-active: var(--color-primary-dark-1);
|
||||||
|
|
||||||
|
/* Secondary drives nearly every 1px border in Gitea (segments, tables,
|
||||||
|
menus, inputs, dropdowns). Must be distinct from body. */
|
||||||
|
--color-secondary: var(--fw-border);
|
||||||
|
--color-secondary-dark-1: #A49F90;
|
||||||
|
--color-secondary-dark-2: #908B7D;
|
||||||
|
--color-secondary-dark-3: #7D7869;
|
||||||
|
--color-secondary-dark-4: #6A6558;
|
||||||
|
--color-secondary-dark-5: #5A554A;
|
||||||
|
--color-secondary-dark-6: #49453C;
|
||||||
|
--color-secondary-dark-7: #3A372F;
|
||||||
|
--color-secondary-dark-8: #2C2924;
|
||||||
|
--color-secondary-dark-9: #221F1B;
|
||||||
|
--color-secondary-dark-10: #181613;
|
||||||
|
--color-secondary-dark-11: #0F0E0C;
|
||||||
|
--color-secondary-dark-12: #050504;
|
||||||
|
--color-secondary-dark-13: #010100;
|
||||||
|
--color-secondary-light-1: #C7C3B6;
|
||||||
|
--color-secondary-light-2: #D4D0C4;
|
||||||
|
--color-secondary-light-3: #E0DCD1;
|
||||||
|
--color-secondary-light-4: #ECE9E1;
|
||||||
|
--color-secondary-alpha-10: #B8B3A419;
|
||||||
|
--color-secondary-alpha-20: #B8B3A433;
|
||||||
|
--color-secondary-alpha-30: #B8B3A44B;
|
||||||
|
--color-secondary-alpha-40: #B8B3A466;
|
||||||
|
--color-secondary-alpha-50: #B8B3A480;
|
||||||
|
--color-secondary-alpha-60: #B8B3A499;
|
||||||
|
--color-secondary-alpha-70: #B8B3A4B3;
|
||||||
|
--color-secondary-alpha-80: #B8B3A4CC;
|
||||||
|
--color-secondary-alpha-90: #B8B3A4E1;
|
||||||
|
--color-secondary-button: var(--color-secondary-dark-4);
|
||||||
|
--color-secondary-hover: var(--color-secondary-dark-5);
|
||||||
|
--color-secondary-active: var(--color-secondary-dark-6);
|
||||||
|
|
||||||
|
/* console colors */
|
||||||
|
--color-console-fg: #F7F5F0;
|
||||||
|
--color-console-fg-subtle: #BDC4CC;
|
||||||
|
--color-console-bg: #0F1419;
|
||||||
|
--color-console-border: #2A323A;
|
||||||
|
--color-console-hover-bg: #1E2730;
|
||||||
|
--color-console-active-bg: #2A323A;
|
||||||
|
--color-console-menu-bg: #1A2028;
|
||||||
|
--color-console-menu-border: #3A4552;
|
||||||
|
|
||||||
|
/* named colors (darkened where upstream hue fails AA on cream) */
|
||||||
|
--color-red: var(--fw-destructive);
|
||||||
|
--color-orange: var(--fw-chart-5);
|
||||||
|
--color-yellow: #D4A017;
|
||||||
|
--color-olive: #8B9A4C;
|
||||||
|
--color-green: #247058; /* 5.45:1 on cream (upstream #21ba45 = 2.9:1) */
|
||||||
|
--color-teal: #3A9A94;
|
||||||
|
--color-blue: var(--fw-link);
|
||||||
|
--color-violet: #6B52A3;
|
||||||
|
--color-purple: #8A4BA8;
|
||||||
|
--color-pink: #B85890;
|
||||||
|
--color-brown: var(--fw-chart-3);
|
||||||
|
--color-black: var(--fw-foreground);
|
||||||
|
--color-red-light: #D66A5C;
|
||||||
|
--color-orange-light: #DB8A4C;
|
||||||
|
--color-yellow-light: #E5C04A;
|
||||||
|
--color-olive-light: #B3C45C;
|
||||||
|
--color-green-light: #4EA988;
|
||||||
|
--color-teal-light: #5FB5AF;
|
||||||
|
--color-blue-light: #4E82A3;
|
||||||
|
--color-violet-light: #8974B8;
|
||||||
|
--color-purple-light: #A26BC0;
|
||||||
|
--color-pink-light: #CE7AAA;
|
||||||
|
--color-brown-light: #C0A280;
|
||||||
|
--color-black-light: #4A4A47;
|
||||||
|
--color-red-dark-1: #A93024;
|
||||||
|
--color-orange-dark-1: #B4621F;
|
||||||
|
--color-yellow-dark-1: #B38712;
|
||||||
|
--color-olive-dark-1: #6E7A38;
|
||||||
|
--color-green-dark-1: #1E5E49;
|
||||||
|
--color-teal-dark-1: #2C7E78;
|
||||||
|
--color-blue-dark-1: #244B63;
|
||||||
|
--color-violet-dark-1: #55418A;
|
||||||
|
--color-purple-dark-1: #6E388A;
|
||||||
|
--color-pink-dark-1: #963F70;
|
||||||
|
--color-brown-dark-1: #876F49;
|
||||||
|
--color-black-dark-1: #26252B;
|
||||||
|
--color-red-dark-2: #8D2720;
|
||||||
|
--color-orange-dark-2: #9A5018;
|
||||||
|
--color-yellow-dark-2: #8F6C0E;
|
||||||
|
--color-olive-dark-2: #54602A;
|
||||||
|
--color-green-dark-2: #17483A;
|
||||||
|
--color-teal-dark-2: #225F5B;
|
||||||
|
--color-blue-dark-2: #1C3A4E;
|
||||||
|
--color-violet-dark-2: #42336C;
|
||||||
|
--color-purple-dark-2: #562B6C;
|
||||||
|
--color-pink-dark-2: #723057;
|
||||||
|
--color-brown-dark-2: #6B583A;
|
||||||
|
--color-black-dark-2: #18181C;
|
||||||
|
|
||||||
|
/* ansi (terminal) — leave upstream values, these render on dark console bg */
|
||||||
|
--color-ansi-black: #1E2327;
|
||||||
|
--color-ansi-red: #CC4848;
|
||||||
|
--color-ansi-green: #87AB63;
|
||||||
|
--color-ansi-yellow: #CC9903;
|
||||||
|
--color-ansi-blue: #3A8AC6;
|
||||||
|
--color-ansi-magenta: #D22E8B;
|
||||||
|
--color-ansi-cyan: #00918A;
|
||||||
|
--color-ansi-white: var(--color-console-fg-subtle);
|
||||||
|
--color-ansi-bright-black: #46494D;
|
||||||
|
--color-ansi-bright-red: #D15A5A;
|
||||||
|
--color-ansi-bright-green: #93B373;
|
||||||
|
--color-ansi-bright-yellow: #EAAF03;
|
||||||
|
--color-ansi-bright-blue: #4E96CC;
|
||||||
|
--color-ansi-bright-magenta: #D74397;
|
||||||
|
--color-ansi-bright-cyan: #00B6AD;
|
||||||
|
--color-ansi-bright-white: var(--color-console-fg);
|
||||||
|
|
||||||
|
/* other */
|
||||||
|
--color-grey: var(--fw-muted-foreground);
|
||||||
|
--color-grey-light: #8A8A87;
|
||||||
|
--color-gold: var(--fw-chart-3);
|
||||||
|
--color-white: #FFFFFF;
|
||||||
|
|
||||||
|
/* diff colors — softened for cream body */
|
||||||
|
--color-diff-added-linenum-bg: #CDEED6;
|
||||||
|
--color-diff-added-row-bg: #E2F5E8;
|
||||||
|
--color-diff-added-row-border: #C2E0CB;
|
||||||
|
--color-diff-added-word-bg: #A8E0B7;
|
||||||
|
--color-diff-moved-row-bg: #EEF4D8;
|
||||||
|
--color-diff-moved-row-border: #D4DE93;
|
||||||
|
--color-diff-removed-linenum-bg: #F5C8C5;
|
||||||
|
--color-diff-removed-row-bg: #FAE8E8;
|
||||||
|
--color-diff-removed-row-border: #EBBFBF;
|
||||||
|
--color-diff-removed-word-bg: #F3B5BC;
|
||||||
|
--color-diff-inactive: #EEECE7;
|
||||||
|
|
||||||
|
--color-error-border: #D48B8B;
|
||||||
|
--color-error-bg: #F6E5E4;
|
||||||
|
--color-error-bg-active:#E8ADAD;
|
||||||
|
--color-error-bg-hover: #F0C8C8;
|
||||||
|
--color-error-text: #8E2E2E;
|
||||||
|
--color-success-border: #A3C293;
|
||||||
|
--color-success-bg: #E8F2DD;
|
||||||
|
--color-success-text: #264F26;
|
||||||
|
--color-warning-border: #C9BA9B;
|
||||||
|
--color-warning-bg: #F8F0DC;
|
||||||
|
--color-warning-text: #5A3D10;
|
||||||
|
--color-info-border: #A5BCCD;
|
||||||
|
--color-info-bg: #E8F0F6;
|
||||||
|
--color-info-text: var(--color-blue); /* distinct from primary navy */
|
||||||
|
|
||||||
|
--color-red-badge: var(--color-red);
|
||||||
|
--color-red-badge-bg: #C0392B1A;
|
||||||
|
--color-red-badge-hover-bg: #C0392B4D;
|
||||||
|
--color-green-badge: var(--color-green);
|
||||||
|
--color-green-badge-bg: #2470581A;
|
||||||
|
--color-green-badge-hover-bg:#2470584D;
|
||||||
|
--color-yellow-badge: var(--color-yellow);
|
||||||
|
--color-yellow-badge-bg: #D4A0171A;
|
||||||
|
--color-yellow-badge-hover-bg:#D4A0174D;
|
||||||
|
--color-orange-badge: var(--color-orange);
|
||||||
|
--color-orange-badge-bg: #C8702A1A;
|
||||||
|
--color-orange-badge-hover-bg:#C8702A4D;
|
||||||
|
|
||||||
|
--color-git: #F05133;
|
||||||
|
--color-logo: var(--fw-primary);
|
||||||
|
|
||||||
|
/* target-based colors */
|
||||||
|
--color-body: var(--fw-background);
|
||||||
|
--color-box-header: var(--fw-secondary);
|
||||||
|
--color-box-body: var(--fw-card);
|
||||||
|
--color-box-body-highlight: var(--color-primary-light-7);
|
||||||
|
--color-text-dark: #0D0D0C;
|
||||||
|
--color-text: var(--fw-foreground);
|
||||||
|
--color-text-light: #2E2E2C;
|
||||||
|
--color-text-light-1: #454542;
|
||||||
|
--color-text-light-2: #595955;
|
||||||
|
--color-text-light-3: #6E6E6A;
|
||||||
|
--color-footer: var(--color-nav-bg);
|
||||||
|
--color-timeline: var(--fw-border);
|
||||||
|
--color-input-text: var(--color-text-dark);
|
||||||
|
--color-input-background: var(--fw-card);
|
||||||
|
--color-input-toggle-background: var(--fw-border);
|
||||||
|
--color-input-border: var(--color-secondary);
|
||||||
|
--color-input-border-hover: var(--color-secondary-dark-2);
|
||||||
|
|
||||||
|
/* Striped-table "--color-light": upstream is ~3% black on white; on cream
|
||||||
|
that's ~1.5% darker = invisible stripe. Opaque light-tan instead. */
|
||||||
|
--color-light: #EFECE5;
|
||||||
|
--color-light-mimic-enabled: rgba(0, 0, 0, calc(16 / 255 * 222 / 255 / var(--opacity-disabled)));
|
||||||
|
--color-light-border: #00001728;
|
||||||
|
--color-hover: #00001710;
|
||||||
|
--color-hover-opaque: var(--fw-card-hover);
|
||||||
|
--color-active: #0000171F;
|
||||||
|
|
||||||
|
/* Dropdown/menu panel bg: upstream uses #f8f9fb (off-white). We promote to
|
||||||
|
pure white so panels over the cream body have a real surface break.
|
||||||
|
This is THE variable the user's recent "transparent dropdown" bug hit —
|
||||||
|
if it resolves to `initial`/transparent the panel disappears. */
|
||||||
|
--color-menu: var(--fw-card);
|
||||||
|
--color-card: var(--fw-card);
|
||||||
|
--color-markup-table-row: #00306008;
|
||||||
|
--color-markup-code-block: #00306010;
|
||||||
|
--color-markup-code-inline:#00306014;
|
||||||
|
--color-button: var(--fw-card);
|
||||||
|
--color-code-bg: var(--fw-card);
|
||||||
|
--color-shadow: #00001726;
|
||||||
|
--color-shadow-opaque: var(--fw-border);
|
||||||
|
--color-secondary-bg: var(--fw-secondary);
|
||||||
|
--color-expand-button: var(--color-primary-light-6);
|
||||||
|
--color-placeholder-text: var(--color-text-light-3);
|
||||||
|
--color-editor-line-highlight: var(--color-primary-light-7);
|
||||||
|
--color-project-column-bg: var(--color-secondary-light-4);
|
||||||
|
--color-caret: var(--color-text-dark);
|
||||||
|
--color-reaction-bg: #0000170A;
|
||||||
|
--color-reaction-hover-bg: var(--color-primary-light-6);
|
||||||
|
--color-reaction-active-bg: var(--color-primary-light-5);
|
||||||
|
--color-tooltip-text: var(--fw-primary-foreground);
|
||||||
|
--color-tooltip-bg: #111B25F0;
|
||||||
|
|
||||||
|
/* NAVBAR — fhirworx signature: dark navy bar over cream body. */
|
||||||
|
--color-nav-bg: var(--fw-nav-bg);
|
||||||
|
--color-nav-hover-bg: var(--fw-nav-hover-bg);
|
||||||
|
--color-nav-text: var(--fw-nav-text);
|
||||||
|
--color-secondary-nav-bg: var(--color-secondary-light-4);
|
||||||
|
|
||||||
|
--color-label-text: var(--fw-foreground);
|
||||||
|
--color-label-bg: #94908648;
|
||||||
|
--color-label-hover-bg: #9490869E;
|
||||||
|
--color-label-active-bg: #949086E6;
|
||||||
|
|
||||||
|
--color-accent: var(--fw-primary);
|
||||||
|
--color-small-accent: var(--color-primary-light-6);
|
||||||
|
--color-highlight-fg: var(--fw-chart-3);
|
||||||
|
--color-highlight-bg: #FFF3C4;
|
||||||
|
--color-overlay-backdrop: #000017C0;
|
||||||
|
|
||||||
|
/* Convenience aliases consumed by fhirworx custom templates. */
|
||||||
|
--color-warning: #7F5E08; /* AA both as bg and fg on cream (5.5:1) */
|
||||||
|
--color-success: var(--color-green);
|
||||||
|
--color-info: var(--color-blue);
|
||||||
|
--color-error: var(--color-red);
|
||||||
|
--color-primary-foreground: var(--fw-primary-foreground);
|
||||||
|
|
||||||
|
accent-color: var(--color-accent);
|
||||||
|
|
||||||
|
/* `light only` opts out of Chrome auto-dark, Firefox content-override, and
|
||||||
|
Safari Reader dark. `only` keyword is critical — plain `light` allows UA
|
||||||
|
overrides; `light only` forbids them. */
|
||||||
|
color-scheme: light only;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ============================================================================
|
||||||
|
Component polish — rules the variable system cannot express.
|
||||||
|
============================================================================ */
|
||||||
|
|
||||||
|
h1, h2, h3, h4, h5, h6 {
|
||||||
|
font-family: "Playfair Display", Georgia, serif;
|
||||||
|
letter-spacing: -0.01em;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Dark navbar accent line (Gitea nav id is #navbar, not .navbar). */
|
||||||
|
#navbar {
|
||||||
|
border-bottom: 3px solid var(--fw-foreground);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ---------- #navbar dark-surface sweep --------------------------------------
|
||||||
|
Fhirworx inverts the nav: DARK bar over LIGHT body. Gitea base CSS assumes
|
||||||
|
nav luminance ≈ body luminance, so any Fomantic class with its own color
|
||||||
|
rule (.ui.button, .ui.label, .ui.input, .ui.dropdown, .ui.menu .item) leaks
|
||||||
|
near-black text into the dark bar — invisible. Sweep forces nav-text color
|
||||||
|
on every interactive descendant. `#navbar` id-specificity (1-1-0) beats
|
||||||
|
Fomantic's single-class rules (0-1-0) without !important. */
|
||||||
|
|
||||||
|
#navbar,
|
||||||
|
#navbar a,
|
||||||
|
#navbar button,
|
||||||
|
#navbar .item,
|
||||||
|
#navbar .ui.button,
|
||||||
|
#navbar .ui.label,
|
||||||
|
#navbar .ui.menu,
|
||||||
|
#navbar .ui.menu .item,
|
||||||
|
#navbar .ui.dropdown,
|
||||||
|
#navbar .ui.dropdown > .text,
|
||||||
|
#navbar .ui.input,
|
||||||
|
#navbar #navbar-expand-toggle,
|
||||||
|
#navbar .navbar-left > *,
|
||||||
|
#navbar .navbar-right > *,
|
||||||
|
#navbar .navbar-mobile-right > * {
|
||||||
|
color: var(--color-nav-text);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* SVG icons in nav: follow currentcolor so the sweep above reaches them. */
|
||||||
|
#navbar svg,
|
||||||
|
#navbar svg path,
|
||||||
|
#navbar svg circle,
|
||||||
|
#navbar svg rect,
|
||||||
|
#navbar svg polygon {
|
||||||
|
fill: currentcolor;
|
||||||
|
color: currentcolor;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Hover/active state across all interactive elements in nav */
|
||||||
|
#navbar a:hover,
|
||||||
|
#navbar button:hover,
|
||||||
|
#navbar .ui.button:hover,
|
||||||
|
#navbar .item:hover,
|
||||||
|
#navbar .item.active,
|
||||||
|
#navbar #navbar-expand-toggle:hover {
|
||||||
|
background: var(--color-nav-hover-bg);
|
||||||
|
color: var(--fw-primary-foreground);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Fomantic backgrounds inside #navbar must clear so nav bg shows through. */
|
||||||
|
#navbar .ui.button,
|
||||||
|
#navbar .item.button {
|
||||||
|
background: transparent;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Dropdown popouts FROM the navbar are floating panels OVER the body; they
|
||||||
|
should NOT inherit the dark nav palette. Flip to body (cream surface, dark
|
||||||
|
text). This is the main acute pain point — without these 4 rules the menu
|
||||||
|
panels render with transparent bg + near-black text = invisible. */
|
||||||
|
#navbar .ui.dropdown > .menu {
|
||||||
|
background: var(--color-menu);
|
||||||
|
color: var(--color-text);
|
||||||
|
border: 1px solid var(--color-secondary);
|
||||||
|
}
|
||||||
|
#navbar .ui.dropdown > .menu .item,
|
||||||
|
#navbar .ui.dropdown > .menu a.item,
|
||||||
|
#navbar .ui.dropdown > .menu .header.item {
|
||||||
|
color: var(--color-text);
|
||||||
|
background: transparent;
|
||||||
|
}
|
||||||
|
#navbar .ui.dropdown > .menu .item:hover,
|
||||||
|
#navbar .ui.dropdown > .menu .item.selected,
|
||||||
|
#navbar .ui.dropdown > .menu .item.active {
|
||||||
|
background: var(--color-hover);
|
||||||
|
color: var(--color-text);
|
||||||
|
}
|
||||||
|
#navbar .ui.dropdown > .menu .item svg,
|
||||||
|
#navbar .ui.dropdown > .menu .item .svg {
|
||||||
|
color: var(--color-text);
|
||||||
|
fill: currentcolor;
|
||||||
|
}
|
||||||
|
#navbar .ui.dropdown > .menu .divider {
|
||||||
|
border-color: var(--color-secondary);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Notification badge: Gitea sets color:var(--color-nav-bg) on the count span
|
||||||
|
(navbar.css:111), assuming nav-bg is light enough to read on the primary-
|
||||||
|
colored badge bg. In fhirworx nav-bg == primary — same color = invisible.
|
||||||
|
Force badge text to primary-contrast. */
|
||||||
|
#navbar a.item .notification_count,
|
||||||
|
#navbar a.item .header-stopwatch-dot {
|
||||||
|
color: var(--fw-primary-foreground);
|
||||||
|
background: var(--fw-chart-5); /* orange badge, visible on navy */
|
||||||
|
border-color: var(--fw-nav-bg);
|
||||||
|
}
|
||||||
|
#navbar a.item:hover .notification_count,
|
||||||
|
#navbar a.item:hover .header-stopwatch-dot {
|
||||||
|
border-color: var(--fw-nav-hover-bg);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Navbar search input on dark bg: use a slightly-lifted navy so the input is
|
||||||
|
visible against the bar, with near-white placeholder. */
|
||||||
|
#navbar input,
|
||||||
|
#navbar .ui.input input {
|
||||||
|
background: var(--fw-nav-hover-bg);
|
||||||
|
color: var(--fw-nav-text);
|
||||||
|
border-color: var(--fw-nav-active-bg);
|
||||||
|
}
|
||||||
|
#navbar input::placeholder,
|
||||||
|
#navbar .ui.input input::placeholder {
|
||||||
|
color: var(--color-console-fg-subtle);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Footer — same dark-surface inversion as navbar. Footer bg defaults to
|
||||||
|
var(--color-nav-bg) (= dark navy in fhirworx); without overriding, every
|
||||||
|
text descendant inherits var(--color-text) (= near-black) → contrast 1.2:1.
|
||||||
|
Force every text/icon descendant to nav-text, same as the #navbar sweep. */
|
||||||
|
.page-footer {
|
||||||
|
border-top: 3px solid var(--fw-foreground);
|
||||||
|
font-family: var(--fonts-monospace);
|
||||||
|
font-size: 12px;
|
||||||
|
color: var(--color-nav-text);
|
||||||
|
}
|
||||||
|
.page-footer,
|
||||||
|
.page-footer a,
|
||||||
|
.page-footer span,
|
||||||
|
.page-footer strong,
|
||||||
|
.page-footer .left-links,
|
||||||
|
.page-footer .right-links,
|
||||||
|
.page-footer .flex-text-inline,
|
||||||
|
.page-footer .ui.dropdown,
|
||||||
|
.page-footer .item {
|
||||||
|
color: var(--color-nav-text);
|
||||||
|
}
|
||||||
|
.page-footer a:hover {
|
||||||
|
color: var(--fw-primary-foreground);
|
||||||
|
}
|
||||||
|
.page-footer svg,
|
||||||
|
.page-footer svg path {
|
||||||
|
fill: currentcolor;
|
||||||
|
}
|
||||||
|
.commit-sha, .sha, [class*="sha"],
|
||||||
|
.branch-name, .tag-name, .file-name, code.ref {
|
||||||
|
font-family: var(--fonts-monospace);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Yellow / olive / orange labels + buttons: white on these hues fails WCAG
|
||||||
|
AA. Use --fw-foreground (near-black). The .ui.ui.ui chain matches
|
||||||
|
Fomantic's triple-class specificity for state-modified labels. */
|
||||||
|
.ui.ui.ui.yellow.label,
|
||||||
|
.ui.ui.ui.olive.label,
|
||||||
|
.ui.ui.ui.orange.label,
|
||||||
|
.ui.yellow.button,
|
||||||
|
.ui.olive.button,
|
||||||
|
.ui.orange.button,
|
||||||
|
.ui.yellow.button:hover,
|
||||||
|
.ui.olive.button:hover,
|
||||||
|
.ui.orange.button:hover {
|
||||||
|
color: var(--fw-foreground);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Basic-green label text must darken for AA on cream (base green is 3.8:1). */
|
||||||
|
.ui.basic.green.label,
|
||||||
|
.ui.basic.green.labels .label {
|
||||||
|
color: var(--color-green-dark-1);
|
||||||
|
border-color: var(--color-green-dark-1);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Grey-label hover: Fomantic flips text to var(--color-white) while bg stays
|
||||||
|
semi-transparent beige → invisible. Preserve label-text on hover. */
|
||||||
|
a.ui.ui.ui.grey.label:hover,
|
||||||
|
a.ui.ui.ui.grey.label:focus {
|
||||||
|
color: var(--color-label-text);
|
||||||
|
background: var(--color-label-hover-bg);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* PR review-comments-counter uses --color-primary-light-4 (pale navy) bg
|
||||||
|
with cream text → fails contrast. Use opaque primary instead. */
|
||||||
|
.review-comments-counter {
|
||||||
|
background-color: var(--color-primary);
|
||||||
|
color: var(--color-primary-contrast);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Card .extra divider uses --color-secondary-light-1 (paler than secondary
|
||||||
|
on cream = invisible divider). Force visible. */
|
||||||
|
.ui.card > .extra,
|
||||||
|
.ui.cards .card > .extra {
|
||||||
|
border-top-color: var(--color-secondary);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* `.text.small` at 0.75em serif renders poorly; keep system font for it. */
|
||||||
|
.text.small,
|
||||||
|
.flex-item-body .text.small {
|
||||||
|
font-family: var(--fonts-proportional), sans-serif;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* base.css uses `color: var(--color-COLOR) !important` for .text.COLOR
|
||||||
|
helpers. Several hues fail AA on cream; route them through darker shades.
|
||||||
|
We must use !important here because base.css line 920-ish does too —
|
||||||
|
without !important we lose the cascade. */
|
||||||
|
.text.orange { color: var(--color-orange-dark-2) !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.yellow { color: #7F5E08 !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.olive { color: var(--color-olive-dark-2) !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.teal { color: var(--color-teal-dark-2) !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.brown { color: var(--color-brown-dark-2) !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.pink { color: var(--color-pink-dark-1) !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.gold { color: var(--color-brown-dark-2) !important; /* AA: base.css uses !important */ }
|
||||||
|
.text.green { color: var(--color-green-dark-1) !important; /* AA: base.css uses !important */ }
|
||||||
|
|
||||||
|
/* Flash messages: force readable text on pastel bg tints. */
|
||||||
|
.ui.positive.message, .ui.success.message { color: var(--color-success-text); }
|
||||||
|
.ui.negative.message, .ui.error.message { color: var(--color-error-text); }
|
||||||
|
.ui.warning.message { color: var(--color-warning-text); }
|
||||||
|
.ui.info.message { color: var(--color-info-text); }
|
||||||
|
|
||||||
|
/* Secondary-nav (repo breadcrumb strip) — use !important because Fomantic's
|
||||||
|
.ui.secondary.menu rule uses it too; matching specificity. */
|
||||||
|
.secondary-nav {
|
||||||
|
background: var(--color-secondary-nav-bg) !important; /* beats .ui.secondary.menu !important in Fomantic */
|
||||||
|
border-bottom: 1px solid var(--color-secondary);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Link color: base.css sets `a { color: var(--color-primary) }`. In fhirworx
|
||||||
|
--color-primary = dark navy = same family as --color-text, so links look
|
||||||
|
identical to prose. Route real hyperlinks through a brighter blue. Exclude
|
||||||
|
UI atoms (.item/.button/.label/.tab) which carry Fomantic color logic. */
|
||||||
|
a:not(.item):not(.button):not(.label):not(.tab) {
|
||||||
|
color: var(--color-blue);
|
||||||
|
}
|
||||||
|
a:not(.item):not(.button):not(.label):not(.tab):hover {
|
||||||
|
color: var(--color-blue-dark-1);
|
||||||
|
}
|
||||||
|
a.muted, a.suppressed, a.silenced, .muted-links a {
|
||||||
|
color: inherit;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Diff detail yellow count fails AA on cream */
|
||||||
|
.repository .diff-detail-box .diff-detail-stats strong:nth-of-type(1) {
|
||||||
|
color: var(--color-yellow-dark-1);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ============================================================================
|
||||||
|
Defense against client-side dark mode (Chrome auto-dark, Firefox content-
|
||||||
|
override). Even with `color-scheme: light only`, some UAs still apply
|
||||||
|
per-element dark adjustments when prefers-color-scheme:dark. Re-assert.
|
||||||
|
============================================================================ */
|
||||||
|
|
||||||
|
@media (prefers-color-scheme: dark) {
|
||||||
|
:root {
|
||||||
|
color-scheme: light only !important; /* beats UA-injected dark scheme */
|
||||||
|
--color-body: var(--fw-background) !important;
|
||||||
|
--color-text: var(--fw-foreground) !important;
|
||||||
|
--color-nav-bg: var(--fw-nav-bg) !important;
|
||||||
|
--color-nav-text: var(--fw-nav-text) !important;
|
||||||
|
--color-box-body: var(--fw-card) !important;
|
||||||
|
--color-card: var(--fw-card) !important;
|
||||||
|
--color-menu: var(--fw-card) !important;
|
||||||
|
}
|
||||||
|
body {
|
||||||
|
background: var(--fw-background) !important; /* beats UA dark bg */
|
||||||
|
color: var(--fw-foreground) !important; /* beats UA dark text */
|
||||||
|
}
|
||||||
|
#navbar {
|
||||||
|
background: var(--fw-nav-bg) !important; /* beats UA dark bg */
|
||||||
|
color: var(--fw-nav-text) !important; /* beats UA dark text */
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (forced-colors: active) {
|
||||||
|
#navbar { border-bottom: 3px solid CanvasText; }
|
||||||
|
}
|
||||||
@@ -6,8 +6,11 @@ WORKDIR /app
|
|||||||
# Local package registry (Gitea) — set via --build-arg to pull from mirror
|
# Local package registry (Gitea) — set via --build-arg to pull from mirror
|
||||||
ARG PYPI_INDEX_URL=""
|
ARG PYPI_INDEX_URL=""
|
||||||
|
|
||||||
# Patch base image CVEs
|
# Patch base image CVEs + install curl for healthcheck.
|
||||||
RUN apt-get update && apt-get upgrade -y && rm -rf /var/lib/apt/lists/*
|
# (Python cold-start on this image is 10-13s — too slow for the 5s
|
||||||
|
# healthcheck timeout, so Docker marks the container unhealthy even
|
||||||
|
# though /health responds in milliseconds.)
|
||||||
|
RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
# Copy project files for install
|
# Copy project files for install
|
||||||
COPY pyproject.toml uv.lock README.md ./
|
COPY pyproject.toml uv.lock README.md ./
|
||||||
@@ -26,7 +29,7 @@ COPY stack.toml ./
|
|||||||
EXPOSE 8000
|
EXPOSE 8000
|
||||||
|
|
||||||
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
|
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
|
||||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" || exit 1
|
CMD curl -sf http://localhost:8000/health || exit 1
|
||||||
|
|
||||||
CMD ["uv", "run", "--no-sync", "uvicorn", "api.server:app", \
|
CMD ["uv", "run", "--no-sync", "uvicorn", "api.server:app", \
|
||||||
"--host", "0.0.0.0", "--port", "8000", \
|
"--host", "0.0.0.0", "--port", "8000", \
|
||||||
|
|||||||
@@ -10,8 +10,13 @@ COPY docs/scripts/ docs/scripts/
|
|||||||
# Extract API docs via griffe (pure AST, no imports)
|
# Extract API docs via griffe (pure AST, no imports)
|
||||||
RUN uv run --with griffe python docs/scripts/extract_docs.py
|
RUN uv run --with griffe python docs/scripts/extract_docs.py
|
||||||
|
|
||||||
# Export bibliography (gracefully handles missing bib.sqlite)
|
# Export bibliography. The COPY uses a multi-source form so the
|
||||||
COPY data/bib.sqlit[e] data/
|
# bib.sqlite is optional — `data/.gitkeep` guarantees there's always
|
||||||
|
# at least one matching source so BuildKit doesn't error on an empty
|
||||||
|
# glob like `data/bib.sqlit[e]` did. The export script in turn
|
||||||
|
# tolerates a missing bib.sqlite and writes an empty library.json.
|
||||||
|
RUN mkdir -p data
|
||||||
|
COPY data/.gitkeep data/bib.sqlit[e] data/
|
||||||
RUN uv run --with pydantic python docs/scripts/export_library.py
|
RUN uv run --with pydantic python docs/scripts/export_library.py
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,69 @@
|
|||||||
# syntax=docker/dockerfile:1
|
# syntax=docker/dockerfile:1
|
||||||
|
#
|
||||||
|
# Notebooks runtime — fhirworx-themed marimo on CUDA + Python 3.13.
|
||||||
|
#
|
||||||
|
# Stage 1 (fe): node + pnpm. Overlay fhirworx theme onto a pinned
|
||||||
|
# marimo source tree and compile frontend + lsp into
|
||||||
|
# marimo/_static/ and marimo/_lsp/.
|
||||||
|
# Stage 2 (wheel): uv builds a marimo wheel with the fhirworx bundle
|
||||||
|
# baked in.
|
||||||
|
# Stage 3 (final): CUDA runtime + Python stack; installs the wheel into
|
||||||
|
# the workspace venv.
|
||||||
|
#
|
||||||
|
# Iterate by editing infra/marimo/theme/**. BuildKit cache mounts keep
|
||||||
|
# pnpm and turbo warm between rebuilds, so CSS/icon edits land in under
|
||||||
|
# a minute after first bootstrap.
|
||||||
|
|
||||||
|
ARG MARIMO_VERSION=0.23.1
|
||||||
|
|
||||||
|
# ---- stage 1: frontend ------------------------------------------------------
|
||||||
|
|
||||||
|
FROM node:22-bookworm-slim AS fe
|
||||||
|
ARG MARIMO_VERSION
|
||||||
|
WORKDIR /src
|
||||||
|
|
||||||
|
RUN apt-get update \
|
||||||
|
&& apt-get install -y --no-install-recommends git ca-certificates python3 jq \
|
||||||
|
&& rm -rf /var/lib/apt/lists/* \
|
||||||
|
&& corepack enable \
|
||||||
|
&& corepack prepare pnpm@latest --activate
|
||||||
|
|
||||||
|
# Shallow clone marimo at the pinned tag.
|
||||||
|
RUN git clone --depth 1 --branch ${MARIMO_VERSION} --filter=blob:none \
|
||||||
|
https://github.com/marimo-team/marimo /src
|
||||||
|
|
||||||
|
COPY infra/marimo/theme /overlay
|
||||||
|
RUN /overlay/scripts/apply-overlay.sh /src /overlay
|
||||||
|
|
||||||
|
RUN --mount=type=cache,target=/root/.cache/pnpm,sharing=locked \
|
||||||
|
pnpm install --no-frozen-lockfile
|
||||||
|
|
||||||
|
RUN --mount=type=cache,target=/root/.cache/turbo,sharing=locked \
|
||||||
|
--mount=type=cache,target=/src/frontend/node_modules/.cache,sharing=locked \
|
||||||
|
NODE_ENV=production pnpm turbo build --filter @marimo-team/frontend --output-logs=full
|
||||||
|
|
||||||
|
RUN --mount=type=cache,target=/root/.cache/turbo,sharing=locked \
|
||||||
|
NODE_ENV=production pnpm turbo build --filter @marimo-team/lsp --output-logs=full
|
||||||
|
|
||||||
|
RUN rm -rf marimo/_static marimo/_lsp \
|
||||||
|
&& mkdir -p marimo/_static marimo/_lsp \
|
||||||
|
&& cp -R frontend/dist/. marimo/_static/ \
|
||||||
|
&& rm -f marimo/_static/files/wasm-intro.py \
|
||||||
|
&& cp docs/_static/CLAUDE.md marimo/_static/CLAUDE.md 2>/dev/null || true \
|
||||||
|
&& cp packages/lsp/dist/index.cjs marimo/_lsp/ \
|
||||||
|
&& { [ -d packages/lsp/dist/copilot/dist ] \
|
||||||
|
&& cp -R packages/lsp/dist/copilot/dist/. marimo/_lsp/copilot/ \
|
||||||
|
|| true; }
|
||||||
|
|
||||||
|
# ---- stage 2: wheel ---------------------------------------------------------
|
||||||
|
|
||||||
|
FROM ghcr.io/astral-sh/uv:python3.13-bookworm-slim AS wheel
|
||||||
|
WORKDIR /src
|
||||||
|
COPY --from=fe /src/ /src/
|
||||||
|
RUN uv build --wheel
|
||||||
|
|
||||||
|
# ---- stage 3: runtime -------------------------------------------------------
|
||||||
|
|
||||||
FROM nvidia/cuda:12.6.0-runtime-ubuntu24.04
|
FROM nvidia/cuda:12.6.0-runtime-ubuntu24.04
|
||||||
|
|
||||||
ARG USERNAME=kert
|
ARG USERNAME=kert
|
||||||
@@ -15,32 +80,30 @@ ENV DEBIAN_FRONTEND=noninteractive \
|
|||||||
NVIDIA_VISIBLE_DEVICES=all \
|
NVIDIA_VISIBLE_DEVICES=all \
|
||||||
NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
||||||
|
|
||||||
# System dependencies
|
|
||||||
RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends \
|
RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends \
|
||||||
curl \
|
curl ca-certificates git build-essential \
|
||||||
ca-certificates \
|
|
||||||
git \
|
|
||||||
build-essential \
|
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
# Rename existing ubuntu user/group to kert and fix home ownership
|
|
||||||
RUN groupmod -n ${USERNAME} ubuntu \
|
RUN groupmod -n ${USERNAME} ubuntu \
|
||||||
&& usermod -l ${USERNAME} -d /home/${USERNAME} -m -s /bin/bash ubuntu \
|
&& usermod -l ${USERNAME} -d /home/${USERNAME} -m -s /bin/bash ubuntu \
|
||||||
&& chown -R ${USER_UID}:${USER_GID} /home/${USERNAME}
|
&& chown -R ${USER_UID}:${USER_GID} /home/${USERNAME}
|
||||||
|
|
||||||
# Install uv
|
|
||||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
|
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
|
||||||
COPY --from=ghcr.io/astral-sh/uv:latest /uvx /usr/local/bin/uvx
|
COPY --from=ghcr.io/astral-sh/uv:latest /uvx /usr/local/bin/uvx
|
||||||
|
|
||||||
# Stay as root for rootless Docker compatibility
|
|
||||||
# (root in container = host user in rootless Docker)
|
|
||||||
WORKDIR /home/${USERNAME}
|
WORKDIR /home/${USERNAME}
|
||||||
|
|
||||||
# Initialize uv project and install dependencies
|
# Bring in the fhirworx marimo wheel from the wheel stage.
|
||||||
|
COPY --from=wheel /src/dist/ /tmp/marimo-wheel/
|
||||||
|
|
||||||
|
# Install python + init workspace venv. Use the local wheel for marimo so
|
||||||
|
# the fhirworx frontend ships in the image; no upstream PyPI fetch for
|
||||||
|
# marimo itself.
|
||||||
RUN uv python install ${PYTHON_VERSION} \
|
RUN uv python install ${PYTHON_VERSION} \
|
||||||
&& uv init workspace --python ${PYTHON_VERSION} \
|
&& uv init workspace --python ${PYTHON_VERSION} \
|
||||||
&& cd workspace \
|
&& cd workspace \
|
||||||
&& uv add "marimo[recommended]" polars cudf-polars-cu12 pandas numpy pyarrow \
|
&& MARIMO_WHL=$(ls /tmp/marimo-wheel/marimo-*.whl | head -1) \
|
||||||
|
&& uv add "${MARIMO_WHL}[recommended]" polars cudf-polars-cu12 pandas numpy pyarrow \
|
||||||
"pyiceberg[s3,pyarrow]>=0.7.0" "duckdb>=1.0.0" "narwhals>=1.0.0" "trino>=0.328.0" \
|
"pyiceberg[s3,pyarrow]>=0.7.0" "duckdb>=1.0.0" "narwhals>=1.0.0" "trino>=0.328.0" \
|
||||||
"sqlglot>=26.0.0" \
|
"sqlglot>=26.0.0" \
|
||||||
vega_datasets pyzotero obstore s3fs \
|
vega_datasets pyzotero obstore s3fs \
|
||||||
|
|||||||
147
infra/mail/maddy.conf.tpl
Normal file
@@ -0,0 +1,147 @@
|
|||||||
|
# Maddy mail server config — minimal single-domain setup with DKIM
|
||||||
|
# autosigning. Tracks the shape of the upstream reference config so
|
||||||
|
# upgrades track cleanly.
|
||||||
|
#
|
||||||
|
# Placeholders (substituted by mail-setup.sh via envsubst):
|
||||||
|
# $HOSTNAME -> mail.corwins.media
|
||||||
|
# $PRIMARY_DOMAIN -> corwins.media
|
||||||
|
|
||||||
|
hostname $HOSTNAME
|
||||||
|
$(primary_domain) = $PRIMARY_DOMAIN
|
||||||
|
# Accept mail at both the apex (e.g. git@fhirworx.io for outbound
|
||||||
|
# SMTP submission and DKIM signing) and the host subdomain (e.g.
|
||||||
|
# cmsupdates@mail.fhirworx.io, which is what actually MX-routes to
|
||||||
|
# this droplet when another provider holds the apex MX).
|
||||||
|
$(local_domains) = $(primary_domain) $HOSTNAME
|
||||||
|
|
||||||
|
tls file /data/tls/fullchain.pem /data/tls/privkey.pem
|
||||||
|
|
||||||
|
|
||||||
|
# ── Authentication / storage ─────────────────────────────────────
|
||||||
|
#
|
||||||
|
# Accounts live in credentials.db; IMAP mailboxes live in imapsql.db.
|
||||||
|
# Both are SQLite files under /data and survive container restarts.
|
||||||
|
|
||||||
|
auth.pass_table local_authdb {
|
||||||
|
table sql_table {
|
||||||
|
driver sqlite3
|
||||||
|
dsn credentials.db
|
||||||
|
table_name passwords
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
storage.imapsql local_mailboxes {
|
||||||
|
driver sqlite3
|
||||||
|
dsn imapsql.db
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ── SMTP receive (port 25, from the world) ───────────────────────
|
||||||
|
#
|
||||||
|
# Plain SMTP inbound, STARTTLS opportunistically, no auth (inbound
|
||||||
|
# MTAs shouldn't authenticate). DMARC + DKIM + SPF checks gate the
|
||||||
|
# inbound stream; anything forging our domains is rejected.
|
||||||
|
|
||||||
|
smtp tcp://0.0.0.0:25 {
|
||||||
|
limits {
|
||||||
|
all rate 20 1s
|
||||||
|
all concurrency 10
|
||||||
|
}
|
||||||
|
dmarc yes
|
||||||
|
check {
|
||||||
|
require_mx_record
|
||||||
|
dkim
|
||||||
|
spf
|
||||||
|
}
|
||||||
|
source $(local_domains) {
|
||||||
|
reject 501 5.1.8 "Use Submission (port 587) to send as us"
|
||||||
|
}
|
||||||
|
default_source {
|
||||||
|
destination postmaster $(local_domains) {
|
||||||
|
deliver_to &local_routing
|
||||||
|
}
|
||||||
|
default_destination {
|
||||||
|
reject 550 5.1.1 "User not found"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ── SMTP submission (ports 465 TLS, 587 STARTTLS, authenticated) ──
|
||||||
|
#
|
||||||
|
# Our own outbound send path. Clients (Gitea, Thunderbird, mail-merge
|
||||||
|
# scripts) authenticate against local_authdb, and outbound mail is
|
||||||
|
# DKIM-signed before hitting the remote queue.
|
||||||
|
|
||||||
|
submission tls://0.0.0.0:465 tcp://0.0.0.0:587 {
|
||||||
|
limits {
|
||||||
|
all rate 50 1s
|
||||||
|
}
|
||||||
|
auth &local_authdb
|
||||||
|
source $(local_domains) {
|
||||||
|
check {
|
||||||
|
authorize_sender {
|
||||||
|
prepare_email identity
|
||||||
|
user_to_email identity
|
||||||
|
}
|
||||||
|
}
|
||||||
|
destination postmaster $(local_domains) {
|
||||||
|
deliver_to &local_routing
|
||||||
|
}
|
||||||
|
default_destination {
|
||||||
|
modify {
|
||||||
|
dkim $(primary_domain) default
|
||||||
|
}
|
||||||
|
deliver_to &remote_queue
|
||||||
|
}
|
||||||
|
}
|
||||||
|
default_source {
|
||||||
|
reject 501 5.1.8 "Unknown sender domain"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# Outbound goes through an external SMTP relay (smarthost), not
|
||||||
|
# direct MX — DigitalOcean silently blocks outbound port 25 on new
|
||||||
|
# droplets. The real `target.smtp outbound_smarthost { … }` block is
|
||||||
|
# injected at the top of this file by `stack mail attach-smarthost`.
|
||||||
|
# Until then, the stub below lets maddy start cleanly so inbound,
|
||||||
|
# IMAP, and DKIM key generation all work — outbound just queues
|
||||||
|
# locally and never delivers (intentional).
|
||||||
|
target.smtp outbound_smarthost {
|
||||||
|
targets tcp://127.0.0.1:9
|
||||||
|
auth plain "stub" "stub"
|
||||||
|
}
|
||||||
|
|
||||||
|
target.queue remote_queue {
|
||||||
|
target &outbound_smarthost
|
||||||
|
autogenerated_msg_domain $(primary_domain)
|
||||||
|
bounce {
|
||||||
|
destination postmaster $(local_domains) {
|
||||||
|
deliver_to &local_routing
|
||||||
|
}
|
||||||
|
default_destination {
|
||||||
|
reject 550 5.0.0 "Refusing to send DSNs to non-local addresses"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ── IMAP (port 993 TLS, 143 STARTTLS) ────────────────────────────
|
||||||
|
|
||||||
|
imap tls://0.0.0.0:993 tcp://0.0.0.0:143 {
|
||||||
|
auth &local_authdb
|
||||||
|
storage &local_mailboxes
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ── Local delivery pipeline ──────────────────────────────────────
|
||||||
|
|
||||||
|
msgpipeline local_routing {
|
||||||
|
destination postmaster $(local_domains) {
|
||||||
|
deliver_to &local_mailboxes
|
||||||
|
}
|
||||||
|
default_destination {
|
||||||
|
reject 550 5.1.1 "User doesn't exist"
|
||||||
|
}
|
||||||
|
}
|
||||||
BIN
infra/marimo/theme/frontend/public/android-chrome-192x192.png
Normal file
|
After Width: | Height: | Size: 28 KiB |
BIN
infra/marimo/theme/frontend/public/android-chrome-512x512.png
Normal file
|
After Width: | Height: | Size: 28 KiB |
BIN
infra/marimo/theme/frontend/public/apple-touch-icon.png
Normal file
|
After Width: | Height: | Size: 18 KiB |
BIN
infra/marimo/theme/frontend/public/favicon-16x16.png
Normal file
|
After Width: | Height: | Size: 741 B |
BIN
infra/marimo/theme/frontend/public/favicon-32x32.png
Normal file
|
After Width: | Height: | Size: 2.0 KiB |
3
infra/marimo/theme/frontend/public/favicon.svg
Normal file
|
After Width: | Height: | Size: 29 KiB |
BIN
infra/marimo/theme/frontend/public/logo.png
Normal file
|
After Width: | Height: | Size: 28 KiB |
148
infra/marimo/theme/frontend/src/css/fhirworx.css
Normal file
@@ -0,0 +1,148 @@
|
|||||||
|
/* fhirworx theme for marimo — ProPublica-editorial × Federal-Register-civic.
|
||||||
|
*
|
||||||
|
* Keep overrides minimal and load AFTER marimo's globals.css so our
|
||||||
|
* cascade wins without !important. Variables first, then component
|
||||||
|
* touch-ups. Matches the Gitea theme's variable-completeness rule:
|
||||||
|
* define everything marimo references or the cascade falls through.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/* Force light mode. The Radix color scales that marimo uses for the home
|
||||||
|
* page (--blue-1, --blue-2, --slate-3, --red-1, …) resolve to dark values
|
||||||
|
* when the browser reports prefers-color-scheme: dark. Without this lock,
|
||||||
|
* the cream --background we set below fights the dark Radix panels and
|
||||||
|
* produces "slate text on dark-blue card" — unreadable. `only light` tells
|
||||||
|
* the UA to ignore the system pref entirely, so light-dark() and the
|
||||||
|
* radix @import'ed -dark files don't kick in. */
|
||||||
|
:root {
|
||||||
|
color-scheme: only light;
|
||||||
|
}
|
||||||
|
|
||||||
|
:root,
|
||||||
|
.marimo {
|
||||||
|
/* Typography: editorial serif + Federal-Register humanist monospace */
|
||||||
|
--marimo-heading-font: "Source Serif 4", "Source Serif Pro", ui-serif, Georgia, serif;
|
||||||
|
--marimo-text-font: "Source Serif 4", "Source Serif Pro", ui-serif, Georgia, serif;
|
||||||
|
--marimo-monospace-font: "IBM Plex Mono", "Berkeley Mono", ui-monospace, SFMono-Regular, Menlo, monospace;
|
||||||
|
|
||||||
|
/* Paper stock + ink */
|
||||||
|
--fw-paper: #F6F2E8; /* warm cream */
|
||||||
|
--fw-paper-sunk: #EDE7D6;
|
||||||
|
--fw-ink: #141B2D; /* midnight navy */
|
||||||
|
--fw-ink-soft: #2A3349;
|
||||||
|
--fw-ink-faint: #5A6378;
|
||||||
|
--fw-rule: #B8B3A4; /* beige-dark — warm border that reads against cream */
|
||||||
|
--fw-accent: #9A2A2A; /* signal red, sparingly */
|
||||||
|
--fw-accent-soft: #D9AD5C; /* brass */
|
||||||
|
|
||||||
|
/* Override marimo's palette in both schemes */
|
||||||
|
--background: var(--fw-paper);
|
||||||
|
--foreground: var(--fw-ink);
|
||||||
|
--muted: var(--fw-paper-sunk);
|
||||||
|
--muted-foreground: var(--fw-ink-faint);
|
||||||
|
--popover: var(--fw-paper);
|
||||||
|
--popover-foreground: var(--fw-ink);
|
||||||
|
--card: var(--fw-paper);
|
||||||
|
--card-foreground: var(--fw-ink);
|
||||||
|
--border: var(--fw-rule);
|
||||||
|
--input: var(--fw-rule);
|
||||||
|
--primary: var(--fw-ink);
|
||||||
|
--primary-foreground: var(--fw-paper);
|
||||||
|
--secondary: var(--fw-paper-sunk);
|
||||||
|
--secondary-foreground: var(--fw-ink);
|
||||||
|
--accent: var(--fw-accent-soft);
|
||||||
|
--accent-foreground: var(--fw-ink);
|
||||||
|
--destructive: var(--fw-accent);
|
||||||
|
--destructive-foreground: var(--fw-paper);
|
||||||
|
|
||||||
|
--radius: 2px; /* civic forms, not rounded chrome */
|
||||||
|
--markdown-max-width: 72ch; /* ProPublica measure */
|
||||||
|
|
||||||
|
/* Radix color scales used directly by marimo's home page UI. The stock
|
||||||
|
* scales don't fit our cream/navy palette; these give notebook panels
|
||||||
|
* and hover states a coherent look. Keep the variable names because
|
||||||
|
* the source references them as `bg-(--blue-2)`, `divide-(--slate-3)`,
|
||||||
|
* etc. via Tailwind arbitrary-value syntax. */
|
||||||
|
--blue-1: var(--fw-paper); /* create-new card rest bg */
|
||||||
|
--blue-2: #EDE1C4; /* card/row hover — warm straw */
|
||||||
|
--blue-3: #E5D6AE;
|
||||||
|
--slate-1: var(--fw-paper);
|
||||||
|
--slate-2: var(--fw-paper-sunk);
|
||||||
|
--slate-3: var(--fw-rule); /* divider between notebook rows */
|
||||||
|
--slate-4: var(--fw-rule);
|
||||||
|
--red-1: #F4E6E0; /* shutdown button hover bg */
|
||||||
|
--red-2: #E8CCC1;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Body & headings — serif first, tight leading, generous measure */
|
||||||
|
body, .marimo {
|
||||||
|
font-family: var(--marimo-text-font);
|
||||||
|
font-feature-settings: "onum" 1, "ss01" 1; /* oldstyle figures, stylistic set 1 */
|
||||||
|
letter-spacing: 0;
|
||||||
|
color: var(--fw-ink);
|
||||||
|
background: var(--fw-paper);
|
||||||
|
}
|
||||||
|
|
||||||
|
.marimo h1, .marimo h2, .marimo h3, .marimo h4, .marimo h5, .marimo h6 {
|
||||||
|
font-family: var(--marimo-heading-font);
|
||||||
|
font-weight: 700;
|
||||||
|
letter-spacing: -0.01em;
|
||||||
|
color: var(--fw-ink);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Icons: Tabler defaults to stroke-width 2 — thin them for editorial feel.
|
||||||
|
* Scoped to .tabler-icon so our utility icons elsewhere aren't affected. */
|
||||||
|
.tabler-icon,
|
||||||
|
svg.tabler-icon {
|
||||||
|
stroke-width: 1.5;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Rules & borders — single hairline, warm. Kill the subtle gray-on-white
|
||||||
|
* that looks muddy against cream. */
|
||||||
|
.marimo hr,
|
||||||
|
.marimo [role="separator"] {
|
||||||
|
border-color: var(--fw-rule);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Code cells — no rounded corners, let the typewriter feel breathe */
|
||||||
|
.marimo .cm-editor {
|
||||||
|
font-family: var(--marimo-monospace-font);
|
||||||
|
border-radius: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Buttons — flatten, no gradients, clear hit target */
|
||||||
|
.marimo button,
|
||||||
|
.marimo [role="button"] {
|
||||||
|
border-radius: var(--radius);
|
||||||
|
font-weight: 500;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Links — underline on hover only, ink color always */
|
||||||
|
.marimo a {
|
||||||
|
color: var(--fw-ink);
|
||||||
|
text-decoration-color: var(--fw-rule);
|
||||||
|
text-underline-offset: 2px;
|
||||||
|
}
|
||||||
|
.marimo a:hover {
|
||||||
|
text-decoration-color: var(--fw-ink);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Home-page panels. Notebook list rows use `hover:bg-(--blue-2)` and
|
||||||
|
* `hover:text-primary` — that means text color on hover switches to
|
||||||
|
* --primary, which we set to --fw-ink. Make sure every text color in
|
||||||
|
* those rows resolves against our palette, not an inherited slate. */
|
||||||
|
.marimo a,
|
||||||
|
.marimo a:visited,
|
||||||
|
.marimo .text-muted-foreground,
|
||||||
|
.marimo .text-primary {
|
||||||
|
color: var(--fw-ink);
|
||||||
|
}
|
||||||
|
.marimo .text-muted-foreground {
|
||||||
|
color: var(--fw-ink-faint);
|
||||||
|
}
|
||||||
|
/* "Create a new notebook" hero card — marimo classes it `bg-(--blue-1)`
|
||||||
|
* with an accent shadow. Keep it on cream with a subtle brass border so
|
||||||
|
* the eye lands there without shouting. */
|
||||||
|
.marimo [class*="bg-(--blue-1)"] {
|
||||||
|
background: var(--fw-paper);
|
||||||
|
border-color: var(--fw-accent-soft);
|
||||||
|
}
|
||||||
514
infra/marimo/theme/frontend/src/lucide-shim.tsx
Normal file
@@ -0,0 +1,514 @@
|
|||||||
|
// Auto-generated: Tabler Icons as a drop-in for lucide-react.
|
||||||
|
// Build-time replacement; do not edit by hand — see gen-lucide-shim.py
|
||||||
|
import type { ComponentType, SVGProps } from 'react';
|
||||||
|
import {
|
||||||
|
IconActivity,
|
||||||
|
IconAlertCircle,
|
||||||
|
IconAlertOctagon,
|
||||||
|
IconAlertTriangle,
|
||||||
|
IconAlignCenter,
|
||||||
|
IconAlignJustified,
|
||||||
|
IconAlignLeft,
|
||||||
|
IconAlignRight,
|
||||||
|
IconAlphabetLatin,
|
||||||
|
IconArrowBackUp,
|
||||||
|
IconArrowBarRight,
|
||||||
|
IconArrowBarToDown,
|
||||||
|
IconArrowBarToRight,
|
||||||
|
IconArrowBarToUp,
|
||||||
|
IconArrowDown,
|
||||||
|
IconArrowLeft,
|
||||||
|
IconArrowRight,
|
||||||
|
IconArrowsMaximize,
|
||||||
|
IconArrowsShuffle,
|
||||||
|
IconArrowsSort,
|
||||||
|
IconArrowsUpDown,
|
||||||
|
IconArticle,
|
||||||
|
IconAt,
|
||||||
|
IconBan,
|
||||||
|
IconBaselineDensitySmall,
|
||||||
|
IconBinary,
|
||||||
|
IconBolt,
|
||||||
|
IconBoltOff,
|
||||||
|
IconBook,
|
||||||
|
IconBookmark,
|
||||||
|
IconBox,
|
||||||
|
IconBraces,
|
||||||
|
IconBrackets,
|
||||||
|
IconBrain,
|
||||||
|
IconBrandGithub,
|
||||||
|
IconBrandYoutube,
|
||||||
|
IconBrush,
|
||||||
|
IconBug,
|
||||||
|
IconCalendar,
|
||||||
|
IconCalendarTime,
|
||||||
|
IconChartArea,
|
||||||
|
IconChartBar,
|
||||||
|
IconChartDots,
|
||||||
|
IconChartLine,
|
||||||
|
IconChartPie,
|
||||||
|
IconCheck,
|
||||||
|
IconChevronDown,
|
||||||
|
IconChevronLeft,
|
||||||
|
IconChevronRight,
|
||||||
|
IconChevronUp,
|
||||||
|
IconChevronsDown,
|
||||||
|
IconChevronsLeft,
|
||||||
|
IconChevronsRight,
|
||||||
|
IconChevronsUp,
|
||||||
|
IconCircle,
|
||||||
|
IconCircleCheck,
|
||||||
|
IconCircleChevronDown,
|
||||||
|
IconCircleChevronRight,
|
||||||
|
IconCircleDashed,
|
||||||
|
IconCirclePlus,
|
||||||
|
IconCircleX,
|
||||||
|
IconClipboardCopy,
|
||||||
|
IconClipboardText,
|
||||||
|
IconClock,
|
||||||
|
IconCloudDownload,
|
||||||
|
IconCode,
|
||||||
|
IconColumns,
|
||||||
|
IconColumns2,
|
||||||
|
IconCommand,
|
||||||
|
IconConfetti,
|
||||||
|
IconCopy,
|
||||||
|
IconCopyOff,
|
||||||
|
IconCornerLeftUp,
|
||||||
|
IconCpu,
|
||||||
|
IconCrosshair,
|
||||||
|
IconCursorText,
|
||||||
|
IconDatabase,
|
||||||
|
IconDeviceDesktop,
|
||||||
|
IconDeviceFloppy,
|
||||||
|
IconDeviceSdCard,
|
||||||
|
IconDiamond,
|
||||||
|
IconDots,
|
||||||
|
IconDotsCircleHorizontal,
|
||||||
|
IconDotsVertical,
|
||||||
|
IconDownload,
|
||||||
|
IconEdit,
|
||||||
|
IconEqual,
|
||||||
|
IconEraser,
|
||||||
|
IconExternalLink,
|
||||||
|
IconEye,
|
||||||
|
IconEyeOff,
|
||||||
|
IconFile,
|
||||||
|
IconFileCode,
|
||||||
|
IconFileMusic,
|
||||||
|
IconFilePencil,
|
||||||
|
IconFilePlus,
|
||||||
|
IconFileSpreadsheet,
|
||||||
|
IconFileSymlink,
|
||||||
|
IconFileText,
|
||||||
|
IconFileTypography,
|
||||||
|
IconFiles,
|
||||||
|
IconFilter,
|
||||||
|
IconFilterOff,
|
||||||
|
IconFilterPlus,
|
||||||
|
IconFlask,
|
||||||
|
IconFolder,
|
||||||
|
IconFolderCog,
|
||||||
|
IconFolderDown,
|
||||||
|
IconFolderPlus,
|
||||||
|
IconFolders,
|
||||||
|
IconGrid3x3,
|
||||||
|
IconGripHorizontal,
|
||||||
|
IconGripVertical,
|
||||||
|
IconHash,
|
||||||
|
IconHelpCircle,
|
||||||
|
IconHistory,
|
||||||
|
IconHome,
|
||||||
|
IconHourglass,
|
||||||
|
IconInfoCircle,
|
||||||
|
IconJson,
|
||||||
|
IconKey,
|
||||||
|
IconKeyboard,
|
||||||
|
IconLayout,
|
||||||
|
IconLayoutBoard,
|
||||||
|
IconLayoutRows,
|
||||||
|
IconLayoutSidebar,
|
||||||
|
IconLayoutSidebarRight,
|
||||||
|
IconLetterCase,
|
||||||
|
IconLink,
|
||||||
|
IconList,
|
||||||
|
IconListNumbers,
|
||||||
|
IconListTree,
|
||||||
|
IconLoader,
|
||||||
|
IconLoader2,
|
||||||
|
IconLock,
|
||||||
|
IconMail,
|
||||||
|
IconMapPin,
|
||||||
|
IconMathFunction,
|
||||||
|
IconMenu2,
|
||||||
|
IconMessageCircle,
|
||||||
|
IconMessageCircleQuestion,
|
||||||
|
IconMessages,
|
||||||
|
IconMinus,
|
||||||
|
IconMoodHappy,
|
||||||
|
IconNetwork,
|
||||||
|
IconNotebook,
|
||||||
|
IconPackage,
|
||||||
|
IconPackageOff,
|
||||||
|
IconPaperclip,
|
||||||
|
IconPencil,
|
||||||
|
IconPhoto,
|
||||||
|
IconPin,
|
||||||
|
IconPinnedOff,
|
||||||
|
IconPlayerPlay,
|
||||||
|
IconPlayerSkipForward,
|
||||||
|
IconPlayerStop,
|
||||||
|
IconPlayerTrackNext,
|
||||||
|
IconPlug,
|
||||||
|
IconPlus,
|
||||||
|
IconPointer,
|
||||||
|
IconPointerShare,
|
||||||
|
IconPower,
|
||||||
|
IconPresentation,
|
||||||
|
IconRefresh,
|
||||||
|
IconRegex,
|
||||||
|
IconRobot,
|
||||||
|
IconRotate,
|
||||||
|
IconRotateClockwise,
|
||||||
|
IconRuler,
|
||||||
|
IconSchool,
|
||||||
|
IconScissors,
|
||||||
|
IconSearch,
|
||||||
|
IconSelector,
|
||||||
|
IconSend,
|
||||||
|
IconSettings,
|
||||||
|
IconShare,
|
||||||
|
IconShieldCheck,
|
||||||
|
IconSitemap,
|
||||||
|
IconSparkles,
|
||||||
|
IconSquare,
|
||||||
|
IconSquareArrowRight,
|
||||||
|
IconSquareCheck,
|
||||||
|
IconSquareDashed,
|
||||||
|
IconSquareLetterM,
|
||||||
|
IconSquarePlus,
|
||||||
|
IconStack,
|
||||||
|
IconStackPush,
|
||||||
|
IconSum,
|
||||||
|
IconTable,
|
||||||
|
IconTerminal,
|
||||||
|
IconTerminal2,
|
||||||
|
IconTextCaption,
|
||||||
|
IconTextWrap,
|
||||||
|
IconToggleLeft,
|
||||||
|
IconTool,
|
||||||
|
IconTrash,
|
||||||
|
IconTriangle,
|
||||||
|
IconTypography,
|
||||||
|
IconUnlink,
|
||||||
|
IconUpload,
|
||||||
|
IconUsers,
|
||||||
|
IconUsersGroup,
|
||||||
|
IconVariable,
|
||||||
|
IconVideo,
|
||||||
|
IconWall,
|
||||||
|
IconWifi,
|
||||||
|
IconWifiOff,
|
||||||
|
IconWorld,
|
||||||
|
IconX,
|
||||||
|
IconZoomCheck,
|
||||||
|
IconZoomCode,
|
||||||
|
type IconProps,
|
||||||
|
} from '@tabler/icons-react';
|
||||||
|
|
||||||
|
// Prop shapes mirror lucide's public types so Omit<LucideProps, 'ref'>
|
||||||
|
// and similar patterns in upstream marimo code keep typechecking.
|
||||||
|
export type LucideProps = SVGProps<SVGSVGElement> & {
|
||||||
|
size?: number | string;
|
||||||
|
absoluteStrokeWidth?: boolean;
|
||||||
|
};
|
||||||
|
export type LucideIcon = ComponentType<LucideProps & IconProps>;
|
||||||
|
|
||||||
|
export const ActivityIcon: LucideIcon = IconActivity as LucideIcon;
|
||||||
|
export const AlertCircle: LucideIcon = IconAlertCircle as LucideIcon;
|
||||||
|
export const AlertCircleIcon: LucideIcon = IconAlertCircle as LucideIcon;
|
||||||
|
export const AlertOctagonIcon: LucideIcon = IconAlertOctagon as LucideIcon;
|
||||||
|
export const AlertTriangle: LucideIcon = IconAlertTriangle as LucideIcon;
|
||||||
|
export const AlertTriangleIcon: LucideIcon = IconAlertTriangle as LucideIcon;
|
||||||
|
export const AlignCenterVerticalIcon: LucideIcon = IconAlignCenter as LucideIcon;
|
||||||
|
export const AlignEndVerticalIcon: LucideIcon = IconAlignRight as LucideIcon;
|
||||||
|
export const AlignHorizontalSpaceAroundIcon: LucideIcon = IconAlignJustified as LucideIcon;
|
||||||
|
export const AlignJustifyIcon: LucideIcon = IconAlignJustified as LucideIcon;
|
||||||
|
export const AlignStartVerticalIcon: LucideIcon = IconAlignLeft as LucideIcon;
|
||||||
|
export const AreaChartIcon: LucideIcon = IconChartArea as LucideIcon;
|
||||||
|
export const ArrowDownIcon: LucideIcon = IconArrowDown as LucideIcon;
|
||||||
|
export const ArrowDownToLineIcon: LucideIcon = IconArrowBarToDown as LucideIcon;
|
||||||
|
export const ArrowDownWideNarrowIcon: LucideIcon = IconArrowsSort as LucideIcon;
|
||||||
|
export const ArrowLeftIcon: LucideIcon = IconArrowLeft as LucideIcon;
|
||||||
|
export const ArrowRightFromLineIcon: LucideIcon = IconArrowBarRight as LucideIcon;
|
||||||
|
export const ArrowRightIcon: LucideIcon = IconArrowRight as LucideIcon;
|
||||||
|
export const ArrowRightSquareIcon: LucideIcon = IconSquareArrowRight as LucideIcon;
|
||||||
|
export const ArrowRightToLineIcon: LucideIcon = IconArrowBarToRight as LucideIcon;
|
||||||
|
export const ArrowUpDownIcon: LucideIcon = IconArrowsUpDown as LucideIcon;
|
||||||
|
export const ArrowUpNarrowWideIcon: LucideIcon = IconArrowsSort as LucideIcon;
|
||||||
|
export const ArrowUpToLineIcon: LucideIcon = IconArrowBarToUp as LucideIcon;
|
||||||
|
export const ArrowUpWideNarrowIcon: LucideIcon = IconArrowsSort as LucideIcon;
|
||||||
|
export const AtSignIcon: LucideIcon = IconAt as LucideIcon;
|
||||||
|
export const BanIcon: LucideIcon = IconBan as LucideIcon;
|
||||||
|
export const BarChart2Icon: LucideIcon = IconChartBar as LucideIcon;
|
||||||
|
export const BarChartBigIcon: LucideIcon = IconChartBar as LucideIcon;
|
||||||
|
export const BarChartIcon: LucideIcon = IconChartBar as LucideIcon;
|
||||||
|
export const BaselineIcon: LucideIcon = IconBaselineDensitySmall as LucideIcon;
|
||||||
|
export const BetweenHorizontalStartIcon: LucideIcon = IconLayoutRows as LucideIcon;
|
||||||
|
export const BinaryIcon: LucideIcon = IconBinary as LucideIcon;
|
||||||
|
export const BookMarkedIcon: LucideIcon = IconBookmark as LucideIcon;
|
||||||
|
export const BookOpenIcon: LucideIcon = IconBook as LucideIcon;
|
||||||
|
export const BookPlusIcon: LucideIcon = IconBook as LucideIcon;
|
||||||
|
export const BookTextIcon: LucideIcon = IconBook as LucideIcon;
|
||||||
|
export const BotIcon: LucideIcon = IconRobot as LucideIcon;
|
||||||
|
export const BotMessageSquareIcon: LucideIcon = IconRobot as LucideIcon;
|
||||||
|
export const BoxIcon: LucideIcon = IconBox as LucideIcon;
|
||||||
|
export const BracesIcon: LucideIcon = IconBraces as LucideIcon;
|
||||||
|
export const BracketsIcon: LucideIcon = IconBrackets as LucideIcon;
|
||||||
|
export const BrainIcon: LucideIcon = IconBrain as LucideIcon;
|
||||||
|
export const BrickWallIcon: LucideIcon = IconWall as LucideIcon;
|
||||||
|
export const BugPlayIcon: LucideIcon = IconBug as LucideIcon;
|
||||||
|
export const CalendarClockIcon: LucideIcon = IconCalendarTime as LucideIcon;
|
||||||
|
export const CalendarIcon: LucideIcon = IconCalendar as LucideIcon;
|
||||||
|
export const CaseSensitiveIcon: LucideIcon = IconLetterCase as LucideIcon;
|
||||||
|
export const ChartColumn: LucideIcon = IconChartBar as LucideIcon;
|
||||||
|
export const ChartColumnIcon: LucideIcon = IconChartBar as LucideIcon;
|
||||||
|
export const ChartNoAxesColumn: LucideIcon = IconChartBar as LucideIcon;
|
||||||
|
export const ChartPieIcon: LucideIcon = IconChartPie as LucideIcon;
|
||||||
|
export const ChartScatterIcon: LucideIcon = IconChartDots as LucideIcon;
|
||||||
|
export const ChartSplineIcon: LucideIcon = IconChartLine as LucideIcon;
|
||||||
|
export const Check: LucideIcon = IconCheck as LucideIcon;
|
||||||
|
export const CheckCircle2Icon: LucideIcon = IconCircleCheck as LucideIcon;
|
||||||
|
export const CheckCircleIcon: LucideIcon = IconCircleCheck as LucideIcon;
|
||||||
|
export const CheckIcon: LucideIcon = IconCheck as LucideIcon;
|
||||||
|
export const CheckSquareIcon: LucideIcon = IconSquareCheck as LucideIcon;
|
||||||
|
export const ChevronDown: LucideIcon = IconChevronDown as LucideIcon;
|
||||||
|
export const ChevronDownCircleIcon: LucideIcon = IconCircleChevronDown as LucideIcon;
|
||||||
|
export const ChevronDownIcon: LucideIcon = IconChevronDown as LucideIcon;
|
||||||
|
export const ChevronLeft: LucideIcon = IconChevronLeft as LucideIcon;
|
||||||
|
export const ChevronLeftIcon: LucideIcon = IconChevronLeft as LucideIcon;
|
||||||
|
export const ChevronRight: LucideIcon = IconChevronRight as LucideIcon;
|
||||||
|
export const ChevronRightCircleIcon: LucideIcon = IconCircleChevronRight as LucideIcon;
|
||||||
|
export const ChevronRightIcon: LucideIcon = IconChevronRight as LucideIcon;
|
||||||
|
export const ChevronUp: LucideIcon = IconChevronUp as LucideIcon;
|
||||||
|
export const ChevronUpIcon: LucideIcon = IconChevronUp as LucideIcon;
|
||||||
|
export const ChevronsDownIcon: LucideIcon = IconChevronsDown as LucideIcon;
|
||||||
|
export const ChevronsDownUpIcon: LucideIcon = IconSelector as LucideIcon;
|
||||||
|
export const ChevronsLeft: LucideIcon = IconChevronsLeft as LucideIcon;
|
||||||
|
export const ChevronsRight: LucideIcon = IconChevronsRight as LucideIcon;
|
||||||
|
export const ChevronsUpDown: LucideIcon = IconSelector as LucideIcon;
|
||||||
|
export const ChevronsUpDownIcon: LucideIcon = IconSelector as LucideIcon;
|
||||||
|
export const ChevronsUpIcon: LucideIcon = IconChevronsUp as LucideIcon;
|
||||||
|
export const Circle: LucideIcon = IconCircle as LucideIcon;
|
||||||
|
export const CircleCheck: LucideIcon = IconCircleCheck as LucideIcon;
|
||||||
|
export const CircleCheckIcon: LucideIcon = IconCircleCheck as LucideIcon;
|
||||||
|
export const CircleEllipsis: LucideIcon = IconDotsCircleHorizontal as LucideIcon;
|
||||||
|
export const CircleHelpIcon: LucideIcon = IconHelpCircle as LucideIcon;
|
||||||
|
export const CircleIcon: LucideIcon = IconCircle as LucideIcon;
|
||||||
|
export const CirclePlayIcon: LucideIcon = IconPlayerPlay as LucideIcon;
|
||||||
|
export const CircleX: LucideIcon = IconCircleX as LucideIcon;
|
||||||
|
export const ClipboardCopyIcon: LucideIcon = IconClipboardCopy as LucideIcon;
|
||||||
|
export const ClipboardPasteIcon: LucideIcon = IconClipboardText as LucideIcon;
|
||||||
|
export const ClockIcon: LucideIcon = IconClock as LucideIcon;
|
||||||
|
export const Code2Icon: LucideIcon = IconCode as LucideIcon;
|
||||||
|
export const CodeIcon: LucideIcon = IconCode as LucideIcon;
|
||||||
|
export const Cog: LucideIcon = IconSettings as LucideIcon;
|
||||||
|
export const Columns2Icon: LucideIcon = IconColumns2 as LucideIcon;
|
||||||
|
export const ColumnsIcon: LucideIcon = IconColumns as LucideIcon;
|
||||||
|
export const CombineIcon: LucideIcon = IconStackPush as LucideIcon;
|
||||||
|
export const CommandIcon: LucideIcon = IconCommand as LucideIcon;
|
||||||
|
export const Copy: LucideIcon = IconCopy as LucideIcon;
|
||||||
|
export const CopyIcon: LucideIcon = IconCopy as LucideIcon;
|
||||||
|
export const CopyMinusIcon: LucideIcon = IconCopy as LucideIcon;
|
||||||
|
export const CopySlashIcon: LucideIcon = IconCopyOff as LucideIcon;
|
||||||
|
export const CornerLeftUp: LucideIcon = IconCornerLeftUp as LucideIcon;
|
||||||
|
export const CpuIcon: LucideIcon = IconCpu as LucideIcon;
|
||||||
|
export const CrosshairIcon: LucideIcon = IconCrosshair as LucideIcon;
|
||||||
|
export const CurlyBracesIcon: LucideIcon = IconBraces as LucideIcon;
|
||||||
|
export const DatabaseIcon: LucideIcon = IconDatabase as LucideIcon;
|
||||||
|
export const DatabaseZap: LucideIcon = IconDatabase as LucideIcon;
|
||||||
|
export const DatabaseZapIcon: LucideIcon = IconDatabase as LucideIcon;
|
||||||
|
export const DiamondPlusIcon: LucideIcon = IconDiamond as LucideIcon;
|
||||||
|
export const DownloadCloudIcon: LucideIcon = IconCloudDownload as LucideIcon;
|
||||||
|
export const DownloadIcon: LucideIcon = IconDownload as LucideIcon;
|
||||||
|
export const Edit3Icon: LucideIcon = IconEdit as LucideIcon;
|
||||||
|
export const EditIcon: LucideIcon = IconEdit as LucideIcon;
|
||||||
|
export const EllipsisIcon: LucideIcon = IconDots as LucideIcon;
|
||||||
|
export const EraserIcon: LucideIcon = IconEraser as LucideIcon;
|
||||||
|
export const ExpandIcon: LucideIcon = IconArrowsMaximize as LucideIcon;
|
||||||
|
export const ExternalLinkIcon: LucideIcon = IconExternalLink as LucideIcon;
|
||||||
|
export const EyeIcon: LucideIcon = IconEye as LucideIcon;
|
||||||
|
export const EyeOffIcon: LucideIcon = IconEyeOff as LucideIcon;
|
||||||
|
export const FastForwardIcon: LucideIcon = IconPlayerTrackNext as LucideIcon;
|
||||||
|
export const FileAudio2Icon: LucideIcon = IconFileMusic as LucideIcon;
|
||||||
|
export const FileAudioIcon: LucideIcon = IconFileMusic as LucideIcon;
|
||||||
|
export const FileCodeIcon: LucideIcon = IconFileCode as LucideIcon;
|
||||||
|
export const FileIcon: LucideIcon = IconFile as LucideIcon;
|
||||||
|
export const FileImageIcon: LucideIcon = IconPhoto as LucideIcon;
|
||||||
|
export const FileJsonIcon: LucideIcon = IconJson as LucideIcon;
|
||||||
|
export const FilePenIcon: LucideIcon = IconFilePencil as LucideIcon;
|
||||||
|
export const FilePlus2Icon: LucideIcon = IconFilePlus as LucideIcon;
|
||||||
|
export const FileSpreadsheetIcon: LucideIcon = IconFileSpreadsheet as LucideIcon;
|
||||||
|
export const FileSymlink: LucideIcon = IconFileSymlink as LucideIcon;
|
||||||
|
export const FileTextIcon: LucideIcon = IconFileText as LucideIcon;
|
||||||
|
export const FileVideoCameraIcon: LucideIcon = IconFileTypography as LucideIcon;
|
||||||
|
export const FileVideoIcon: LucideIcon = IconVideo as LucideIcon;
|
||||||
|
export const Files: LucideIcon = IconFiles as LucideIcon;
|
||||||
|
export const FilterIcon: LucideIcon = IconFilter as LucideIcon;
|
||||||
|
export const FilterX: LucideIcon = IconFilterOff as LucideIcon;
|
||||||
|
export const FlaskConicalIcon: LucideIcon = IconFlask as LucideIcon;
|
||||||
|
export const FolderArchiveIcon: LucideIcon = IconFolder as LucideIcon;
|
||||||
|
export const FolderCog2: LucideIcon = IconFolderCog as LucideIcon;
|
||||||
|
export const FolderDownIcon: LucideIcon = IconFolderDown as LucideIcon;
|
||||||
|
export const FolderIcon: LucideIcon = IconFolder as LucideIcon;
|
||||||
|
export const FolderPlusIcon: LucideIcon = IconFolderPlus as LucideIcon;
|
||||||
|
export const FolderTreeIcon: LucideIcon = IconFolders as LucideIcon;
|
||||||
|
export const FunctionSquareIcon: LucideIcon = IconMathFunction as LucideIcon;
|
||||||
|
export const FunnelPlusIcon: LucideIcon = IconFilterPlus as LucideIcon;
|
||||||
|
export const GithubIcon: LucideIcon = IconBrandGithub as LucideIcon;
|
||||||
|
export const GlobeIcon: LucideIcon = IconWorld as LucideIcon;
|
||||||
|
export const GraduationCapIcon: LucideIcon = IconSchool as LucideIcon;
|
||||||
|
export const Grid3x3Icon: LucideIcon = IconGrid3x3 as LucideIcon;
|
||||||
|
export const GridIcon: LucideIcon = IconGrid3x3 as LucideIcon;
|
||||||
|
export const GripHorizontal: LucideIcon = IconGripHorizontal as LucideIcon;
|
||||||
|
export const GripHorizontalIcon: LucideIcon = IconGripHorizontal as LucideIcon;
|
||||||
|
export const GripVerticalIcon: LucideIcon = IconGripVertical as LucideIcon;
|
||||||
|
export const GroupIcon: LucideIcon = IconUsersGroup as LucideIcon;
|
||||||
|
export const HardDrive: LucideIcon = IconDeviceSdCard as LucideIcon;
|
||||||
|
export const HardDriveDownloadIcon: LucideIcon = IconDeviceSdCard as LucideIcon;
|
||||||
|
export const HardDriveIcon: LucideIcon = IconDeviceSdCard as LucideIcon;
|
||||||
|
export const HashIcon: LucideIcon = IconHash as LucideIcon;
|
||||||
|
export const HatGlasses: LucideIcon = IconMoodHappy as LucideIcon;
|
||||||
|
export const HelpCircleIcon: LucideIcon = IconHelpCircle as LucideIcon;
|
||||||
|
export const HistoryIcon: LucideIcon = IconHistory as LucideIcon;
|
||||||
|
export const Home: LucideIcon = IconHome as LucideIcon;
|
||||||
|
export const HomeIcon: LucideIcon = IconHome as LucideIcon;
|
||||||
|
export const HourglassIcon: LucideIcon = IconHourglass as LucideIcon;
|
||||||
|
export const ImageIcon: LucideIcon = IconPhoto as LucideIcon;
|
||||||
|
export const Info: LucideIcon = IconInfoCircle as LucideIcon;
|
||||||
|
export const InfoIcon: LucideIcon = IconInfoCircle as LucideIcon;
|
||||||
|
export const KeyIcon: LucideIcon = IconKey as LucideIcon;
|
||||||
|
export const KeyRoundIcon: LucideIcon = IconKey as LucideIcon;
|
||||||
|
export const KeyboardIcon: LucideIcon = IconKeyboard as LucideIcon;
|
||||||
|
export const LayersIcon: LucideIcon = IconStack as LucideIcon;
|
||||||
|
export const LayoutIcon: LucideIcon = IconLayout as LucideIcon;
|
||||||
|
export const LayoutTemplateIcon: LucideIcon = IconLayoutBoard as LucideIcon;
|
||||||
|
export const LineChartIcon: LucideIcon = IconChartLine as LucideIcon;
|
||||||
|
export const LinkIcon: LucideIcon = IconLink as LucideIcon;
|
||||||
|
export const ListFilterIcon: LucideIcon = IconFilter as LucideIcon;
|
||||||
|
export const ListFilterPlusIcon: LucideIcon = IconFilterPlus as LucideIcon;
|
||||||
|
export const ListIcon: LucideIcon = IconList as LucideIcon;
|
||||||
|
export const ListOrderedIcon: LucideIcon = IconListNumbers as LucideIcon;
|
||||||
|
export const ListTreeIcon: LucideIcon = IconListTree as LucideIcon;
|
||||||
|
export const Loader2: LucideIcon = IconLoader2 as LucideIcon;
|
||||||
|
export const Loader2Icon: LucideIcon = IconLoader2 as LucideIcon;
|
||||||
|
export const LoaderCircle: LucideIcon = IconLoader as LucideIcon;
|
||||||
|
export const LockIcon: LucideIcon = IconLock as LucideIcon;
|
||||||
|
export const Mail: LucideIcon = IconMail as LucideIcon;
|
||||||
|
export const MapPinIcon: LucideIcon = IconMapPin as LucideIcon;
|
||||||
|
export const MemoryStickIcon: LucideIcon = IconDeviceSdCard as LucideIcon;
|
||||||
|
export const MenuIcon: LucideIcon = IconMenu2 as LucideIcon;
|
||||||
|
export const MessageCircleIcon: LucideIcon = IconMessageCircle as LucideIcon;
|
||||||
|
export const MessageCircleQuestionIcon: LucideIcon = IconMessageCircleQuestion as LucideIcon;
|
||||||
|
export const MessagesSquareIcon: LucideIcon = IconMessages as LucideIcon;
|
||||||
|
export const MicrochipIcon: LucideIcon = IconCpu as LucideIcon;
|
||||||
|
export const MinusIcon: LucideIcon = IconMinus as LucideIcon;
|
||||||
|
export const MonitorIcon: LucideIcon = IconDeviceDesktop as LucideIcon;
|
||||||
|
export const MoreHorizontal: LucideIcon = IconDots as LucideIcon;
|
||||||
|
export const MoreHorizontalIcon: LucideIcon = IconDots as LucideIcon;
|
||||||
|
export const MoreVerticalIcon: LucideIcon = IconDotsVertical as LucideIcon;
|
||||||
|
export const MousePointerSquareDashedIcon: LucideIcon = IconPointerShare as LucideIcon;
|
||||||
|
export const NetworkIcon: LucideIcon = IconNetwork as LucideIcon;
|
||||||
|
export const NotebookIcon: LucideIcon = IconNotebook as LucideIcon;
|
||||||
|
export const NotebookPenIcon: LucideIcon = IconNotebook as LucideIcon;
|
||||||
|
export const NotebookText: LucideIcon = IconNotebook as LucideIcon;
|
||||||
|
export const OrbitIcon: LucideIcon = IconCircleDashed as LucideIcon;
|
||||||
|
export const PackageCheckIcon: LucideIcon = IconPackage as LucideIcon;
|
||||||
|
export const PackageIcon: LucideIcon = IconPackage as LucideIcon;
|
||||||
|
export const PackageXIcon: LucideIcon = IconPackageOff as LucideIcon;
|
||||||
|
export const PaintRollerIcon: LucideIcon = IconBrush as LucideIcon;
|
||||||
|
export const PanelLeftIcon: LucideIcon = IconLayoutSidebar as LucideIcon;
|
||||||
|
export const PanelRightIcon: LucideIcon = IconLayoutSidebarRight as LucideIcon;
|
||||||
|
export const PaperclipIcon: LucideIcon = IconPaperclip as LucideIcon;
|
||||||
|
export const PartyPopperIcon: LucideIcon = IconConfetti as LucideIcon;
|
||||||
|
export const PencilIcon: LucideIcon = IconPencil as LucideIcon;
|
||||||
|
export const PieChartIcon: LucideIcon = IconChartPie as LucideIcon;
|
||||||
|
export const PinIcon: LucideIcon = IconPin as LucideIcon;
|
||||||
|
export const PinOffIcon: LucideIcon = IconPinnedOff as LucideIcon;
|
||||||
|
export const PlayCircleIcon: LucideIcon = IconPlayerPlay as LucideIcon;
|
||||||
|
export const PlayIcon: LucideIcon = IconPlayerPlay as LucideIcon;
|
||||||
|
export const PlaySquareIcon: LucideIcon = IconPlayerPlay as LucideIcon;
|
||||||
|
export const PlugIcon: LucideIcon = IconPlug as LucideIcon;
|
||||||
|
export const PlusCircleIcon: LucideIcon = IconCirclePlus as LucideIcon;
|
||||||
|
export const PlusIcon: LucideIcon = IconPlus as LucideIcon;
|
||||||
|
export const PlusSquareIcon: LucideIcon = IconSquarePlus as LucideIcon;
|
||||||
|
export const PowerOffIcon: LucideIcon = IconPower as LucideIcon;
|
||||||
|
export const PowerSquareIcon: LucideIcon = IconPower as LucideIcon;
|
||||||
|
export const PresentationIcon: LucideIcon = IconPresentation as LucideIcon;
|
||||||
|
export const RefreshCcw: LucideIcon = IconRefresh as LucideIcon;
|
||||||
|
export const RefreshCcwIcon: LucideIcon = IconRefresh as LucideIcon;
|
||||||
|
export const RefreshCwIcon: LucideIcon = IconRefresh as LucideIcon;
|
||||||
|
export const RegexIcon: LucideIcon = IconRegex as LucideIcon;
|
||||||
|
export const RotateCcwIcon: LucideIcon = IconRotate as LucideIcon;
|
||||||
|
export const RotateCwIcon: LucideIcon = IconRotateClockwise as LucideIcon;
|
||||||
|
export const RulerDimensionLine: LucideIcon = IconRuler as LucideIcon;
|
||||||
|
export const SaveIcon: LucideIcon = IconDeviceFloppy as LucideIcon;
|
||||||
|
export const ScissorsIcon: LucideIcon = IconScissors as LucideIcon;
|
||||||
|
export const ScrollIcon: LucideIcon = IconArticle as LucideIcon;
|
||||||
|
export const ScrollTextIcon: LucideIcon = IconArticle as LucideIcon;
|
||||||
|
export const Search: LucideIcon = IconSearch as LucideIcon;
|
||||||
|
export const SearchCheck: LucideIcon = IconZoomCheck as LucideIcon;
|
||||||
|
export const SearchIcon: LucideIcon = IconSearch as LucideIcon;
|
||||||
|
export const SendHorizontalIcon: LucideIcon = IconSend as LucideIcon;
|
||||||
|
export const SettingsIcon: LucideIcon = IconSettings as LucideIcon;
|
||||||
|
export const Share2Icon: LucideIcon = IconShare as LucideIcon;
|
||||||
|
export const ShieldCheckIcon: LucideIcon = IconShieldCheck as LucideIcon;
|
||||||
|
export const ShuffleIcon: LucideIcon = IconArrowsShuffle as LucideIcon;
|
||||||
|
export const SigmaIcon: LucideIcon = IconSum as LucideIcon;
|
||||||
|
export const SkipForwardIcon: LucideIcon = IconPlayerSkipForward as LucideIcon;
|
||||||
|
export const SparklesIcon: LucideIcon = IconSparkles as LucideIcon;
|
||||||
|
export const SquareArrowOutUpRightIcon: LucideIcon = IconExternalLink as LucideIcon;
|
||||||
|
export const SquareCodeIcon: LucideIcon = IconCode as LucideIcon;
|
||||||
|
export const SquareDashedBottomCodeIcon: LucideIcon = IconSquareDashed as LucideIcon;
|
||||||
|
export const SquareEqualIcon: LucideIcon = IconEqual as LucideIcon;
|
||||||
|
export const SquareFunction: LucideIcon = IconMathFunction as LucideIcon;
|
||||||
|
export const SquareFunctionIcon: LucideIcon = IconMathFunction as LucideIcon;
|
||||||
|
export const SquareIcon: LucideIcon = IconSquare as LucideIcon;
|
||||||
|
export const SquareMIcon: LucideIcon = IconSquareLetterM as LucideIcon;
|
||||||
|
export const SquareMousePointerIcon: LucideIcon = IconPointer as LucideIcon;
|
||||||
|
export const SquareStack: LucideIcon = IconStack as LucideIcon;
|
||||||
|
export const StopCircleIcon: LucideIcon = IconPlayerStop as LucideIcon;
|
||||||
|
export const Table2Icon: LucideIcon = IconTable as LucideIcon;
|
||||||
|
export const TableIcon: LucideIcon = IconTable as LucideIcon;
|
||||||
|
export const TerminalIcon: LucideIcon = IconTerminal as LucideIcon;
|
||||||
|
export const TerminalSquareIcon: LucideIcon = IconTerminal2 as LucideIcon;
|
||||||
|
export const TextCursorInputIcon: LucideIcon = IconCursorText as LucideIcon;
|
||||||
|
export const TextIcon: LucideIcon = IconTextCaption as LucideIcon;
|
||||||
|
export const TextSearchIcon: LucideIcon = IconZoomCode as LucideIcon;
|
||||||
|
export const TextSelectionIcon: LucideIcon = IconTextCaption as LucideIcon;
|
||||||
|
export const ToggleLeftIcon: LucideIcon = IconToggleLeft as LucideIcon;
|
||||||
|
export const Trash2Icon: LucideIcon = IconTrash as LucideIcon;
|
||||||
|
export const TrashIcon: LucideIcon = IconTrash as LucideIcon;
|
||||||
|
export const TriangleAlert: LucideIcon = IconAlertTriangle as LucideIcon;
|
||||||
|
export const TriangleIcon: LucideIcon = IconTriangle as LucideIcon;
|
||||||
|
export const TypeIcon: LucideIcon = IconTypography as LucideIcon;
|
||||||
|
export const Undo2Icon: LucideIcon = IconArrowBackUp as LucideIcon;
|
||||||
|
export const UnlinkIcon: LucideIcon = IconUnlink as LucideIcon;
|
||||||
|
export const Upload: LucideIcon = IconUpload as LucideIcon;
|
||||||
|
export const UploadIcon: LucideIcon = IconUpload as LucideIcon;
|
||||||
|
export const UsersIcon: LucideIcon = IconUsers as LucideIcon;
|
||||||
|
export const VariableIcon: LucideIcon = IconVariable as LucideIcon;
|
||||||
|
export const ViewIcon: LucideIcon = IconEye as LucideIcon;
|
||||||
|
export const WholeWordIcon: LucideIcon = IconAlphabetLatin as LucideIcon;
|
||||||
|
export const WifiIcon: LucideIcon = IconWifi as LucideIcon;
|
||||||
|
export const WifiOffIcon: LucideIcon = IconWifiOff as LucideIcon;
|
||||||
|
export const WorkflowIcon: LucideIcon = IconSitemap as LucideIcon;
|
||||||
|
export const WrapTextIcon: LucideIcon = IconTextWrap as LucideIcon;
|
||||||
|
export const WrenchIcon: LucideIcon = IconTool as LucideIcon;
|
||||||
|
export const X: LucideIcon = IconX as LucideIcon;
|
||||||
|
export const XCircle: LucideIcon = IconCircleX as LucideIcon;
|
||||||
|
export const XCircleIcon: LucideIcon = IconCircleX as LucideIcon;
|
||||||
|
export const XIcon: LucideIcon = IconX as LucideIcon;
|
||||||
|
export const YoutubeIcon: LucideIcon = IconBrandYoutube as LucideIcon;
|
||||||
|
export const ZapIcon: LucideIcon = IconBolt as LucideIcon;
|
||||||
|
export const ZapOffIcon: LucideIcon = IconBoltOff as LucideIcon;
|
||||||
107
infra/marimo/theme/scripts/apply-overlay.sh
Executable file
@@ -0,0 +1,107 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# Apply fhirworx overlay onto a vanilla marimo source tree.
|
||||||
|
# Idempotent: safe to run multiple times during iterative development.
|
||||||
|
#
|
||||||
|
# Invoked by the Dockerfile after `git clone`, before `pnpm install`.
|
||||||
|
# Run from the marimo source root (same dir as frontend/ and pyproject.toml).
|
||||||
|
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
SRC="${1:?usage: apply-overlay.sh <marimo-source-root> <overlay-root>}"
|
||||||
|
OVR="${2:?}"
|
||||||
|
|
||||||
|
echo "== fhirworx overlay → $SRC"
|
||||||
|
|
||||||
|
# 1. Drop new files in place (lucide-shim, fhirworx.css, asset overrides).
|
||||||
|
cp -R "$OVR/frontend/." "$SRC/frontend/"
|
||||||
|
|
||||||
|
# 2. Add @tabler/icons-react to frontend dependencies via jq (idempotent).
|
||||||
|
# Pinned to a known-good major that matches our shim's export names.
|
||||||
|
jq '.dependencies["@tabler/icons-react"] = "^3.26.0"' \
|
||||||
|
"$SRC/frontend/package.json" > "$SRC/frontend/package.json.new"
|
||||||
|
mv "$SRC/frontend/package.json.new" "$SRC/frontend/package.json"
|
||||||
|
|
||||||
|
# 3. Rewrite pnpm-lock if it exists so pnpm install doesn't error on drift.
|
||||||
|
rm -f "$SRC/frontend/pnpm-lock.yaml" "$SRC/pnpm-lock.yaml"
|
||||||
|
|
||||||
|
# 4. Inject our CSS import into globals.css.
|
||||||
|
# `@import` must precede all other statements (postcss/CSS spec), so
|
||||||
|
# append-at-EOF would get rejected with a warning + dropped. Instead,
|
||||||
|
# insert our @import right after the last existing top-level @import.
|
||||||
|
if ! grep -q 'fhirworx.css' "$SRC/frontend/src/css/globals.css"; then
|
||||||
|
python3 - "$SRC/frontend/src/css/globals.css" <<'PY'
|
||||||
|
import sys, re
|
||||||
|
p = sys.argv[1]
|
||||||
|
s = open(p).read()
|
||||||
|
# Find the index right after the last top-level @import line.
|
||||||
|
matches = list(re.finditer(r'^@import[^;]*;\s*\n', s, flags=re.M))
|
||||||
|
if not matches:
|
||||||
|
sys.exit(f"overlay: no existing @import in {p}")
|
||||||
|
pos = matches[-1].end()
|
||||||
|
insert = '@import "./fhirworx.css";\n'
|
||||||
|
open(p, "w").write(s[:pos] + insert + s[pos:])
|
||||||
|
PY
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 4b. Strip the "Resources" section from the home page.
|
||||||
|
# Touchless landing page: no upstream documentation links, just the
|
||||||
|
# user's notebooks + workspace.
|
||||||
|
HOME="$SRC/frontend/src/components/pages/home-page.tsx"
|
||||||
|
if [ -f "$HOME" ] && grep -q '<ResourceLinks />' "$HOME"; then
|
||||||
|
python3 - "$HOME" <<'PY'
|
||||||
|
import sys, re
|
||||||
|
p = sys.argv[1]
|
||||||
|
s = open(p).read()
|
||||||
|
# Remove the JSX element render.
|
||||||
|
s = re.sub(r'^\s*<ResourceLinks />\s*\n', '', s, flags=re.M)
|
||||||
|
# Remove it from the imports so TS doesn't complain about unused symbols.
|
||||||
|
s = re.sub(r'(\bResourceLinks,\s*)', '', s)
|
||||||
|
# Relabel the logo so it doesn't say "marimo".
|
||||||
|
s = s.replace('alt="marimo logo"', 'alt="fhirworx"')
|
||||||
|
open(p, 'w').write(s)
|
||||||
|
PY
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 5. Inject the lucide-react → shim alias into vite.config.mts.
|
||||||
|
# Match the `resolve: {` block and insert an `alias` entry right after.
|
||||||
|
if ! grep -q 'lucide-shim' "$SRC/frontend/vite.config.mts"; then
|
||||||
|
python3 - "$SRC/frontend/vite.config.mts" <<'PY'
|
||||||
|
import sys, re
|
||||||
|
path = sys.argv[1]
|
||||||
|
s = open(path).read()
|
||||||
|
|
||||||
|
# Ensure node:path + node:url are imported for the alias resolution.
|
||||||
|
if "from \"node:path\"" not in s and "from 'node:path'" not in s:
|
||||||
|
s = 'import { dirname, resolve as pathResolve } from "node:path";\n' + \
|
||||||
|
'import { fileURLToPath } from "node:url";\n' + s
|
||||||
|
|
||||||
|
# Inject alias immediately after `resolve: {`. fileURLToPath() keeps the
|
||||||
|
# path Windows-safe and anchored to this config file, regardless of where
|
||||||
|
# the build is invoked from.
|
||||||
|
inject_alias = (
|
||||||
|
' alias: {\n'
|
||||||
|
' "lucide-react": pathResolve(\n'
|
||||||
|
' dirname(fileURLToPath(import.meta.url)),\n'
|
||||||
|
' "src/lucide-shim.tsx",\n'
|
||||||
|
' ),\n'
|
||||||
|
' },\n'
|
||||||
|
)
|
||||||
|
s2, n = re.subn(r'(resolve:\s*\{\n)', r'\1' + inject_alias, s, count=1)
|
||||||
|
if n != 1:
|
||||||
|
sys.exit(f"overlay: could not find 'resolve: {{' in {path}")
|
||||||
|
|
||||||
|
# Raise the build target to esnext. Default is es2020+old-browsers, and
|
||||||
|
# vite-plugin-top-level-await's esbuild pass can't downlevel Tabler's
|
||||||
|
# bundled destructuring patterns to that set. All currently-supported
|
||||||
|
# evergreen browsers handle esnext output fine; no need to constrain.
|
||||||
|
inject_target = (
|
||||||
|
' target: "esnext",\n'
|
||||||
|
)
|
||||||
|
s3, n2 = re.subn(r'(build:\s*\{\n)', r'\1' + inject_target, s2, count=1)
|
||||||
|
if n2 != 1:
|
||||||
|
sys.exit(f"overlay: could not find 'build: {{' in {path}")
|
||||||
|
open(path, "w").write(s3)
|
||||||
|
PY
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "== overlay applied"
|
||||||
331
infra/marimo/theme/scripts/gen-lucide-shim.py
Normal file
@@ -0,0 +1,331 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Regenerate `theme/frontend/src/lucide-shim.tsx`.
|
||||||
|
|
||||||
|
Scans a marimo source tree for every `import {…} from 'lucide-react'`
|
||||||
|
and emits a TSX shim that re-exports Tabler icons under each Lucide
|
||||||
|
name. Runs outside the Docker build (the shim is committed), so you
|
||||||
|
only rerun this when bumping the pinned marimo version or when an
|
||||||
|
upstream change introduces new lucide icons.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 gen-lucide-shim.py [MARIMO_SRC_ROOT]
|
||||||
|
|
||||||
|
Default source root is `infra/marimo/src` relative to the stack repo root.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# Lucide-name → Tabler export name. Populated with the divergent cases;
|
||||||
|
# straight-rename names are handled by `transform()` below.
|
||||||
|
SPECIAL: dict[str, str] = {
|
||||||
|
"Play": "IconPlayerPlay",
|
||||||
|
"PlayIcon": "IconPlayerPlay",
|
||||||
|
"PlayCircleIcon": "IconPlayerPlay",
|
||||||
|
"PlaySquareIcon": "IconPlayerPlay",
|
||||||
|
"CirclePlayIcon": "IconPlayerPlay",
|
||||||
|
"Pause": "IconPlayerPause",
|
||||||
|
"StopCircleIcon": "IconPlayerStop",
|
||||||
|
"SkipForwardIcon": "IconPlayerSkipForward",
|
||||||
|
"FastForwardIcon": "IconPlayerTrackNext",
|
||||||
|
"Loader2": "IconLoader2",
|
||||||
|
"Loader2Icon": "IconLoader2",
|
||||||
|
"LoaderCircle": "IconLoader",
|
||||||
|
"TriangleAlert": "IconAlertTriangle",
|
||||||
|
"ChartColumn": "IconChartBar",
|
||||||
|
"ChartColumnIcon": "IconChartBar",
|
||||||
|
"ChartColumnStacked": "IconChartBarOff",
|
||||||
|
"ChartNoAxesColumn": "IconChartBar",
|
||||||
|
"HatGlasses": "IconMoodHappy",
|
||||||
|
"RulerDimensionLine": "IconRuler",
|
||||||
|
"FunnelPlusIcon": "IconFilterPlus",
|
||||||
|
"FilterX": "IconFilterOff",
|
||||||
|
"AreaChartIcon": "IconChartArea",
|
||||||
|
"BarChartIcon": "IconChartBar",
|
||||||
|
"BarChart2Icon": "IconChartBar",
|
||||||
|
"BarChartBigIcon": "IconChartBar",
|
||||||
|
"LineChartIcon": "IconChartLine",
|
||||||
|
"PieChartIcon": "IconChartPie",
|
||||||
|
"ChartPieIcon": "IconChartPie",
|
||||||
|
"ChartScatterIcon": "IconChartDots",
|
||||||
|
"ChartSplineIcon": "IconChartLine",
|
||||||
|
"Grid3x3Icon": "IconGrid3x3",
|
||||||
|
"Columns2Icon": "IconColumns2",
|
||||||
|
"Edit3Icon": "IconEdit",
|
||||||
|
"Code2Icon": "IconCode",
|
||||||
|
"FileAudio2Icon": "IconFileMusic",
|
||||||
|
"FileAudioIcon": "IconFileMusic",
|
||||||
|
"FileVideoCameraIcon": "IconFileTypography",
|
||||||
|
"SquareMIcon": "IconSquareLetterM",
|
||||||
|
"SquareCodeIcon": "IconCode",
|
||||||
|
"SquareDashedBottomCodeIcon": "IconSquareDashed",
|
||||||
|
"CircleHelpIcon": "IconHelpCircle",
|
||||||
|
"HelpCircleIcon": "IconHelpCircle",
|
||||||
|
"MessageCircleQuestionIcon": "IconMessageCircleQuestion",
|
||||||
|
"AlertOctagonIcon": "IconAlertOctagon",
|
||||||
|
"CheckCircle2Icon": "IconCircleCheck",
|
||||||
|
"CheckCircleIcon": "IconCircleCheck",
|
||||||
|
"CheckSquareIcon": "IconSquareCheck",
|
||||||
|
"XCircle": "IconCircleX",
|
||||||
|
"XCircleIcon": "IconCircleX",
|
||||||
|
"DatabaseZap": "IconDatabase",
|
||||||
|
"DatabaseZapIcon": "IconDatabase",
|
||||||
|
"ArrowDownWideNarrowIcon": "IconArrowsSort",
|
||||||
|
"ArrowUpWideNarrowIcon": "IconArrowsSort",
|
||||||
|
"ArrowUpNarrowWideIcon": "IconArrowsSort",
|
||||||
|
"ChevronsUpDown": "IconChevronUpDown",
|
||||||
|
"ChevronsUpDownIcon": "IconChevronUpDown",
|
||||||
|
"ChevronsDownUpIcon": "IconChevronDownUp",
|
||||||
|
"CircleEllipsis": "IconDotsCircleHorizontal",
|
||||||
|
"MousePointerSquareDashedIcon": "IconPointerShare",
|
||||||
|
"SquareMousePointerIcon": "IconPointer",
|
||||||
|
"SquareArrowOutUpRightIcon": "IconExternalLink",
|
||||||
|
"SquareFunction": "IconMathFunction",
|
||||||
|
"SquareFunctionIcon": "IconMathFunction",
|
||||||
|
"FunctionSquareIcon": "IconMathFunction",
|
||||||
|
"SquareEqualIcon": "IconEqual",
|
||||||
|
"Share2Icon": "IconShare",
|
||||||
|
"Undo2Icon": "IconArrowBackUp",
|
||||||
|
"Trash2Icon": "IconTrash",
|
||||||
|
"BetweenHorizontalStartIcon": "IconLayoutRows",
|
||||||
|
"BugPlayIcon": "IconBug",
|
||||||
|
"FolderArchiveIcon": "IconFolder",
|
||||||
|
"FolderCog2": "IconFolderCog",
|
||||||
|
"TerminalSquareIcon": "IconTerminal2",
|
||||||
|
"HardDriveDownloadIcon":"IconDeviceSdCard",
|
||||||
|
"DownloadCloudIcon": "IconCloudDownload",
|
||||||
|
"HardDrive": "IconDeviceSdCard",
|
||||||
|
"HardDriveIcon": "IconDeviceSdCard",
|
||||||
|
"MemoryStickIcon": "IconDeviceSdCard",
|
||||||
|
"MicrochipIcon": "IconCpu",
|
||||||
|
"BrickWallIcon": "IconWall",
|
||||||
|
"ViewIcon": "IconEye",
|
||||||
|
"BotMessageSquareIcon": "IconRobot",
|
||||||
|
"BotIcon": "IconRobot",
|
||||||
|
"TextCursorInputIcon": "IconCursorText",
|
||||||
|
"TextSelectionIcon": "IconTextCaption",
|
||||||
|
"TextSearchIcon": "IconZoomCode",
|
||||||
|
"TextIcon": "IconTextCaption",
|
||||||
|
"TypeIcon": "IconTypography",
|
||||||
|
"CaseSensitiveIcon": "IconLetterCase",
|
||||||
|
"WholeWordIcon": "IconAlphabetLatin",
|
||||||
|
"WrapTextIcon": "IconTextWrap",
|
||||||
|
"SendHorizontalIcon": "IconSend",
|
||||||
|
"NotebookText": "IconNotebook",
|
||||||
|
"NotebookPenIcon": "IconNotebook",
|
||||||
|
"BookMarkedIcon": "IconBookmark",
|
||||||
|
"BookTextIcon": "IconBook",
|
||||||
|
"KeyRoundIcon": "IconKey",
|
||||||
|
"PartyPopperIcon": "IconConfetti",
|
||||||
|
"PaintRollerIcon": "IconBrush",
|
||||||
|
"PackageCheckIcon": "IconPackage",
|
||||||
|
"PackageXIcon": "IconPackageOff",
|
||||||
|
"YoutubeIcon": "IconBrandYoutube",
|
||||||
|
"GithubIcon": "IconBrandGithub",
|
||||||
|
"Info": "IconInfoCircle",
|
||||||
|
"InfoIcon": "IconInfoCircle",
|
||||||
|
"CopyMinusIcon": "IconCopy",
|
||||||
|
"CopySlashIcon": "IconCopyOff",
|
||||||
|
"SearchCheck": "IconZoomCheck",
|
||||||
|
"ChevronDownCircleIcon": "IconCircleChevronDown",
|
||||||
|
"ChevronRightCircleIcon": "IconCircleChevronRight",
|
||||||
|
"MoreHorizontal": "IconDots",
|
||||||
|
"MoreHorizontalIcon": "IconDots",
|
||||||
|
"MoreVerticalIcon": "IconDotsVertical",
|
||||||
|
"RefreshCcw": "IconRefresh",
|
||||||
|
"RefreshCcwIcon": "IconRefresh",
|
||||||
|
"RefreshCwIcon": "IconRefresh",
|
||||||
|
"RotateCcwIcon": "IconRotate",
|
||||||
|
"RotateCwIcon": "IconRotateClockwise",
|
||||||
|
"AtSignIcon": "IconAt",
|
||||||
|
"SigmaIcon": "IconSum",
|
||||||
|
"BaselineIcon": "IconBaselineDensitySmall",
|
||||||
|
"AlignCenterVerticalIcon": "IconAlignCenter",
|
||||||
|
"AlignStartVerticalIcon": "IconAlignLeft",
|
||||||
|
"AlignEndVerticalIcon": "IconAlignRight",
|
||||||
|
"AlignHorizontalSpaceAroundIcon": "IconAlignJustified",
|
||||||
|
"AlignJustifyIcon": "IconAlignJustified",
|
||||||
|
"ZapIcon": "IconBolt",
|
||||||
|
"ZapOffIcon": "IconBoltOff",
|
||||||
|
"ListFilterIcon": "IconFilter",
|
||||||
|
"ListFilterPlusIcon": "IconFilterPlus",
|
||||||
|
"CircleHelp": "IconHelpCircle",
|
||||||
|
"CalendarClockIcon": "IconCalendarTime",
|
||||||
|
"ClipboardPasteIcon": "IconClipboardText",
|
||||||
|
"GraduationCapIcon": "IconSchool",
|
||||||
|
"ExpandIcon": "IconArrowsMaximize",
|
||||||
|
"LayersIcon": "IconStack",
|
||||||
|
"LayoutTemplateIcon": "IconLayoutBoard",
|
||||||
|
"ListOrderedIcon": "IconListNumbers",
|
||||||
|
"GroupIcon": "IconUsersGroup",
|
||||||
|
"GlobeIcon": "IconWorld",
|
||||||
|
"WorkflowIcon": "IconSitemap",
|
||||||
|
"Table2Icon": "IconTable",
|
||||||
|
"MessagesSquareIcon": "IconMessages",
|
||||||
|
"MenuIcon": "IconMenu2",
|
||||||
|
"MonitorIcon": "IconDeviceDesktop",
|
||||||
|
"PlusCircleIcon": "IconCirclePlus",
|
||||||
|
"PlusSquareIcon": "IconSquarePlus",
|
||||||
|
"FilePenIcon": "IconFilePencil",
|
||||||
|
"FilePlus2Icon": "IconFilePlus",
|
||||||
|
"FileImageIcon": "IconPhoto",
|
||||||
|
"FileJsonIcon": "IconJson",
|
||||||
|
"FileVideoIcon": "IconVideo",
|
||||||
|
"ImageIcon": "IconPhoto",
|
||||||
|
"PanelLeftIcon": "IconLayoutSidebar",
|
||||||
|
"PanelRightIcon": "IconLayoutSidebarRight",
|
||||||
|
"PowerOffIcon": "IconPower",
|
||||||
|
"PowerSquareIcon": "IconPower",
|
||||||
|
"SaveIcon": "IconDeviceFloppy",
|
||||||
|
"ShuffleIcon": "IconArrowsShuffle",
|
||||||
|
"GridIcon": "IconGrid3x3",
|
||||||
|
"CombineIcon": "IconStackPush",
|
||||||
|
"ArrowDownToLineIcon": "IconArrowBarToDown",
|
||||||
|
"ArrowRightFromLineIcon": "IconArrowBarRight",
|
||||||
|
"ArrowRightToLineIcon": "IconArrowBarToRight",
|
||||||
|
"ArrowRightSquareIcon": "IconSquareArrowRight",
|
||||||
|
"ArrowUpDownIcon": "IconArrowsUpDown",
|
||||||
|
"ArrowUpToLineIcon": "IconArrowBarToUp",
|
||||||
|
"SquareStack": "IconStack",
|
||||||
|
"WrenchIcon": "IconTool",
|
||||||
|
"SquareIcon": "IconSquare",
|
||||||
|
"CornerLeftUp": "IconCornerLeftUp",
|
||||||
|
"GripHorizontal": "IconGripHorizontal",
|
||||||
|
"GripHorizontalIcon": "IconGripHorizontal",
|
||||||
|
"GripVerticalIcon": "IconGripVertical",
|
||||||
|
"CrosshairIcon": "IconCrosshair",
|
||||||
|
"CommandIcon": "IconCommand",
|
||||||
|
"PinIcon": "IconPin",
|
||||||
|
"PinOffIcon": "IconPinnedOff",
|
||||||
|
"DiamondPlusIcon": "IconDiamondPlus",
|
||||||
|
"BinaryIcon": "IconBinary",
|
||||||
|
"BoxIcon": "IconBox",
|
||||||
|
"BracesIcon": "IconBraces",
|
||||||
|
"BracketsIcon": "IconBrackets",
|
||||||
|
"BrainIcon": "IconBrain",
|
||||||
|
"ActivityIcon": "IconActivity",
|
||||||
|
"Cog": "IconSettings",
|
||||||
|
"CogIcon": "IconSettings",
|
||||||
|
# Names that don't exist as-is in @tabler/icons-react v3.41.
|
||||||
|
"BookOpenIcon": "IconBook",
|
||||||
|
"BookPlusIcon": "IconBook",
|
||||||
|
"ChevronsUpDown": "IconSelector",
|
||||||
|
"ChevronsUpDownIcon": "IconSelector",
|
||||||
|
"ChevronsDownUpIcon": "IconSelector",
|
||||||
|
"CurlyBracesIcon": "IconBraces",
|
||||||
|
"DiamondPlusIcon": "IconDiamond",
|
||||||
|
"EllipsisIcon": "IconDots",
|
||||||
|
"FlaskConicalIcon": "IconFlask",
|
||||||
|
"FolderTreeIcon": "IconFolders",
|
||||||
|
"OrbitIcon": "IconCircleDashed",
|
||||||
|
"ScrollIcon": "IconArticle",
|
||||||
|
"ScrollTextIcon": "IconArticle",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
_FALLBACK = "IconQuestionMark"
|
||||||
|
|
||||||
|
|
||||||
|
def load_tabler_exports() -> set[str] | None:
|
||||||
|
"""Return the set of Tabler icon names if the authoritative list is on
|
||||||
|
disk (generated by probing @tabler/icons-react in a container). If it's
|
||||||
|
missing, return None — generator emits without validation and build-time
|
||||||
|
errors surface the unknowns."""
|
||||||
|
cand = Path(__file__).resolve().parent / "tabler-icons.txt"
|
||||||
|
if not cand.is_file():
|
||||||
|
cand = Path("/tmp/tabler-icons.txt")
|
||||||
|
if not cand.is_file():
|
||||||
|
return None
|
||||||
|
return {line.strip() for line in cand.read_text().splitlines() if line.strip()}
|
||||||
|
|
||||||
|
|
||||||
|
_TABLER: set[str] | None = load_tabler_exports()
|
||||||
|
|
||||||
|
|
||||||
|
def transform(name: str) -> str:
|
||||||
|
"""Map a Lucide export name to its Tabler equivalent, falling back to
|
||||||
|
IconQuestionMark when the intended target doesn't actually exist in the
|
||||||
|
installed @tabler/icons-react version."""
|
||||||
|
if name in SPECIAL:
|
||||||
|
target = SPECIAL[name]
|
||||||
|
else:
|
||||||
|
base = name[:-4] if name.endswith("Icon") else name
|
||||||
|
target = "Icon" + base
|
||||||
|
if _TABLER is not None and target not in _TABLER:
|
||||||
|
return _FALLBACK
|
||||||
|
return target
|
||||||
|
|
||||||
|
|
||||||
|
def extract(src_root: Path) -> set[str]:
|
||||||
|
"""Collect every name marimo imports or re-exports from lucide-react."""
|
||||||
|
names: set[str] = set()
|
||||||
|
# Covers both `import { X, Y as Z } from "lucide-react"` and
|
||||||
|
# `export { X as A } from "lucide-react"` (0.23.1 added re-exports).
|
||||||
|
pattern = re.compile(
|
||||||
|
r'(?:import|export)\s*(?:type\s*)?\{([^}]+)\}\s*from\s*["\']lucide-react["\']'
|
||||||
|
)
|
||||||
|
for root, _, files in os.walk(src_root):
|
||||||
|
for fn in files:
|
||||||
|
if not fn.endswith((".ts", ".tsx")):
|
||||||
|
continue
|
||||||
|
text = (Path(root) / fn).read_text()
|
||||||
|
for match in pattern.finditer(text):
|
||||||
|
for token in match.group(1).split(","):
|
||||||
|
token = token.strip().removeprefix("type ").strip()
|
||||||
|
token = token.split(" as ")[0].strip()
|
||||||
|
if token and re.match(r"^[A-Z]", token):
|
||||||
|
names.add(token)
|
||||||
|
return names
|
||||||
|
|
||||||
|
|
||||||
|
def render(names: set[str]) -> str:
|
||||||
|
TYPE_ONLY = {"LucideIcon", "LucideProps"}
|
||||||
|
sorted_names = sorted(n for n in names if n not in TYPE_ONLY)
|
||||||
|
# De-dup tabler names (many lucide aliases resolve to the same icon).
|
||||||
|
tabler_set = sorted({transform(n) for n in sorted_names})
|
||||||
|
|
||||||
|
header = [
|
||||||
|
"// Auto-generated: Tabler Icons as a drop-in for lucide-react.",
|
||||||
|
"// Build-time replacement; do not edit by hand — see gen-lucide-shim.py",
|
||||||
|
"import type { ComponentType, SVGProps } from 'react';",
|
||||||
|
"import {",
|
||||||
|
]
|
||||||
|
header.extend(f" {name}," for name in tabler_set)
|
||||||
|
header.extend([
|
||||||
|
" type IconProps,",
|
||||||
|
"} from '@tabler/icons-react';",
|
||||||
|
"",
|
||||||
|
"// Prop shapes mirror lucide's public types so Omit<LucideProps, 'ref'>",
|
||||||
|
"// and similar patterns in upstream marimo code keep typechecking.",
|
||||||
|
"export type LucideProps = SVGProps<SVGSVGElement> & {",
|
||||||
|
" size?: number | string;",
|
||||||
|
" absoluteStrokeWidth?: boolean;",
|
||||||
|
"};",
|
||||||
|
"export type LucideIcon = ComponentType<LucideProps & IconProps>;",
|
||||||
|
"",
|
||||||
|
])
|
||||||
|
body = [
|
||||||
|
f"export const {n}: LucideIcon = {transform(n)} as LucideIcon;"
|
||||||
|
for n in sorted_names
|
||||||
|
]
|
||||||
|
return "\n".join(header + body) + "\n"
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
default = Path(__file__).resolve().parents[2] / "src"
|
||||||
|
src = Path(sys.argv[1]) if len(sys.argv) > 1 else default
|
||||||
|
if not src.is_dir():
|
||||||
|
raise SystemExit(f"marimo source not found: {src}")
|
||||||
|
names = extract(src / "frontend" / "src")
|
||||||
|
out = Path(__file__).resolve().parents[1] / "frontend" / "src" / "lucide-shim.tsx"
|
||||||
|
out.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
out.write_text(render(names))
|
||||||
|
print(f"wrote {out} ({len(names)} names)")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
6149
infra/marimo/theme/scripts/tabler-icons.txt
Normal file
@@ -1,24 +1,24 @@
|
|||||||
{{- $domain := env "DOMAIN" | default "fhirworx.io" -}}
|
{{- $domain := env "DOMAIN" | default "fhirworx.io" -}}
|
||||||
{{- $reef := dict
|
{{- $reef := dict
|
||||||
"dashboard" (dict "port" "80" "theme" true "extra_hosts" (list $domain) "mw" "secure-headers")
|
"dashboard" (dict "port" "80" "theme" true "extra_hosts" (list $domain) "mw" "secure-headers")
|
||||||
"docs" (dict "port" "80" "theme" true "mw" "gitea-sso,secure-headers")
|
"docs" (dict "port" "80" "theme" true "mw" "git-sso,secure-headers")
|
||||||
"gitea" (dict "port" "3000" "subdomain" "git" "theme" true "mw" "secure-headers")
|
"git" (dict "port" "3000" "theme" false "mw" "secure-headers")
|
||||||
"woodpecker-server" (dict "port" "8000" "theme" true "subdomain" "ci" "mw" "gitea-sso,secure-headers")
|
"woodpecker-server" (dict "port" "8000" "theme" true "subdomain" "ci" "mw" "git-sso,secure-headers")
|
||||||
"notebooks" (dict "port" "2718" "theme" true "mw" "gitea-sso,secure-headers")
|
"notebooks" (dict "port" "2718" "theme" true "mw" "git-sso,secure-headers")
|
||||||
"zotero" (dict "port" "8080" "theme" true "mw" "gitea-sso,secure-headers")
|
"zotero" (dict "port" "8080" "theme" true "mw" "git-sso,secure-headers")
|
||||||
"webdav" (dict "port" "8080" "theme" false "mw" "secure-headers")
|
"webdav" (dict "port" "8080" "theme" false "mw" "secure-headers")
|
||||||
"api" (dict "port" "8000" "theme" false "mw" "secure-headers")
|
"api" (dict "port" "8000" "theme" false "mw" "secure-headers")
|
||||||
"nessie" (dict "port" "19120" "theme" false "mw" "gitea-sso,infra-headers")
|
"nessie" (dict "port" "19120" "theme" false "mw" "git-sso,infra-headers")
|
||||||
"trino" (dict "port" "8080" "theme" true "mw" "gitea-sso,infra-headers")
|
"trino" (dict "port" "8080" "theme" true "mw" "git-sso,infra-headers")
|
||||||
"polaris" (dict "port" "8181" "theme" false "mw" "gitea-sso,infra-headers")
|
"polaris" (dict "port" "8181" "theme" false "mw" "git-sso,infra-headers")
|
||||||
"grafana" (dict "port" "3000" "theme" true "mw" "gitea-sso,secure-headers")
|
"grafana" (dict "port" "3000" "theme" true "mw" "git-sso,secure-headers")
|
||||||
"prometheus" (dict "port" "9090" "theme" true "mw" "gitea-sso,infra-headers")
|
"prometheus" (dict "port" "9090" "theme" true "mw" "git-sso,infra-headers")
|
||||||
"jaeger" (dict "port" "16686" "theme" true "mw" "gitea-sso,infra-headers")
|
"jaeger" (dict "port" "16686" "theme" true "mw" "git-sso,infra-headers")
|
||||||
"loki" (dict "port" "3100" "theme" false "mw" "gitea-sso,infra-headers")
|
"loki" (dict "port" "3100" "theme" false "mw" "git-sso,infra-headers")
|
||||||
-}}
|
-}}
|
||||||
{{- $multi := dict
|
{{- $multi := dict
|
||||||
"rustfs-api" (dict "container" "rustfs" "port" "9000" "subdomain" "s3" "theme" false "mw" "gitea-sso,infra-headers")
|
"rustfs-api" (dict "container" "rustfs" "port" "9000" "subdomain" "s3" "theme" false "mw" "git-sso,infra-headers")
|
||||||
"rustfs-console" (dict "container" "rustfs" "port" "9001" "subdomain" "s3console" "theme" true "mw" "gitea-sso,infra-headers")
|
"rustfs-console" (dict "container" "rustfs" "port" "9001" "subdomain" "s3console" "theme" true "mw" "git-sso,infra-headers")
|
||||||
-}}
|
-}}
|
||||||
http:
|
http:
|
||||||
middlewares:
|
middlewares:
|
||||||
@@ -29,7 +29,7 @@ http:
|
|||||||
contentTypeNosniff: true
|
contentTypeNosniff: true
|
||||||
# SSO: auth-handler (nginx) wraps oauth2-proxy to convert 401 → 302.
|
# SSO: auth-handler (nginx) wraps oauth2-proxy to convert 401 → 302.
|
||||||
# Same logic as corwins.media: auth_request + error_page 401 = @signin.
|
# Same logic as corwins.media: auth_request + error_page 401 = @signin.
|
||||||
gitea-sso:
|
git-sso:
|
||||||
forwardAuth:
|
forwardAuth:
|
||||||
address: "http://auth-handler:4181"
|
address: "http://auth-handler:4181"
|
||||||
trustForwardHeader: true
|
trustForwardHeader: true
|
||||||
@@ -129,7 +129,7 @@ http:
|
|||||||
- web
|
- web
|
||||||
middlewares:
|
middlewares:
|
||||||
- inject-fhirworx
|
- inject-fhirworx
|
||||||
- gitea-sso
|
- git-sso
|
||||||
- infra-headers
|
- infra-headers
|
||||||
traefik-dashboard-tls:
|
traefik-dashboard-tls:
|
||||||
rule: "Host(`traefik.{{ $domain }}`)"
|
rule: "Host(`traefik.{{ $domain }}`)"
|
||||||
@@ -139,7 +139,7 @@ http:
|
|||||||
tls: {}
|
tls: {}
|
||||||
middlewares:
|
middlewares:
|
||||||
- inject-fhirworx
|
- inject-fhirworx
|
||||||
- gitea-sso
|
- git-sso
|
||||||
- infra-headers
|
- infra-headers
|
||||||
services:
|
services:
|
||||||
{{- range $name, $svc := $reef }}
|
{{- range $name, $svc := $reef }}
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ def _():
|
|||||||
def _():
|
def _():
|
||||||
import altair as alt
|
import altair as alt
|
||||||
import polars as pl
|
import polars as pl
|
||||||
|
|
||||||
from conf import connect
|
from conf import connect
|
||||||
|
|
||||||
connect.theme()
|
connect.theme()
|
||||||
|
|||||||
@@ -254,7 +254,11 @@ def _(PALETTE, alt, mo, q):
|
|||||||
.encode(
|
.encode(
|
||||||
x=alt.X("patients:Q", title="Patients"),
|
x=alt.X("patients:Q", title="Patients"),
|
||||||
y=alt.Y("condition:N", sort="-x", title=None),
|
y=alt.Y("condition:N", sort="-x", title=None),
|
||||||
color=alt.Color("patients:Q", scale=alt.Scale(range=[PALETTE[1] + "33", PALETTE[1]]), legend=None),
|
color=alt.Color(
|
||||||
|
"patients:Q",
|
||||||
|
scale=alt.Scale(range=[PALETTE[1] + "33", PALETTE[1]]),
|
||||||
|
legend=None,
|
||||||
|
),
|
||||||
tooltip=["condition:N", "patients:Q"],
|
tooltip=["condition:N", "patients:Q"],
|
||||||
)
|
)
|
||||||
.properties(
|
.properties(
|
||||||
|
|||||||
@@ -44,10 +44,12 @@ def _(client, mo):
|
|||||||
_rows = []
|
_rows = []
|
||||||
for _svc in health["services"]:
|
for _svc in health["services"]:
|
||||||
_icon = "\u2705" if _svc["status"] == "ok" else "\u26a0\ufe0f"
|
_icon = "\u2705" if _svc["status"] == "ok" else "\u26a0\ufe0f"
|
||||||
_rows.append(f"| {_icon} | {_svc['name']} | {_svc['status']} | {_svc.get('detail', '')} |")
|
_rows.append(
|
||||||
|
f"| {_icon} | {_svc['name']} | {_svc['status']} | {_svc.get('detail', '')} |"
|
||||||
|
)
|
||||||
|
|
||||||
mo.md(f"""
|
mo.md(f"""
|
||||||
**Status:** {health['status']} | **Version:** {health['version']}
|
**Status:** {health["status"]} | **Version:** {health["version"]}
|
||||||
|
|
||||||
| | Service | Status | Detail |
|
| | Service | Status | Detail |
|
||||||
|---|---------|--------|--------|
|
|---|---------|--------|--------|
|
||||||
@@ -93,9 +95,9 @@ def _(client, mo, pipelines):
|
|||||||
_outputs_list = "\n".join(f"- `{o}`" for o in detail.get("outputs", []))
|
_outputs_list = "\n".join(f"- `{o}`" for o in detail.get("outputs", []))
|
||||||
|
|
||||||
mo.md(f"""
|
mo.md(f"""
|
||||||
### Pipeline Detail: `{detail['name']}`
|
### Pipeline Detail: `{detail["name"]}`
|
||||||
|
|
||||||
**Steps:** {detail['steps']} | **Inputs:** {len(detail.get('inputs', []))} | **Outputs:** {len(detail.get('outputs', []))}
|
**Steps:** {detail["steps"]} | **Inputs:** {len(detail.get("inputs", []))} | **Outputs:** {len(detail.get("outputs", []))}
|
||||||
|
|
||||||
**Inputs:**
|
**Inputs:**
|
||||||
{_inputs_list}
|
{_inputs_list}
|
||||||
@@ -110,7 +112,9 @@ def _(client, mo, pipelines):
|
|||||||
def _(client, mo):
|
def _(client, mo):
|
||||||
_r = client.get("/pipelines/nonexistent_xyz")
|
_r = client.get("/pipelines/nonexistent_xyz")
|
||||||
pipeline_404 = _r.status_code == 404
|
pipeline_404 = _r.status_code == 404
|
||||||
mo.md(f"**GET /pipelines/nonexistent_xyz** \u2192 `{_r.status_code}` {'PASS' if pipeline_404 else 'FAIL'}")
|
mo.md(
|
||||||
|
f"**GET /pipelines/nonexistent_xyz** \u2192 `{_r.status_code}` {'PASS' if pipeline_404 else 'FAIL'}"
|
||||||
|
)
|
||||||
return (pipeline_404,)
|
return (pipeline_404,)
|
||||||
|
|
||||||
|
|
||||||
@@ -150,9 +154,9 @@ def _(client, lineage, mo):
|
|||||||
_r = client.get(f"/lineage/{_test_table}")
|
_r = client.get(f"/lineage/{_test_table}")
|
||||||
_tl = _r.json()
|
_tl = _r.json()
|
||||||
_content = f"""
|
_content = f"""
|
||||||
### Table Lineage: `{_tl['table']}`
|
### Table Lineage: `{_tl["table"]}`
|
||||||
|
|
||||||
**Inputs:** {', '.join(f'`{i}`' for i in _tl['inputs'])}
|
**Inputs:** {", ".join(f"`{i}`" for i in _tl["inputs"])}
|
||||||
"""
|
"""
|
||||||
else:
|
else:
|
||||||
_content = "No table with dependencies found"
|
_content = "No table with dependencies found"
|
||||||
@@ -170,7 +174,7 @@ def _(client, mo):
|
|||||||
mo.md(f"""
|
mo.md(f"""
|
||||||
### Mermaid Export
|
### Mermaid Export
|
||||||
|
|
||||||
**Format:** {mermaid_data['format']} | **Lines:** {len(_lines)}
|
**Format:** {mermaid_data["format"]} | **Lines:** {len(_lines)}
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
{chr(10).join(_lines[:15])}
|
{chr(10).join(_lines[:15])}
|
||||||
@@ -189,7 +193,7 @@ def _(client, mo):
|
|||||||
mo.md(f"""
|
mo.md(f"""
|
||||||
### DOT Export
|
### DOT Export
|
||||||
|
|
||||||
**Format:** {dot_data['format']} | **Lines:** {len(_lines)}
|
**Format:** {dot_data["format"]} | **Lines:** {len(_lines)}
|
||||||
|
|
||||||
```dot
|
```dot
|
||||||
{chr(10).join(_lines[:10])}
|
{chr(10).join(_lines[:10])}
|
||||||
@@ -217,7 +221,9 @@ def _(client, mo):
|
|||||||
|
|
||||||
_item_rows = []
|
_item_rows = []
|
||||||
for _item in bib_items[:10]:
|
for _item in bib_items[:10]:
|
||||||
_item_rows.append(f"| {_item['key']} | {_item['title'][:50]} | {_item['item_type']} |")
|
_item_rows.append(
|
||||||
|
f"| {_item['key']} | {_item['title'][:50]} | {_item['item_type']} |"
|
||||||
|
)
|
||||||
|
|
||||||
_tag_rows = []
|
_tag_rows = []
|
||||||
for _tag in bib_tags[:10]:
|
for _tag in bib_tags[:10]:
|
||||||
@@ -253,7 +259,9 @@ def _(mo):
|
|||||||
def _(client, mo):
|
def _(client, mo):
|
||||||
_r = client.get("/schema/nonexistent_table")
|
_r = client.get("/schema/nonexistent_table")
|
||||||
missing_ok = _r.status_code == 404
|
missing_ok = _r.status_code == 404
|
||||||
mo.md(f"**GET /schema/nonexistent_table** \u2192 `{_r.status_code}` {'PASS' if missing_ok else 'FAIL'}")
|
mo.md(
|
||||||
|
f"**GET /schema/nonexistent_table** \u2192 `{_r.status_code}` {'PASS' if missing_ok else 'FAIL'}"
|
||||||
|
)
|
||||||
return (missing_ok,)
|
return (missing_ok,)
|
||||||
|
|
||||||
|
|
||||||
@@ -276,8 +284,8 @@ def _(client, mo):
|
|||||||
mo.md(f"""
|
mo.md(f"""
|
||||||
| Test | Status | Result |
|
| Test | Status | Result |
|
||||||
|------|--------|--------|
|
|------|--------|--------|
|
||||||
| Run requires auth | `{_r1.status_code}` | {'PASS' if auth_required else 'FAIL'} |
|
| Run requires auth | `{_r1.status_code}` | {"PASS" if auth_required else "FAIL"} |
|
||||||
| Wrong secret rejected | `{_r2.status_code}` | {'PASS' if wrong_secret else 'FAIL'} |
|
| Wrong secret rejected | `{_r2.status_code}` | {"PASS" if wrong_secret else "FAIL"} |
|
||||||
""")
|
""")
|
||||||
return auth_required, wrong_secret
|
return auth_required, wrong_secret
|
||||||
|
|
||||||
@@ -305,10 +313,14 @@ def _(
|
|||||||
tests = {
|
tests = {
|
||||||
"GET /health": health["status"] in ("ok", "degraded"),
|
"GET /health": health["status"] in ("ok", "degraded"),
|
||||||
"GET /pipelines": len(pipelines) > 0,
|
"GET /pipelines": len(pipelines) > 0,
|
||||||
"GET /pipelines/{name}": client.get(f"/pipelines/{pipelines[0]['name']}").status_code == 200,
|
"GET /pipelines/{name}": client.get(
|
||||||
|
f"/pipelines/{pipelines[0]['name']}"
|
||||||
|
).status_code
|
||||||
|
== 200,
|
||||||
"GET /pipelines/404": pipeline_404,
|
"GET /pipelines/404": pipeline_404,
|
||||||
"GET /lineage": len(lineage["tables"]) > 0,
|
"GET /lineage": len(lineage["tables"]) > 0,
|
||||||
"GET /lineage/export/mermaid": client.get("/lineage/export/mermaid").status_code == 200,
|
"GET /lineage/export/mermaid": client.get("/lineage/export/mermaid").status_code
|
||||||
|
== 200,
|
||||||
"GET /lineage/export/dot": client.get("/lineage/export/dot").status_code == 200,
|
"GET /lineage/export/dot": client.get("/lineage/export/dot").status_code == 200,
|
||||||
"GET /bib/items": client.get("/bib/items").status_code == 200,
|
"GET /bib/items": client.get("/bib/items").status_code == 200,
|
||||||
"GET /bib/tags": client.get("/bib/tags").status_code == 200,
|
"GET /bib/tags": client.get("/bib/tags").status_code == 200,
|
||||||
|
|||||||
@@ -29,11 +29,11 @@ def _():
|
|||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(mo):
|
def _(mo):
|
||||||
from bib.client import COLLECTIONS
|
from bib.client import COLLECTIONS
|
||||||
from bib.format import format_bibliography, format_citation
|
from bib.format import format_citation
|
||||||
from bib.item import Download, Item, Manual, Regulation, Rule, Source
|
from bib.item import Download, Manual, Regulation, Rule, Source
|
||||||
from bib.store import Store
|
from bib.tag import Tag
|
||||||
from bib.tag import Tag, filter_tags
|
from conf import connect
|
||||||
from conf import connect, path as _conf_path
|
from conf import path as _conf_path
|
||||||
|
|
||||||
store = connect.bib()
|
store = connect.bib()
|
||||||
ZOTERO_DB = _conf_path("db.zotero")
|
ZOTERO_DB = _conf_path("db.zotero")
|
||||||
@@ -98,7 +98,9 @@ def _(ZOTERO_DB, mo, store):
|
|||||||
_count = store._con().execute("SELECT COUNT(*) FROM items").fetchone()[0]
|
_count = store._con().execute("SELECT COUNT(*) FROM items").fetchone()[0]
|
||||||
_show_migrate = _count == 0 and ZOTERO_DB.exists()
|
_show_migrate = _count == 0 and ZOTERO_DB.exists()
|
||||||
|
|
||||||
migrate_btn = mo.ui.run_button(label="Migrate from Zotero") if _show_migrate else None
|
migrate_btn = (
|
||||||
|
mo.ui.run_button(label="Migrate from Zotero") if _show_migrate else None
|
||||||
|
)
|
||||||
migrate_btn
|
migrate_btn
|
||||||
return (migrate_btn,)
|
return (migrate_btn,)
|
||||||
|
|
||||||
@@ -127,7 +129,9 @@ def _(ZOTERO_DB, migrate_btn, mo, store):
|
|||||||
if _zc["parentCollectionID"]:
|
if _zc["parentCollectionID"]:
|
||||||
_pk = _col_id_to_key.get(_zc["parentCollectionID"])
|
_pk = _col_id_to_key.get(_zc["parentCollectionID"])
|
||||||
if _pk:
|
if _pk:
|
||||||
_pr = bcon.execute("SELECT id FROM collections WHERE key = ?", (_pk,)).fetchone()
|
_pr = bcon.execute(
|
||||||
|
"SELECT id FROM collections WHERE key = ?", (_pk,)
|
||||||
|
).fetchone()
|
||||||
if _pr:
|
if _pr:
|
||||||
_pid = _pr["id"]
|
_pid = _pr["id"]
|
||||||
bcon.execute(
|
bcon.execute(
|
||||||
@@ -172,23 +176,38 @@ def _(ZOTERO_DB, migrate_btn, mo, store):
|
|||||||
_it = "source"
|
_it = "source"
|
||||||
|
|
||||||
# Title
|
# Title
|
||||||
_title = _fields.get("nameOfAct", "") if _it in ("rule", "regulation") else _fields.get("title", "")
|
_title = (
|
||||||
_date = _fields.get("dateEnacted", "") if _it in ("rule", "regulation") else _fields.get("date", "")
|
_fields.get("nameOfAct", "")
|
||||||
|
if _it in ("rule", "regulation")
|
||||||
|
else _fields.get("title", "")
|
||||||
|
)
|
||||||
|
_date = (
|
||||||
|
_fields.get("dateEnacted", "")
|
||||||
|
if _it in ("rule", "regulation")
|
||||||
|
else _fields.get("date", "")
|
||||||
|
)
|
||||||
|
|
||||||
bcon.execute(
|
bcon.execute(
|
||||||
"""INSERT OR IGNORE INTO items
|
"""INSERT OR IGNORE INTO items
|
||||||
(key, item_type, title, url, date_published, access_date, abstract, institution, extra, extra_json)
|
(key, item_type, title, url, date_published, access_date, abstract, institution, extra, extra_json)
|
||||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||||
(
|
(
|
||||||
_zi["key"], _it, _title,
|
_zi["key"],
|
||||||
_fields.get("url", ""), _date,
|
_it,
|
||||||
_fields.get("accessDate", ""), _fields.get("abstractNote", ""),
|
_title,
|
||||||
|
_fields.get("url", ""),
|
||||||
|
_date,
|
||||||
|
_fields.get("accessDate", ""),
|
||||||
|
_fields.get("abstractNote", ""),
|
||||||
_fields.get("institution", "") or _fields.get("publisher", ""),
|
_fields.get("institution", "") or _fields.get("publisher", ""),
|
||||||
_fields.get("extra", ""), "{}",
|
_fields.get("extra", ""),
|
||||||
|
"{}",
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
_bib_row = bcon.execute("SELECT id FROM items WHERE key = ?", (_zi["key"],)).fetchone()
|
_bib_row = bcon.execute(
|
||||||
|
"SELECT id FROM items WHERE key = ?", (_zi["key"],)
|
||||||
|
).fetchone()
|
||||||
if _bib_row is None:
|
if _bib_row is None:
|
||||||
continue
|
continue
|
||||||
_bid = _bib_row["id"]
|
_bid = _bib_row["id"]
|
||||||
@@ -200,14 +219,19 @@ def _(ZOTERO_DB, migrate_btn, mo, store):
|
|||||||
(_zi["itemID"],),
|
(_zi["itemID"],),
|
||||||
).fetchall():
|
).fetchall():
|
||||||
_tid = store._ensure_tag(_tr["name"])
|
_tid = store._ensure_tag(_tr["name"])
|
||||||
bcon.execute("INSERT OR IGNORE INTO item_tags (item_id, tag_id) VALUES (?, ?)", (_bid, _tid))
|
bcon.execute(
|
||||||
|
"INSERT OR IGNORE INTO item_tags (item_id, tag_id) VALUES (?, ?)",
|
||||||
|
(_bid, _tid),
|
||||||
|
)
|
||||||
|
|
||||||
# Collection membership
|
# Collection membership
|
||||||
for _cr in zcon.execute(
|
for _cr in zcon.execute(
|
||||||
"SELECT c.key FROM collectionItems ci JOIN collections c ON ci.collectionID = c.collectionID WHERE ci.itemID = ?",
|
"SELECT c.key FROM collectionItems ci JOIN collections c ON ci.collectionID = c.collectionID WHERE ci.itemID = ?",
|
||||||
(_zi["itemID"],),
|
(_zi["itemID"],),
|
||||||
).fetchall():
|
).fetchall():
|
||||||
_cid_row = bcon.execute("SELECT id FROM collections WHERE key = ?", (_cr["key"],)).fetchone()
|
_cid_row = bcon.execute(
|
||||||
|
"SELECT id FROM collections WHERE key = ?", (_cr["key"],)
|
||||||
|
).fetchone()
|
||||||
if _cid_row:
|
if _cid_row:
|
||||||
bcon.execute(
|
bcon.execute(
|
||||||
"INSERT OR IGNORE INTO collection_items (collection_id, item_id) VALUES (?, ?)",
|
"INSERT OR IGNORE INTO collection_items (collection_id, item_id) VALUES (?, ?)",
|
||||||
@@ -242,10 +266,14 @@ def _(mo, store):
|
|||||||
|
|
||||||
_stats = {}
|
_stats = {}
|
||||||
_stats["Items"] = _con.execute("SELECT COUNT(*) FROM items").fetchone()[0]
|
_stats["Items"] = _con.execute("SELECT COUNT(*) FROM items").fetchone()[0]
|
||||||
_stats["Collections"] = _con.execute("SELECT COUNT(*) FROM collections").fetchone()[0]
|
_stats["Collections"] = _con.execute("SELECT COUNT(*) FROM collections").fetchone()[
|
||||||
|
0
|
||||||
|
]
|
||||||
_stats["Tags"] = _con.execute("SELECT COUNT(*) FROM tags").fetchone()[0]
|
_stats["Tags"] = _con.execute("SELECT COUNT(*) FROM tags").fetchone()[0]
|
||||||
_stats["Creators"] = _con.execute("SELECT COUNT(*) FROM creators").fetchone()[0]
|
_stats["Creators"] = _con.execute("SELECT COUNT(*) FROM creators").fetchone()[0]
|
||||||
_stats["Attachments"] = _con.execute("SELECT COUNT(*) FROM attachments").fetchone()[0]
|
_stats["Attachments"] = _con.execute("SELECT COUNT(*) FROM attachments").fetchone()[
|
||||||
|
0
|
||||||
|
]
|
||||||
_stats["Notes"] = _con.execute("SELECT COUNT(*) FROM notes").fetchone()[0]
|
_stats["Notes"] = _con.execute("SELECT COUNT(*) FROM notes").fetchone()[0]
|
||||||
|
|
||||||
_stat_table = "\n".join(f"| {k} | {v:,} |" for k, v in _stats.items())
|
_stat_table = "\n".join(f"| {k} | {v:,} |" for k, v in _stats.items())
|
||||||
@@ -269,13 +297,21 @@ def _(mo):
|
|||||||
def _(mo, store):
|
def _(mo, store):
|
||||||
import polars as pl
|
import polars as pl
|
||||||
|
|
||||||
_types_rows = store._con().execute(
|
_types_rows = (
|
||||||
"""SELECT item_type, COUNT(*) as count
|
store._con()
|
||||||
|
.execute(
|
||||||
|
"""SELECT item_type, COUNT(*) as count
|
||||||
FROM items GROUP BY item_type ORDER BY count DESC"""
|
FROM items GROUP BY item_type ORDER BY count DESC"""
|
||||||
).fetchall()
|
)
|
||||||
_types_df = pl.DataFrame([dict(r) for r in _types_rows]) if _types_rows else pl.DataFrame()
|
.fetchall()
|
||||||
|
)
|
||||||
|
_types_df = (
|
||||||
|
pl.DataFrame([dict(r) for r in _types_rows]) if _types_rows else pl.DataFrame()
|
||||||
|
)
|
||||||
|
|
||||||
mo.ui.table(_types_df, label="Item Types") if _types_df.height > 0 else mo.md("No items yet.")
|
mo.ui.table(_types_df, label="Item Types") if _types_df.height > 0 else mo.md(
|
||||||
|
"No items yet."
|
||||||
|
)
|
||||||
return (pl,)
|
return (pl,)
|
||||||
|
|
||||||
|
|
||||||
@@ -307,10 +343,18 @@ def _(mo, store):
|
|||||||
_stack = [(n, 0) for n in sorted(_roots, key=lambda x: x["name"], reverse=True)]
|
_stack = [(n, 0) for n in sorted(_roots, key=lambda x: x["name"], reverse=True)]
|
||||||
while _stack:
|
while _stack:
|
||||||
_node, _depth = _stack.pop()
|
_node, _depth = _stack.pop()
|
||||||
_tree_lines.append(f"{' ' * _depth}- **{_node['name']}** ({_node['item_count']} items)")
|
_tree_lines.append(
|
||||||
for _ch in sorted(_children.get(_node["key"], []), key=lambda x: x["name"], reverse=True):
|
f"{' ' * _depth}- **{_node['name']}** ({_node['item_count']} items)"
|
||||||
|
)
|
||||||
|
for _ch in sorted(
|
||||||
|
_children.get(_node["key"], []), key=lambda x: x["name"], reverse=True
|
||||||
|
):
|
||||||
_stack.append((_ch, _depth + 1))
|
_stack.append((_ch, _depth + 1))
|
||||||
mo.md("\n".join(_tree_lines) if _tree_lines else "No collections. Ensure collections above.")
|
mo.md(
|
||||||
|
"\n".join(_tree_lines)
|
||||||
|
if _tree_lines
|
||||||
|
else "No collections. Ensure collections above."
|
||||||
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
||||||
@@ -341,19 +385,20 @@ def _(mo):
|
|||||||
|
|
||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(mo, store):
|
def _(mo, store):
|
||||||
_type_opts = [""] + sorted({
|
_type_opts = [""] + sorted(
|
||||||
r[0] for r in store._con().execute(
|
{
|
||||||
"SELECT DISTINCT item_type FROM items"
|
r[0]
|
||||||
).fetchall()
|
for r in store._con()
|
||||||
})
|
.execute("SELECT DISTINCT item_type FROM items")
|
||||||
_tag_opts = [""] + [
|
.fetchall()
|
||||||
t["name"] for t in store.list_tags()
|
}
|
||||||
]
|
)
|
||||||
_col_opts = [""] + [
|
_tag_opts = [""] + [t["name"] for t in store.list_tags()]
|
||||||
f"{c['name']}|{c['key']}" for c in store.list_collections()
|
_col_opts = [""] + [f"{c['name']}|{c['key']}" for c in store.list_collections()]
|
||||||
]
|
|
||||||
|
|
||||||
search_input = mo.ui.text(placeholder="Search titles and abstracts...", label="Search", full_width=True)
|
search_input = mo.ui.text(
|
||||||
|
placeholder="Search titles and abstracts...", label="Search", full_width=True
|
||||||
|
)
|
||||||
type_filter = mo.ui.dropdown(options=_type_opts, value="", label="Type")
|
type_filter = mo.ui.dropdown(options=_type_opts, value="", label="Type")
|
||||||
tag_filter = mo.ui.dropdown(options=_tag_opts, value="", label="Tag")
|
tag_filter = mo.ui.dropdown(options=_tag_opts, value="", label="Tag")
|
||||||
col_filter = mo.ui.dropdown(options=_col_opts, value="", label="Collection")
|
col_filter = mo.ui.dropdown(options=_col_opts, value="", label="Collection")
|
||||||
@@ -415,6 +460,7 @@ def _(items_table, mo):
|
|||||||
if items_table is not None:
|
if items_table is not None:
|
||||||
_val = items_table.value
|
_val = items_table.value
|
||||||
import polars as _pl
|
import polars as _pl
|
||||||
|
|
||||||
if isinstance(_val, _pl.DataFrame) and _val.height > 0:
|
if isinstance(_val, _pl.DataFrame) and _val.height > 0:
|
||||||
_default_key = _val["key"][0]
|
_default_key = _val["key"][0]
|
||||||
elif isinstance(_val, list) and len(_val) > 0:
|
elif isinstance(_val, list) and len(_val) > 0:
|
||||||
@@ -470,7 +516,9 @@ def _(format_citation, key_input, mo, store):
|
|||||||
|
|
||||||
# Abstract
|
# Abstract
|
||||||
if _item.abstract:
|
if _item.abstract:
|
||||||
_sections.append(f"\n**Abstract**: {_item.abstract[:300]}{'...' if len(_item.abstract) > 300 else ''}")
|
_sections.append(
|
||||||
|
f"\n**Abstract**: {_item.abstract[:300]}{'...' if len(_item.abstract) > 300 else ''}"
|
||||||
|
)
|
||||||
|
|
||||||
# Citation preview
|
# Citation preview
|
||||||
_cite_apa = format_citation(_item, style="apa")
|
_cite_apa = format_citation(_item, style="apa")
|
||||||
@@ -481,22 +529,32 @@ def _(format_citation, key_input, mo, store):
|
|||||||
_sections.append(f"\n**Bluebook Citation**:\n> {_cite_bb}")
|
_sections.append(f"\n**Bluebook Citation**:\n> {_cite_bb}")
|
||||||
|
|
||||||
# Attachments
|
# Attachments
|
||||||
_atts = store._con().execute(
|
_atts = (
|
||||||
"""SELECT key, filename, content_type FROM attachments
|
store._con()
|
||||||
|
.execute(
|
||||||
|
"""SELECT key, filename, content_type FROM attachments
|
||||||
WHERE item_id = (SELECT id FROM items WHERE key = ?)""",
|
WHERE item_id = (SELECT id FROM items WHERE key = ?)""",
|
||||||
(_key,),
|
(_key,),
|
||||||
).fetchall()
|
)
|
||||||
|
.fetchall()
|
||||||
|
)
|
||||||
if _atts:
|
if _atts:
|
||||||
_sections.append("\n**Attachments**")
|
_sections.append("\n**Attachments**")
|
||||||
for _a in _atts:
|
for _a in _atts:
|
||||||
_sections.append(f"- `{_a['filename']}` ({_a['content_type'] or 'unknown'})")
|
_sections.append(
|
||||||
|
f"- `{_a['filename']}` ({_a['content_type'] or 'unknown'})"
|
||||||
|
)
|
||||||
|
|
||||||
# Notes
|
# Notes
|
||||||
_notes = store._con().execute(
|
_notes = (
|
||||||
"""SELECT title, content FROM notes
|
store._con()
|
||||||
|
.execute(
|
||||||
|
"""SELECT title, content FROM notes
|
||||||
WHERE item_id = (SELECT id FROM items WHERE key = ?)""",
|
WHERE item_id = (SELECT id FROM items WHERE key = ?)""",
|
||||||
(_key,),
|
(_key,),
|
||||||
).fetchall()
|
)
|
||||||
|
.fetchall()
|
||||||
|
)
|
||||||
if _notes:
|
if _notes:
|
||||||
_sections.append("\n**Notes**")
|
_sections.append("\n**Notes**")
|
||||||
for _n in _notes:
|
for _n in _notes:
|
||||||
@@ -527,14 +585,20 @@ def _(mo):
|
|||||||
)
|
)
|
||||||
new_title = mo.ui.text(placeholder="Title", label="Title", full_width=True)
|
new_title = mo.ui.text(placeholder="Title", label="Title", full_width=True)
|
||||||
new_url = mo.ui.text(placeholder="URL", label="URL", full_width=True)
|
new_url = mo.ui.text(placeholder="URL", label="URL", full_width=True)
|
||||||
new_tags_input = mo.ui.text(placeholder="module:pfs, year:2026", label="Tags (comma-separated)", full_width=True)
|
new_tags_input = mo.ui.text(
|
||||||
|
placeholder="module:pfs, year:2026",
|
||||||
|
label="Tags (comma-separated)",
|
||||||
|
full_width=True,
|
||||||
|
)
|
||||||
create_btn = mo.ui.run_button(label="Create Item")
|
create_btn = mo.ui.run_button(label="Create Item")
|
||||||
|
|
||||||
mo.vstack([
|
mo.vstack(
|
||||||
mo.hstack([new_type, new_title], widths=[1, 3]),
|
[
|
||||||
new_url,
|
mo.hstack([new_type, new_title], widths=[1, 3]),
|
||||||
mo.hstack([new_tags_input, create_btn], widths=[3, 1]),
|
new_url,
|
||||||
])
|
mo.hstack([new_tags_input, create_btn], widths=[3, 1]),
|
||||||
|
]
|
||||||
|
)
|
||||||
return create_btn, new_tags_input, new_title, new_type, new_url
|
return create_btn, new_tags_input, new_title, new_type, new_url
|
||||||
|
|
||||||
|
|
||||||
@@ -646,7 +710,7 @@ def _(
|
|||||||
_msg = f"""
|
_msg = f"""
|
||||||
Saved as `{_key}`: **{_item.title}**
|
Saved as `{_key}`: **{_item.title}**
|
||||||
|
|
||||||
Tags: {', '.join(f'`{t}`' for t in _item.tags)}
|
Tags: {", ".join(f"`{t}`" for t in _item.tags)}
|
||||||
|
|
||||||
> {_cite}
|
> {_cite}
|
||||||
"""
|
"""
|
||||||
@@ -670,11 +734,14 @@ def _(mo):
|
|||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(mo, store):
|
def _(mo, store):
|
||||||
_bib_tag_opts = [""] + [t["name"] for t in store.list_tags()]
|
_bib_tag_opts = [""] + [t["name"] for t in store.list_tags()]
|
||||||
_bib_type_opts = [""] + sorted({
|
_bib_type_opts = [""] + sorted(
|
||||||
r[0] for r in store._con().execute(
|
{
|
||||||
"SELECT DISTINCT item_type FROM items"
|
r[0]
|
||||||
).fetchall()
|
for r in store._con()
|
||||||
})
|
.execute("SELECT DISTINCT item_type FROM items")
|
||||||
|
.fetchall()
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
bib_tag = mo.ui.dropdown(options=_bib_tag_opts, value="", label="Filter by tag")
|
bib_tag = mo.ui.dropdown(options=_bib_tag_opts, value="", label="Filter by tag")
|
||||||
bib_type = mo.ui.dropdown(options=_bib_type_opts, value="", label="Filter by type")
|
bib_type = mo.ui.dropdown(options=_bib_type_opts, value="", label="Filter by type")
|
||||||
@@ -730,6 +797,7 @@ def _(mo):
|
|||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(mo):
|
def _(mo):
|
||||||
import os
|
import os
|
||||||
|
|
||||||
import s3fs
|
import s3fs
|
||||||
|
|
||||||
_endpoint = os.environ.get("S3_ENDPOINT", "http://rustfs:9000")
|
_endpoint = os.environ.get("S3_ENDPOINT", "http://rustfs:9000")
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
import marimo
|
import marimo
|
||||||
|
|
||||||
__generated_with = "0.20.2"
|
__generated_with = "0.21.1"
|
||||||
app = marimo.App(width="full")
|
app = marimo.App(width="full")
|
||||||
|
|
||||||
|
|
||||||
@@ -41,6 +41,7 @@ def _():
|
|||||||
import textwrap
|
import textwrap
|
||||||
|
|
||||||
import polars as pl
|
import polars as pl
|
||||||
|
|
||||||
from conf import connect
|
from conf import connect
|
||||||
|
|
||||||
con = connect.duckdb()
|
con = connect.duckdb()
|
||||||
@@ -1037,11 +1038,13 @@ def _(con, ex, inspect, mo, pl):
|
|||||||
else:
|
else:
|
||||||
nearest = min(avail, key=lambda y: abs(y - spec_year))
|
nearest = min(avail, key=lambda y: abs(y - spec_year))
|
||||||
_df = _df.filter(pl.col("performance_year") == nearest)
|
_df = _df.filter(pl.col("performance_year") == nearest)
|
||||||
_fallback_log.append({
|
_fallback_log.append(
|
||||||
"table": table_ref.split(".")[-1],
|
{
|
||||||
"requested_year": spec_year,
|
"table": table_ref.split(".")[-1],
|
||||||
"actual_year": nearest,
|
"requested_year": spec_year,
|
||||||
})
|
"actual_year": nearest,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
_year_cache[_key] = _df
|
_year_cache[_key] = _df
|
||||||
return _df
|
return _df
|
||||||
@@ -1074,33 +1077,66 @@ def _(con, ex, inspect, mo, pl):
|
|||||||
_LOOKBACK_BEGIN = _date(2017, 1, 1)
|
_LOOKBACK_BEGIN = _date(2017, 1, 1)
|
||||||
_LOOKBACK_END = _date(2017, 12, 31)
|
_LOOKBACK_END = _date(2017, 12, 31)
|
||||||
|
|
||||||
_uamcc_pp = pl.DataFrame({
|
_uamcc_pp = pl.DataFrame(
|
||||||
"measure_id": ["UAMCC"], "measure_name": ["All-Cause Unplanned Admissions for MCC"],
|
{
|
||||||
"nqf_id": ["2888"], "performance_year": [_PERF_YEAR],
|
"measure_id": ["UAMCC"],
|
||||||
"performance_period_begin": [_PERF_BEGIN], "performance_period_end": [_PERF_END],
|
"measure_name": ["All-Cause Unplanned Admissions for MCC"],
|
||||||
"lookback_period_begin": [_LOOKBACK_BEGIN], "lookback_period_end": [_LOOKBACK_END],
|
"nqf_id": ["2888"],
|
||||||
})
|
"performance_year": [_PERF_YEAR],
|
||||||
_acr_pp = pl.DataFrame({
|
"performance_period_begin": [_PERF_BEGIN],
|
||||||
"measure_id": ["ACR"], "measure_name": ["Risk-Standardized All-Condition Readmission"],
|
"performance_period_end": [_PERF_END],
|
||||||
"nqf_id": ["1789"], "performance_year": [_PERF_YEAR],
|
"lookback_period_begin": [_LOOKBACK_BEGIN],
|
||||||
"performance_period_begin": [_PERF_BEGIN], "performance_period_end": [_PERF_END],
|
"lookback_period_end": [_LOOKBACK_END],
|
||||||
})
|
}
|
||||||
_hwr_pp = pl.DataFrame({
|
)
|
||||||
"measure_id": ["HWR"], "measure_name": ["Hospital-Wide 30-Day All-Cause Unplanned Readmission"],
|
_acr_pp = pl.DataFrame(
|
||||||
"performance_year": [_PERF_YEAR],
|
{
|
||||||
"performance_period_begin": [_PERF_BEGIN], "performance_period_end": [_PERF_END],
|
"measure_id": ["ACR"],
|
||||||
})
|
"measure_name": ["Risk-Standardized All-Condition Readmission"],
|
||||||
|
"nqf_id": ["1789"],
|
||||||
|
"performance_year": [_PERF_YEAR],
|
||||||
|
"performance_period_begin": [_PERF_BEGIN],
|
||||||
|
"performance_period_end": [_PERF_END],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
_hwr_pp = pl.DataFrame(
|
||||||
|
{
|
||||||
|
"measure_id": ["HWR"],
|
||||||
|
"measure_name": ["Hospital-Wide 30-Day All-Cause Unplanned Readmission"],
|
||||||
|
"performance_year": [_PERF_YEAR],
|
||||||
|
"performance_period_begin": [_PERF_BEGIN],
|
||||||
|
"performance_period_end": [_PERF_END],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
_pipelines = {
|
_pipelines = {
|
||||||
"UAMCC": {
|
"UAMCC": {
|
||||||
"steps": [
|
"steps": [
|
||||||
("cms_quality_measures._uamcc_performance_period", ex.uamcc_performance_period),
|
(
|
||||||
|
"cms_quality_measures._uamcc_performance_period",
|
||||||
|
ex.uamcc_performance_period,
|
||||||
|
),
|
||||||
("cms_quality_measures._uamcc_int_mcc_cohort", ex.uamcc_int_mcc_cohort),
|
("cms_quality_measures._uamcc_int_mcc_cohort", ex.uamcc_int_mcc_cohort),
|
||||||
("cms_quality_measures._uamcc_int_denominator", ex.uamcc_int_denominator),
|
(
|
||||||
("cms_quality_measures._uamcc_int_denominator_exclusion", ex.uamcc_int_denominator_exclusion),
|
"cms_quality_measures._uamcc_int_denominator",
|
||||||
("cms_quality_measures._uamcc_int_planned_admission", ex.uamcc_int_planned_admission),
|
ex.uamcc_int_denominator,
|
||||||
("cms_quality_measures._uamcc_int_outcome_exclusion", ex.uamcc_int_outcome_exclusion),
|
),
|
||||||
("cms_quality_measures._uamcc_int_person_time", ex.uamcc_int_person_time),
|
(
|
||||||
|
"cms_quality_measures._uamcc_int_denominator_exclusion",
|
||||||
|
ex.uamcc_int_denominator_exclusion,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"cms_quality_measures._uamcc_int_planned_admission",
|
||||||
|
ex.uamcc_int_planned_admission,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"cms_quality_measures._uamcc_int_outcome_exclusion",
|
||||||
|
ex.uamcc_int_outcome_exclusion,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"cms_quality_measures._uamcc_int_person_time",
|
||||||
|
ex.uamcc_int_person_time,
|
||||||
|
),
|
||||||
("cms_quality_measures._uamcc_int_numerator", ex.uamcc_int_numerator),
|
("cms_quality_measures._uamcc_int_numerator", ex.uamcc_int_numerator),
|
||||||
("cms_quality_measures.uamcc_summary", ex.uamcc_summary),
|
("cms_quality_measures.uamcc_summary", ex.uamcc_summary),
|
||||||
],
|
],
|
||||||
@@ -1110,10 +1146,22 @@ def _(con, ex, inspect, mo, pl):
|
|||||||
},
|
},
|
||||||
"ACR": {
|
"ACR": {
|
||||||
"steps": [
|
"steps": [
|
||||||
("cms_quality_measures._acr_performance_period", ex.acr_performance_period),
|
(
|
||||||
("cms_quality_measures._acr_int_index_admission", ex.acr_int_index_admission),
|
"cms_quality_measures._acr_performance_period",
|
||||||
("cms_quality_measures._acr_int_specialty_cohort", ex.acr_int_specialty_cohort),
|
ex.acr_performance_period,
|
||||||
("cms_quality_measures._acr_int_planned_readmission", ex.acr_int_planned_readmission),
|
),
|
||||||
|
(
|
||||||
|
"cms_quality_measures._acr_int_index_admission",
|
||||||
|
ex.acr_int_index_admission,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"cms_quality_measures._acr_int_specialty_cohort",
|
||||||
|
ex.acr_int_specialty_cohort,
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"cms_quality_measures._acr_int_planned_readmission",
|
||||||
|
ex.acr_int_planned_readmission,
|
||||||
|
),
|
||||||
("cms_quality_measures.acr_summary", ex.acr_summary),
|
("cms_quality_measures.acr_summary", ex.acr_summary),
|
||||||
],
|
],
|
||||||
"pp_key": "cms_quality_measures._acr_performance_period",
|
"pp_key": "cms_quality_measures._acr_performance_period",
|
||||||
@@ -1122,9 +1170,15 @@ def _(con, ex, inspect, mo, pl):
|
|||||||
},
|
},
|
||||||
"HWR": {
|
"HWR": {
|
||||||
"steps": [
|
"steps": [
|
||||||
("cms_quality_measures._hwr_performance_period", ex.hwr_performance_period),
|
(
|
||||||
|
"cms_quality_measures._hwr_performance_period",
|
||||||
|
ex.hwr_performance_period,
|
||||||
|
),
|
||||||
("cms_quality_measures._hwr_int_denominator", ex.hwr_int_denominator),
|
("cms_quality_measures._hwr_int_denominator", ex.hwr_int_denominator),
|
||||||
("cms_quality_measures._hwr_int_planned_readmission", ex.hwr_int_planned_readmission),
|
(
|
||||||
|
"cms_quality_measures._hwr_int_planned_readmission",
|
||||||
|
ex.hwr_int_planned_readmission,
|
||||||
|
),
|
||||||
("cms_quality_measures.hwr_summary", ex.hwr_summary),
|
("cms_quality_measures.hwr_summary", ex.hwr_summary),
|
||||||
],
|
],
|
||||||
"pp_key": "cms_quality_measures._hwr_performance_period",
|
"pp_key": "cms_quality_measures._hwr_performance_period",
|
||||||
@@ -1168,20 +1222,24 @@ def _(con, ex, inspect, mo, pl):
|
|||||||
try:
|
try:
|
||||||
_result = _run_step_for_year(_fn, _cache, _sy)
|
_result = _run_step_for_year(_fn, _cache, _sy)
|
||||||
_cache[_step_name] = _result
|
_cache[_step_name] = _result
|
||||||
_step_detail_rows.append({
|
_step_detail_rows.append(
|
||||||
"measure": _mname,
|
{
|
||||||
"spec_year": _sy,
|
"measure": _mname,
|
||||||
"step": _step_name.split(".")[-1],
|
"spec_year": _sy,
|
||||||
"rows": len(_result),
|
"step": _step_name.split(".")[-1],
|
||||||
})
|
"rows": len(_result),
|
||||||
|
}
|
||||||
|
)
|
||||||
except Exception as _exc:
|
except Exception as _exc:
|
||||||
_error = str(_exc)
|
_error = str(_exc)
|
||||||
_step_detail_rows.append({
|
_step_detail_rows.append(
|
||||||
"measure": _mname,
|
{
|
||||||
"spec_year": _sy,
|
"measure": _mname,
|
||||||
"step": _step_name.split(".")[-1],
|
"spec_year": _sy,
|
||||||
"rows": -1,
|
"step": _step_name.split(".")[-1],
|
||||||
})
|
"rows": -1,
|
||||||
|
}
|
||||||
|
)
|
||||||
break
|
break
|
||||||
|
|
||||||
_summary = _cache.get(_mcfg["summary_key"])
|
_summary = _cache.get(_mcfg["summary_key"])
|
||||||
@@ -1191,9 +1249,13 @@ def _(con, ex, inspect, mo, pl):
|
|||||||
_row[c] = _summary[c][0]
|
_row[c] = _summary[c][0]
|
||||||
_sensitivity_rows.append(_row)
|
_sensitivity_rows.append(_row)
|
||||||
else:
|
else:
|
||||||
_sensitivity_rows.append({
|
_sensitivity_rows.append(
|
||||||
"measure": _mname, "spec_year": _sy, "error": _error or "no summary",
|
{
|
||||||
})
|
"measure": _mname,
|
||||||
|
"spec_year": _sy,
|
||||||
|
"error": _error or "no summary",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
if _sensitivity_rows:
|
if _sensitivity_rows:
|
||||||
_sens_df = pl.DataFrame(_sensitivity_rows)
|
_sens_df = pl.DataFrame(_sensitivity_rows)
|
||||||
@@ -1203,8 +1265,13 @@ def _(con, ex, inspect, mo, pl):
|
|||||||
# ── Build a concise comparison view ─────────────────────────────────
|
# ── Build a concise comparison view ─────────────────────────────────
|
||||||
_display_cols = ["measure", "spec_year"]
|
_display_cols = ["measure", "spec_year"]
|
||||||
_optional = [
|
_optional = [
|
||||||
"denominator_count", "observed_admissions", "total_person_years",
|
"denominator_count",
|
||||||
"observed_rate_per_100", "observed_readmissions", "observed_rate", "error",
|
"observed_admissions",
|
||||||
|
"total_person_years",
|
||||||
|
"observed_rate_per_100",
|
||||||
|
"observed_readmissions",
|
||||||
|
"observed_rate",
|
||||||
|
"error",
|
||||||
]
|
]
|
||||||
for _c in _optional:
|
for _c in _optional:
|
||||||
if _c in _sens_df.columns:
|
if _c in _sens_df.columns:
|
||||||
@@ -1237,28 +1304,30 @@ def _(con, ex, inspect, mo, pl):
|
|||||||
for _m, _piv in _step_pivots.items():
|
for _m, _piv in _step_pivots.items():
|
||||||
_items[f"{_m} — row counts per step"] = mo.ui.table(_piv)
|
_items[f"{_m} — row counts per step"] = mo.ui.table(_piv)
|
||||||
|
|
||||||
mo.vstack([
|
mo.vstack(
|
||||||
mo.md(f"""## Spec Year Sensitivity Analysis
|
[
|
||||||
|
mo.md(f"""## Spec Year Sensitivity Analysis
|
||||||
|
|
||||||
How do year-over-year changes in CMS value set specifications affect measure
|
How do year-over-year changes in CMS value set specifications affect measure
|
||||||
results on the **same population** (Synthea CY{_PERF_YEAR})?
|
results on the **same population** (Synthea CY{_PERF_YEAR})?
|
||||||
|
|
||||||
Each measure is re-run using value sets from each available performance year spec,
|
Each measure is re-run using value sets from each available performance year spec,
|
||||||
with nearest-year fallback for tables missing a specific year.
|
with nearest-year fallback for tables missing a specific year.
|
||||||
|
|
||||||
**Available spec years per measure:**
|
**Available spec years per measure:**
|
||||||
|
|
||||||
{_year_list_md}
|
{_year_list_md}
|
||||||
{_fb_md}
|
{_fb_md}
|
||||||
> **Note:** Identical results across spec years likely mean that the ~5,600 unique
|
> **Note:** Identical results across spec years likely mean that the ~5,600 unique
|
||||||
> diagnosis codes in the Synthea synthetic population don't overlap with the codes
|
> diagnosis codes in the Synthea synthetic population don't overlap with the codes
|
||||||
> that CMS added or removed between spec years. With real-world claims data covering
|
> that CMS added or removed between spec years. With real-world claims data covering
|
||||||
> a broader code space, spec year changes would be more likely to produce observable
|
> a broader code space, spec year changes would be more likely to produce observable
|
||||||
> differences in measure outcomes.
|
> differences in measure outcomes.
|
||||||
"""),
|
"""),
|
||||||
mo.ui.table(_sens_display, label="Summary by Spec Year"),
|
mo.ui.table(_sens_display, label="Summary by Spec Year"),
|
||||||
mo.accordion(_items),
|
mo.accordion(_items),
|
||||||
])
|
]
|
||||||
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
||||||
@@ -1269,37 +1338,172 @@ def _(con, mo, pl):
|
|||||||
|
|
||||||
# Every value-set table, its code column, measure, and functional role.
|
# Every value-set table, its code column, measure, and functional role.
|
||||||
_VS = [
|
_VS = [
|
||||||
("_uamcc_value_set_cohort", "icd_10_cm", "UAMCC", "MCC cohort inclusion", "ICD-10-CM"),
|
(
|
||||||
("_uamcc_value_set_exclusions", "category_or_code", "UAMCC", "Outcome exclusion", "CCS/ICD-10"),
|
"_uamcc_value_set_cohort",
|
||||||
("_uamcc_value_set_paa1", "ccs_procedure_category", "UAMCC", "PAA Rule 1 — always-planned procedure CCS", "CCS"),
|
"icd_10_cm",
|
||||||
("_uamcc_value_set_paa2", "ccs_diagnosis_category", "UAMCC", "PAA Rule 2 — always-planned diagnosis CCS", "CCS"),
|
"UAMCC",
|
||||||
("_uamcc_value_set_paa3", "category_or_code", "UAMCC", "PAA Rule 3 — potentially-planned procedure", "CCS/ICD-10-PCS"),
|
"MCC cohort inclusion",
|
||||||
("_uamcc_value_set_paa4", "category_or_code", "UAMCC", "PAA Rule 3 gate — acute diagnosis", "CCS/ICD-10-CM"),
|
"ICD-10-CM",
|
||||||
("_uamcc_value_set_ccs_icd10_cm", "icd_10_cm", "UAMCC", "CCS crosswalk (diagnosis)", "ICD-10-CM"),
|
),
|
||||||
("_uamcc_value_set_ccs_icd10_pcs", "icd_10_pcs", "UAMCC", "CCS crosswalk (procedure)", "ICD-10-PCS"),
|
(
|
||||||
("_acr_value_set_cohort_ccs", "ccs_category", "ACR", "Specialty cohort CCS", "CCS"),
|
"_uamcc_value_set_exclusions",
|
||||||
("_acr_value_set_cohort_icd10", "icd_10_pcs", "ACR", "Specialty cohort ICD-10-PCS", "ICD-10-PCS"),
|
"category_or_code",
|
||||||
("_acr_value_set_exclusions", "ccs_diagnosis_category", "ACR", "Cohort exclusion CCS", "CCS"),
|
"UAMCC",
|
||||||
("_acr_value_set_paa1", "ccs_procedure_category", "ACR", "PAA Rule 1 — always-planned procedure CCS", "CCS"),
|
"Outcome exclusion",
|
||||||
("_acr_value_set_paa2", "ccs_diagnosis_category", "ACR", "PAA Rule 2 — always-planned diagnosis CCS", "CCS"),
|
"CCS/ICD-10",
|
||||||
("_acr_value_set_paa3", "category_or_code", "ACR", "PAA Rule 3 — potentially-planned procedure", "CCS/ICD-10-PCS"),
|
),
|
||||||
("_acr_value_set_paa4", "category_or_code", "ACR", "PAA Rule 3 gate — acute diagnosis", "CCS/ICD-10-CM"),
|
(
|
||||||
("_hwr_value_set_specialty_cohort", "ccs_category", "HWR", "Specialty cohort CCS", "CCS"),
|
"_uamcc_value_set_paa1",
|
||||||
("_hwr_value_set_surg_gyn_cohort", "icd_10_pcs", "HWR", "Surgery/Gyn cohort ICD-10-PCS", "ICD-10-PCS"),
|
"ccs_procedure_category",
|
||||||
("_hwr_value_set_cohort_exclusions", "ccs_diagnosis_category", "HWR", "Cohort exclusion CCS", "CCS"),
|
"UAMCC",
|
||||||
("_hwr_value_set_paa1", "ccs_procedure_category", "HWR", "PAA Rule 1 — always-planned procedure CCS", "CCS"),
|
"PAA Rule 1 — always-planned procedure CCS",
|
||||||
("_hwr_value_set_paa2", "ccs_diagnosis_category", "HWR", "PAA Rule 2 — always-planned diagnosis CCS", "CCS"),
|
"CCS",
|
||||||
("_hwr_value_set_paa3", "category_or_code", "HWR", "PAA Rule 3 — potentially-planned procedure", "CCS/ICD-10-PCS"),
|
),
|
||||||
("_hwr_value_set_paa4", "category_or_code", "HWR", "PAA Rule 3 gate — acute diagnosis", "CCS/ICD-10-CM"),
|
(
|
||||||
|
"_uamcc_value_set_paa2",
|
||||||
|
"ccs_diagnosis_category",
|
||||||
|
"UAMCC",
|
||||||
|
"PAA Rule 2 — always-planned diagnosis CCS",
|
||||||
|
"CCS",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_uamcc_value_set_paa3",
|
||||||
|
"category_or_code",
|
||||||
|
"UAMCC",
|
||||||
|
"PAA Rule 3 — potentially-planned procedure",
|
||||||
|
"CCS/ICD-10-PCS",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_uamcc_value_set_paa4",
|
||||||
|
"category_or_code",
|
||||||
|
"UAMCC",
|
||||||
|
"PAA Rule 3 gate — acute diagnosis",
|
||||||
|
"CCS/ICD-10-CM",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_uamcc_value_set_ccs_icd10_cm",
|
||||||
|
"icd_10_cm",
|
||||||
|
"UAMCC",
|
||||||
|
"CCS crosswalk (diagnosis)",
|
||||||
|
"ICD-10-CM",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_uamcc_value_set_ccs_icd10_pcs",
|
||||||
|
"icd_10_pcs",
|
||||||
|
"UAMCC",
|
||||||
|
"CCS crosswalk (procedure)",
|
||||||
|
"ICD-10-PCS",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_acr_value_set_cohort_ccs",
|
||||||
|
"ccs_category",
|
||||||
|
"ACR",
|
||||||
|
"Specialty cohort CCS",
|
||||||
|
"CCS",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_acr_value_set_cohort_icd10",
|
||||||
|
"icd_10_pcs",
|
||||||
|
"ACR",
|
||||||
|
"Specialty cohort ICD-10-PCS",
|
||||||
|
"ICD-10-PCS",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_acr_value_set_exclusions",
|
||||||
|
"ccs_diagnosis_category",
|
||||||
|
"ACR",
|
||||||
|
"Cohort exclusion CCS",
|
||||||
|
"CCS",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_acr_value_set_paa1",
|
||||||
|
"ccs_procedure_category",
|
||||||
|
"ACR",
|
||||||
|
"PAA Rule 1 — always-planned procedure CCS",
|
||||||
|
"CCS",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_acr_value_set_paa2",
|
||||||
|
"ccs_diagnosis_category",
|
||||||
|
"ACR",
|
||||||
|
"PAA Rule 2 — always-planned diagnosis CCS",
|
||||||
|
"CCS",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_acr_value_set_paa3",
|
||||||
|
"category_or_code",
|
||||||
|
"ACR",
|
||||||
|
"PAA Rule 3 — potentially-planned procedure",
|
||||||
|
"CCS/ICD-10-PCS",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_acr_value_set_paa4",
|
||||||
|
"category_or_code",
|
||||||
|
"ACR",
|
||||||
|
"PAA Rule 3 gate — acute diagnosis",
|
||||||
|
"CCS/ICD-10-CM",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_hwr_value_set_specialty_cohort",
|
||||||
|
"ccs_category",
|
||||||
|
"HWR",
|
||||||
|
"Specialty cohort CCS",
|
||||||
|
"CCS",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_hwr_value_set_surg_gyn_cohort",
|
||||||
|
"icd_10_pcs",
|
||||||
|
"HWR",
|
||||||
|
"Surgery/Gyn cohort ICD-10-PCS",
|
||||||
|
"ICD-10-PCS",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_hwr_value_set_cohort_exclusions",
|
||||||
|
"ccs_diagnosis_category",
|
||||||
|
"HWR",
|
||||||
|
"Cohort exclusion CCS",
|
||||||
|
"CCS",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_hwr_value_set_paa1",
|
||||||
|
"ccs_procedure_category",
|
||||||
|
"HWR",
|
||||||
|
"PAA Rule 1 — always-planned procedure CCS",
|
||||||
|
"CCS",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_hwr_value_set_paa2",
|
||||||
|
"ccs_diagnosis_category",
|
||||||
|
"HWR",
|
||||||
|
"PAA Rule 2 — always-planned diagnosis CCS",
|
||||||
|
"CCS",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_hwr_value_set_paa3",
|
||||||
|
"category_or_code",
|
||||||
|
"HWR",
|
||||||
|
"PAA Rule 3 — potentially-planned procedure",
|
||||||
|
"CCS/ICD-10-PCS",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"_hwr_value_set_paa4",
|
||||||
|
"category_or_code",
|
||||||
|
"HWR",
|
||||||
|
"PAA Rule 3 gate — acute diagnosis",
|
||||||
|
"CCS/ICD-10-CM",
|
||||||
|
),
|
||||||
]
|
]
|
||||||
|
|
||||||
# Tables where `code_type` mixes CCS categories with redundant ICD-10
|
# Tables where `code_type` mixes CCS categories with redundant ICD-10
|
||||||
# detail expansions. The measure logic operates at CCS level — ICD-10
|
# detail expansions. The measure logic operates at CCS level — ICD-10
|
||||||
# detail rows are reference-only and should be diffed separately.
|
# detail rows are reference-only and should be diffed separately.
|
||||||
_HAS_CODE_TYPE = {
|
_HAS_CODE_TYPE = {
|
||||||
"_uamcc_value_set_paa3", "_uamcc_value_set_paa4",
|
"_uamcc_value_set_paa3",
|
||||||
"_acr_value_set_paa3", "_acr_value_set_paa4",
|
"_uamcc_value_set_paa4",
|
||||||
"_hwr_value_set_paa3", "_hwr_value_set_paa4",
|
"_acr_value_set_paa3",
|
||||||
|
"_acr_value_set_paa4",
|
||||||
|
"_hwr_value_set_paa3",
|
||||||
|
"_hwr_value_set_paa4",
|
||||||
"_uamcc_value_set_exclusions",
|
"_uamcc_value_set_exclusions",
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1307,9 +1511,12 @@ def _(con, mo, pl):
|
|||||||
"""Return {year: {norm_code: orig_code}} dicts."""
|
"""Return {year: {norm_code: orig_code}} dicts."""
|
||||||
_by_year = {}
|
_by_year = {}
|
||||||
for _y in _years:
|
for _y in _years:
|
||||||
_where = f" AND code_type = '{_code_type_filter}'" if _code_type_filter else ""
|
_where = (
|
||||||
|
f" AND code_type = '{_code_type_filter}'" if _code_type_filter else ""
|
||||||
|
)
|
||||||
_raw = [
|
_raw = [
|
||||||
r[0] for r in con.execute(
|
r[0]
|
||||||
|
for r in con.execute(
|
||||||
f'SELECT DISTINCT "{_key_col}" FROM {_q}'
|
f'SELECT DISTINCT "{_key_col}" FROM {_q}'
|
||||||
f" WHERE performance_year = {_y}"
|
f" WHERE performance_year = {_y}"
|
||||||
f' AND "{_key_col}" IS NOT NULL{_where}'
|
f' AND "{_key_col}" IS NOT NULL{_where}'
|
||||||
@@ -1326,17 +1533,29 @@ def _(con, mo, pl):
|
|||||||
_sa = set(_by_year[_ya])
|
_sa = set(_by_year[_ya])
|
||||||
_sb = set(_by_year[_yb])
|
_sb = set(_by_year[_yb])
|
||||||
for _c in sorted(_sb - _sa):
|
for _c in sorted(_sb - _sa):
|
||||||
_rows.append({
|
_rows.append(
|
||||||
"measure": _measure, "role": _role, "code_type": _code_type,
|
{
|
||||||
"code": _by_year[_yb][_c], "change": "added",
|
"measure": _measure,
|
||||||
"transition": f"{_ya} -> {_yb}", "table": _tbl,
|
"role": _role,
|
||||||
})
|
"code_type": _code_type,
|
||||||
|
"code": _by_year[_yb][_c],
|
||||||
|
"change": "added",
|
||||||
|
"transition": f"{_ya} -> {_yb}",
|
||||||
|
"table": _tbl,
|
||||||
|
}
|
||||||
|
)
|
||||||
for _c in sorted(_sa - _sb):
|
for _c in sorted(_sa - _sb):
|
||||||
_rows.append({
|
_rows.append(
|
||||||
"measure": _measure, "role": _role, "code_type": _code_type,
|
{
|
||||||
"code": _by_year[_ya][_c], "change": "removed",
|
"measure": _measure,
|
||||||
"transition": f"{_ya} -> {_yb}", "table": _tbl,
|
"role": _role,
|
||||||
})
|
"code_type": _code_type,
|
||||||
|
"code": _by_year[_ya][_c],
|
||||||
|
"change": "removed",
|
||||||
|
"transition": f"{_ya} -> {_yb}",
|
||||||
|
"table": _tbl,
|
||||||
|
}
|
||||||
|
)
|
||||||
return _rows
|
return _rows
|
||||||
|
|
||||||
_code_rows = []
|
_code_rows = []
|
||||||
@@ -1344,7 +1563,8 @@ def _(con, mo, pl):
|
|||||||
_q = f'cms_quality_measures."{_tbl}"'
|
_q = f'cms_quality_measures."{_tbl}"'
|
||||||
try:
|
try:
|
||||||
_years = sorted(
|
_years = sorted(
|
||||||
r[0] for r in con.execute(
|
r[0]
|
||||||
|
for r in con.execute(
|
||||||
f"SELECT DISTINCT performance_year FROM {_q} ORDER BY 1"
|
f"SELECT DISTINCT performance_year FROM {_q} ORDER BY 1"
|
||||||
).fetchall()
|
).fetchall()
|
||||||
)
|
)
|
||||||
@@ -1356,25 +1576,47 @@ def _(con, mo, pl):
|
|||||||
if _tbl in _HAS_CODE_TYPE:
|
if _tbl in _HAS_CODE_TYPE:
|
||||||
# Diff CCS-level entries (functionally meaningful)
|
# Diff CCS-level entries (functionally meaningful)
|
||||||
_ccs_sets = _build_code_sets(_q, _key_col, _years, "CCS")
|
_ccs_sets = _build_code_sets(_q, _key_col, _years, "CCS")
|
||||||
_code_rows.extend(_diff_years(
|
_code_rows.extend(
|
||||||
_ccs_sets, _years, _measure, _role + " (CCS — operative)", "CCS", _tbl,
|
_diff_years(
|
||||||
))
|
_ccs_sets,
|
||||||
|
_years,
|
||||||
|
_measure,
|
||||||
|
_role + " (CCS — operative)",
|
||||||
|
"CCS",
|
||||||
|
_tbl,
|
||||||
|
)
|
||||||
|
)
|
||||||
# Diff ICD-10 detail entries separately (reference-only)
|
# Diff ICD-10 detail entries separately (reference-only)
|
||||||
for _icd_type in ("ICD-10-CM", "ICD-10-PCS"):
|
for _icd_type in ("ICD-10-CM", "ICD-10-PCS"):
|
||||||
_icd_sets = _build_code_sets(_q, _key_col, _years, _icd_type)
|
_icd_sets = _build_code_sets(_q, _key_col, _years, _icd_type)
|
||||||
if any(len(v) > 0 for v in _icd_sets.values()):
|
if any(len(v) > 0 for v in _icd_sets.values()):
|
||||||
_code_rows.extend(_diff_years(
|
_code_rows.extend(
|
||||||
_icd_sets, _years, _measure,
|
_diff_years(
|
||||||
_role + f" ({_icd_type} — reference detail)", _icd_type, _tbl,
|
_icd_sets,
|
||||||
))
|
_years,
|
||||||
|
_measure,
|
||||||
|
_role + f" ({_icd_type} — reference detail)",
|
||||||
|
_icd_type,
|
||||||
|
_tbl,
|
||||||
|
)
|
||||||
|
)
|
||||||
else:
|
else:
|
||||||
_by_year = _build_code_sets(_q, _key_col, _years)
|
_by_year = _build_code_sets(_q, _key_col, _years)
|
||||||
_code_rows.extend(_diff_years(
|
_code_rows.extend(
|
||||||
_by_year, _years, _measure, _role, _code_type, _tbl,
|
_diff_years(
|
||||||
))
|
_by_year,
|
||||||
|
_years,
|
||||||
|
_measure,
|
||||||
|
_role,
|
||||||
|
_code_type,
|
||||||
|
_tbl,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
_codes_df = pl.DataFrame(_code_rows) if _code_rows else pl.DataFrame(
|
_codes_df = (
|
||||||
{"note": ["No code changes detected"]}
|
pl.DataFrame(_code_rows)
|
||||||
|
if _code_rows
|
||||||
|
else pl.DataFrame({"note": ["No code changes detected"]})
|
||||||
)
|
)
|
||||||
|
|
||||||
# Summary by measure × role × direction
|
# Summary by measure × role × direction
|
||||||
@@ -1386,47 +1628,53 @@ def _(con, mo, pl):
|
|||||||
|
|
||||||
# Impact classification: which changes could shift measure results?
|
# Impact classification: which changes could shift measure results?
|
||||||
_impact_md = """
|
_impact_md = """
|
||||||
| Change Type | Potential Impact | What to Query |
|
| Change Type | Potential Impact | What to Query |
|
||||||
|-------------|-----------------|---------------|
|
|-------------|-----------------|---------------|
|
||||||
| **Cohort inclusion** codes added | More patients enter the denominator | `WHERE dx_code IN ({codes}) AND encounter_type = 'acute inpatient'` |
|
| **Cohort inclusion** codes added | More patients enter the denominator | `WHERE dx_code IN ({codes}) AND encounter_type = 'acute inpatient'` |
|
||||||
| **Cohort inclusion** codes removed | Fewer patients in denominator | Same query — patients with these codes drop out |
|
| **Cohort inclusion** codes removed | Fewer patients in denominator | Same query — patients with these codes drop out |
|
||||||
| **Exclusion** codes added | More encounters excluded from numerator/denominator | `WHERE dx_ccs IN ({codes})` on your index admissions |
|
| **Exclusion** codes added | More encounters excluded from numerator/denominator | `WHERE dx_ccs IN ({codes})` on your index admissions |
|
||||||
| **Exclusion** codes removed | Fewer exclusions → larger effective denominator | Same query — previously excluded patients now included |
|
| **Exclusion** codes removed | Fewer exclusions → larger effective denominator | Same query — previously excluded patients now included |
|
||||||
| **PAA Rule 1/2/3** codes added | More admissions classified as *planned* → lower unplanned rate | `WHERE procedure_ccs IN ({codes})` or `WHERE dx_ccs IN ({codes})` on readmissions |
|
| **PAA Rule 1/2/3** codes added | More admissions classified as *planned* → lower unplanned rate | `WHERE procedure_ccs IN ({codes})` or `WHERE dx_ccs IN ({codes})` on readmissions |
|
||||||
| **PAA Rule 3 gate (acute dx)** codes added | More procedures remain *unplanned* (acute dx negates Rule 3) → higher unplanned rate | `WHERE dx_ccs IN ({codes})` on readmissions with potentially-planned procedures |
|
| **PAA Rule 3 gate (acute dx)** codes added | More procedures remain *unplanned* (acute dx negates Rule 3) → higher unplanned rate | `WHERE dx_ccs IN ({codes})` on readmissions with potentially-planned procedures |
|
||||||
| **CCS crosswalk** codes added/remapped | Diagnosis-to-CCS mapping changes cascade into all CCS-based logic above | `WHERE dx_code IN ({codes})` — check if CCS category assignment changed |
|
| **CCS crosswalk** codes added/remapped | Diagnosis-to-CCS mapping changes cascade into all CCS-based logic above | `WHERE dx_code IN ({codes})` — check if CCS category assignment changed |
|
||||||
"""
|
"""
|
||||||
|
|
||||||
mo.vstack([
|
mo.vstack(
|
||||||
mo.md("""## Research Strategy — Spec Year Code Changes
|
[
|
||||||
|
mo.md(
|
||||||
|
"""## Research Strategy — Spec Year Code Changes
|
||||||
|
|
||||||
To measure the real-world impact of spec year changes, query the **specific codes
|
To measure the real-world impact of spec year changes, query the **specific codes
|
||||||
that changed** against a target population. The tables below enumerate every code
|
that changed** against a target population. The tables below enumerate every code
|
||||||
added or removed between consecutive spec years, tagged by measure, functional role,
|
added or removed between consecutive spec years, tagged by measure, functional role,
|
||||||
and code type.
|
and code type.
|
||||||
|
|
||||||
### How spec changes propagate through the measures
|
### How spec changes propagate through the measures
|
||||||
|
|
||||||
""" + _impact_md + """
|
"""
|
||||||
|
+ _impact_md
|
||||||
|
+ """
|
||||||
|
|
||||||
### Step-by-step research protocol
|
### Step-by-step research protocol
|
||||||
|
|
||||||
1. **Export the code change table** below (CSV download via table widget)
|
1. **Export the code change table** below (CSV download via table widget)
|
||||||
2. **Filter to your measure of interest** (UAMCC, ACR, or HWR)
|
2. **Filter to your measure of interest** (UAMCC, ACR, or HWR)
|
||||||
3. **Query your claims population** for encounters matching the changed codes:
|
3. **Query your claims population** for encounters matching the changed codes:
|
||||||
- For ICD-10-CM changes: join on `principal_diagnosis_code` or `condition.normalized_code`
|
- For ICD-10-CM changes: join on `principal_diagnosis_code` or `condition.normalized_code`
|
||||||
- For ICD-10-PCS changes: join on `procedure.normalized_code` or `hcpcs_code`
|
- For ICD-10-PCS changes: join on `procedure.normalized_code` or `hcpcs_code`
|
||||||
- For CCS changes: first map your ICD codes through the CCS crosswalk, then match
|
- For CCS changes: first map your ICD codes through the CCS crosswalk, then match
|
||||||
4. **Count affected encounters** — the overlap between changed codes and your population
|
4. **Count affected encounters** — the overlap between changed codes and your population
|
||||||
determines whether the spec change would shift the measure result
|
determines whether the spec change would shift the measure result
|
||||||
5. **Re-run the pipeline** with each spec year's value sets (using the sensitivity
|
5. **Re-run the pipeline** with each spec year's value sets (using the sensitivity
|
||||||
analysis cell above) on your real data to quantify the actual difference
|
analysis cell above) on your real data to quantify the actual difference
|
||||||
"""),
|
"""
|
||||||
mo.md(f"### Change Summary — {len(_codes_df)} total code changes"),
|
),
|
||||||
mo.ui.table(_summary, label="Changes by Measure / Role / Direction"),
|
mo.md(f"### Change Summary — {len(_codes_df)} total code changes"),
|
||||||
mo.md("### Full Code Change Inventory"),
|
mo.ui.table(_summary, label="Changes by Measure / Role / Direction"),
|
||||||
mo.ui.table(_codes_df, label="All Changed Codes (exportable)"),
|
mo.md("### Full Code Change Inventory"),
|
||||||
])
|
mo.ui.table(_codes_df, label="All Changed Codes (exportable)"),
|
||||||
|
]
|
||||||
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -4,23 +4,23 @@ __generated_with = "0.19.7"
|
|||||||
app = marimo.App(width="medium")
|
app = marimo.App(width="medium")
|
||||||
|
|
||||||
with app.setup:
|
with app.setup:
|
||||||
import marimo as mo
|
|
||||||
import subprocess
|
|
||||||
import platform
|
|
||||||
import os
|
import os
|
||||||
|
import platform
|
||||||
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
import time
|
import time
|
||||||
|
|
||||||
|
import marimo as mo
|
||||||
import numpy as np
|
import numpy as np
|
||||||
import polars as pl
|
|
||||||
import pandas as pd
|
import pandas as pd
|
||||||
|
import polars as pl
|
||||||
|
|
||||||
|
|
||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def gpu_diagnostics():
|
def gpu_diagnostics():
|
||||||
try:
|
try:
|
||||||
smi_result = subprocess.run(
|
smi_result = subprocess.run(
|
||||||
["nvidia-smi"],
|
["nvidia-smi"], capture_output=True, text=True, timeout=10
|
||||||
capture_output=True, text=True, timeout=10
|
|
||||||
)
|
)
|
||||||
nvidia_smi_output = smi_result.stdout
|
nvidia_smi_output = smi_result.stdout
|
||||||
gpu_detected = smi_result.returncode == 0
|
gpu_detected = smi_result.returncode == 0
|
||||||
@@ -51,13 +51,15 @@ def polars_gpu_benchmark():
|
|||||||
bench_rng = np.random.default_rng(42)
|
bench_rng = np.random.default_rng(42)
|
||||||
|
|
||||||
bench_gen_start = time.perf_counter()
|
bench_gen_start = time.perf_counter()
|
||||||
bench_df = pl.DataFrame({
|
bench_df = pl.DataFrame(
|
||||||
"id": np.arange(benchmark_rows),
|
{
|
||||||
"group": bench_rng.choice(["A", "B", "C", "D", "E"], size=benchmark_rows),
|
"id": np.arange(benchmark_rows),
|
||||||
"value_1": bench_rng.standard_normal(benchmark_rows),
|
"group": bench_rng.choice(["A", "B", "C", "D", "E"], size=benchmark_rows),
|
||||||
"value_2": bench_rng.uniform(0, 1000, size=benchmark_rows),
|
"value_1": bench_rng.standard_normal(benchmark_rows),
|
||||||
"value_3": bench_rng.integers(0, 100, size=benchmark_rows),
|
"value_2": bench_rng.uniform(0, 1000, size=benchmark_rows),
|
||||||
})
|
"value_3": bench_rng.integers(0, 100, size=benchmark_rows),
|
||||||
|
}
|
||||||
|
)
|
||||||
bench_gen_elapsed = time.perf_counter() - bench_gen_start
|
bench_gen_elapsed = time.perf_counter() - bench_gen_start
|
||||||
|
|
||||||
# Pre-create lazy frame to exclude setup from timing
|
# Pre-create lazy frame to exclude setup from timing
|
||||||
@@ -77,8 +79,7 @@ def polars_gpu_benchmark():
|
|||||||
# --- GPU collect ---
|
# --- GPU collect ---
|
||||||
bench_gpu_agg_start = time.perf_counter()
|
bench_gpu_agg_start = time.perf_counter()
|
||||||
bench_gpu_agg_result = (
|
bench_gpu_agg_result = (
|
||||||
bench_lazy
|
bench_lazy.group_by("group")
|
||||||
.group_by("group")
|
|
||||||
.agg(*bench_agg_expr)
|
.agg(*bench_agg_expr)
|
||||||
.sort("group")
|
.sort("group")
|
||||||
.collect(engine="gpu")
|
.collect(engine="gpu")
|
||||||
@@ -88,15 +89,15 @@ def polars_gpu_benchmark():
|
|||||||
# --- CPU collect ---
|
# --- CPU collect ---
|
||||||
bench_cpu_agg_start = time.perf_counter()
|
bench_cpu_agg_start = time.perf_counter()
|
||||||
bench_cpu_agg_result = (
|
bench_cpu_agg_result = (
|
||||||
bench_lazy
|
bench_lazy.group_by("group").agg(*bench_agg_expr).sort("group").collect()
|
||||||
.group_by("group")
|
|
||||||
.agg(*bench_agg_expr)
|
|
||||||
.sort("group")
|
|
||||||
.collect()
|
|
||||||
)
|
)
|
||||||
bench_cpu_agg_elapsed = time.perf_counter() - bench_cpu_agg_start
|
bench_cpu_agg_elapsed = time.perf_counter() - bench_cpu_agg_start
|
||||||
|
|
||||||
bench_agg_speedup = bench_cpu_agg_elapsed / bench_gpu_agg_elapsed if bench_gpu_agg_elapsed > 0 else float("inf")
|
bench_agg_speedup = (
|
||||||
|
bench_cpu_agg_elapsed / bench_gpu_agg_elapsed
|
||||||
|
if bench_gpu_agg_elapsed > 0
|
||||||
|
else float("inf")
|
||||||
|
)
|
||||||
|
|
||||||
# GPU-supported window functions only (no rank which causes fallback)
|
# GPU-supported window functions only (no rank which causes fallback)
|
||||||
bench_window_expr = [
|
bench_window_expr = [
|
||||||
@@ -107,24 +108,22 @@ def polars_gpu_benchmark():
|
|||||||
# --- GPU window ---
|
# --- GPU window ---
|
||||||
bench_gpu_window_start = time.perf_counter()
|
bench_gpu_window_start = time.perf_counter()
|
||||||
bench_gpu_window_result = (
|
bench_gpu_window_result = (
|
||||||
bench_lazy
|
bench_lazy.with_columns(*bench_window_expr).head(5).collect(engine="gpu")
|
||||||
.with_columns(*bench_window_expr)
|
|
||||||
.head(5)
|
|
||||||
.collect(engine="gpu")
|
|
||||||
)
|
)
|
||||||
bench_gpu_window_elapsed = time.perf_counter() - bench_gpu_window_start
|
bench_gpu_window_elapsed = time.perf_counter() - bench_gpu_window_start
|
||||||
|
|
||||||
# --- CPU window ---
|
# --- CPU window ---
|
||||||
bench_cpu_window_start = time.perf_counter()
|
bench_cpu_window_start = time.perf_counter()
|
||||||
bench_cpu_window_result = (
|
bench_cpu_window_result = (
|
||||||
bench_lazy
|
bench_lazy.with_columns(*bench_window_expr).head(5).collect()
|
||||||
.with_columns(*bench_window_expr)
|
|
||||||
.head(5)
|
|
||||||
.collect()
|
|
||||||
)
|
)
|
||||||
bench_cpu_window_elapsed = time.perf_counter() - bench_cpu_window_start
|
bench_cpu_window_elapsed = time.perf_counter() - bench_cpu_window_start
|
||||||
|
|
||||||
bench_window_speedup = bench_cpu_window_elapsed / bench_gpu_window_elapsed if bench_gpu_window_elapsed > 0 else float("inf")
|
bench_window_speedup = (
|
||||||
|
bench_cpu_window_elapsed / bench_gpu_window_elapsed
|
||||||
|
if bench_gpu_window_elapsed > 0
|
||||||
|
else float("inf")
|
||||||
|
)
|
||||||
|
|
||||||
mo.md(f"""
|
mo.md(f"""
|
||||||
# Polars GPU vs CPU — {benchmark_rows:,} rows
|
# Polars GPU vs CPU — {benchmark_rows:,} rows
|
||||||
@@ -136,10 +135,12 @@ def polars_gpu_benchmark():
|
|||||||
| Window functions | `{bench_gpu_window_elapsed:.3f}s` | `{bench_cpu_window_elapsed:.3f}s` | **{bench_window_speedup:.1f}x** |
|
| Window functions | `{bench_gpu_window_elapsed:.3f}s` | `{bench_cpu_window_elapsed:.3f}s` | **{bench_window_speedup:.1f}x** |
|
||||||
""")
|
""")
|
||||||
|
|
||||||
mo.hstack([
|
mo.hstack(
|
||||||
mo.ui.table(bench_gpu_agg_result, label="GPU Aggregation Results"),
|
[
|
||||||
mo.ui.table(bench_gpu_window_result, label="GPU Window Functions (head 5)"),
|
mo.ui.table(bench_gpu_agg_result, label="GPU Aggregation Results"),
|
||||||
])
|
mo.ui.table(bench_gpu_window_result, label="GPU Window Functions (head 5)"),
|
||||||
|
]
|
||||||
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
||||||
@@ -155,7 +156,9 @@ def pandas_vs_polars_gpu():
|
|||||||
# --- Pandas (CPU only) - time only the compute, not DataFrame creation ---
|
# --- Pandas (CPU only) - time only the compute, not DataFrame creation ---
|
||||||
cmp_pandas_df = pd.DataFrame({"category": cmp_categories, "amount": cmp_amounts})
|
cmp_pandas_df = pd.DataFrame({"category": cmp_categories, "amount": cmp_amounts})
|
||||||
cmp_pandas_start = time.perf_counter()
|
cmp_pandas_start = time.perf_counter()
|
||||||
cmp_pandas_agg = cmp_pandas_df.groupby("category")["amount"].agg(["mean", "std", "sum"])
|
cmp_pandas_agg = cmp_pandas_df.groupby("category")["amount"].agg(
|
||||||
|
["mean", "std", "sum"]
|
||||||
|
)
|
||||||
cmp_pandas_elapsed = time.perf_counter() - cmp_pandas_start
|
cmp_pandas_elapsed = time.perf_counter() - cmp_pandas_start
|
||||||
|
|
||||||
# --- Polars GPU - time only the compute ---
|
# --- Polars GPU - time only the compute ---
|
||||||
@@ -164,8 +167,7 @@ def pandas_vs_polars_gpu():
|
|||||||
|
|
||||||
cmp_gpu_start = time.perf_counter()
|
cmp_gpu_start = time.perf_counter()
|
||||||
cmp_gpu_agg = (
|
cmp_gpu_agg = (
|
||||||
cmp_lazy
|
cmp_lazy.group_by("category")
|
||||||
.group_by("category")
|
|
||||||
.agg(
|
.agg(
|
||||||
pl.col("amount").mean().alias("mean"),
|
pl.col("amount").mean().alias("mean"),
|
||||||
pl.col("amount").std().alias("std"),
|
pl.col("amount").std().alias("std"),
|
||||||
@@ -179,8 +181,7 @@ def pandas_vs_polars_gpu():
|
|||||||
# --- Polars CPU ---
|
# --- Polars CPU ---
|
||||||
cmp_cpu_start = time.perf_counter()
|
cmp_cpu_start = time.perf_counter()
|
||||||
cmp_cpu_agg = (
|
cmp_cpu_agg = (
|
||||||
cmp_lazy
|
cmp_lazy.group_by("category")
|
||||||
.group_by("category")
|
|
||||||
.agg(
|
.agg(
|
||||||
pl.col("amount").mean().alias("mean"),
|
pl.col("amount").mean().alias("mean"),
|
||||||
pl.col("amount").std().alias("std"),
|
pl.col("amount").std().alias("std"),
|
||||||
@@ -191,8 +192,12 @@ def pandas_vs_polars_gpu():
|
|||||||
)
|
)
|
||||||
cmp_cpu_elapsed = time.perf_counter() - cmp_cpu_start
|
cmp_cpu_elapsed = time.perf_counter() - cmp_cpu_start
|
||||||
|
|
||||||
cmp_gpu_vs_pandas = cmp_pandas_elapsed / cmp_gpu_elapsed if cmp_gpu_elapsed > 0 else float("inf")
|
cmp_gpu_vs_pandas = (
|
||||||
cmp_cpu_vs_pandas = cmp_pandas_elapsed / cmp_cpu_elapsed if cmp_cpu_elapsed > 0 else float("inf")
|
cmp_pandas_elapsed / cmp_gpu_elapsed if cmp_gpu_elapsed > 0 else float("inf")
|
||||||
|
)
|
||||||
|
cmp_cpu_vs_pandas = (
|
||||||
|
cmp_pandas_elapsed / cmp_cpu_elapsed if cmp_cpu_elapsed > 0 else float("inf")
|
||||||
|
)
|
||||||
|
|
||||||
mo.md(f"""
|
mo.md(f"""
|
||||||
# Three-Way Comparison — {cmp_rows:,} rows
|
# Three-Way Comparison — {cmp_rows:,} rows
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ app = marimo.App(width="medium")
|
|||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _():
|
def _():
|
||||||
import marimo as mo
|
import marimo as mo
|
||||||
|
|
||||||
mo.md("""
|
mo.md("""
|
||||||
# Nessie Tutorial - Git for Your Data Lake
|
# Nessie Tutorial - Git for Your Data Lake
|
||||||
|
|
||||||
@@ -19,9 +20,10 @@ def _():
|
|||||||
|
|
||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _():
|
def _():
|
||||||
import requests
|
|
||||||
import json
|
import json
|
||||||
|
|
||||||
|
import requests
|
||||||
|
|
||||||
from conf import cfg
|
from conf import cfg
|
||||||
|
|
||||||
NESSIE_API = f"{cfg.services.nessie}/api/v2"
|
NESSIE_API = f"{cfg.services.nessie}/api/v2"
|
||||||
@@ -98,11 +100,9 @@ def _(NESSIE_API, json, main_ref, requests):
|
|||||||
resp = requests.post(
|
resp = requests.post(
|
||||||
f"{NESSIE_API}/trees?name={new_branch}&type=branch",
|
f"{NESSIE_API}/trees?name={new_branch}&type=branch",
|
||||||
headers={"Content-Type": "application/json"},
|
headers={"Content-Type": "application/json"},
|
||||||
data=json.dumps({
|
data=json.dumps(
|
||||||
"type": "BRANCH",
|
{"type": "BRANCH", "name": main_ref["name"], "hash": main_ref["hash"]}
|
||||||
"name": main_ref["name"],
|
),
|
||||||
"hash": main_ref["hash"]
|
|
||||||
})
|
|
||||||
)
|
)
|
||||||
|
|
||||||
if resp.status_code == 200:
|
if resp.status_code == 200:
|
||||||
@@ -249,7 +249,9 @@ def _(mo):
|
|||||||
|
|
||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(query):
|
def _(query):
|
||||||
count = query("SELECT count(*) FROM iceberg.tutorial.events FOR VERSION AS OF 'main'")
|
count = query(
|
||||||
|
"SELECT count(*) FROM iceberg.tutorial.events FOR VERSION AS OF 'main'"
|
||||||
|
)
|
||||||
print(f"Events on main: {count[0][0]}")
|
print(f"Events on main: {count[0][0]}")
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -271,8 +273,7 @@ def _(NESSIE_API, requests):
|
|||||||
print(f"Branch {name} not found")
|
print(f"Branch {name} not found")
|
||||||
return
|
return
|
||||||
resp = requests.delete(
|
resp = requests.delete(
|
||||||
f"{NESSIE_API}/trees/{name}",
|
f"{NESSIE_API}/trees/{name}", headers={"Expected-Hash": _hash}
|
||||||
headers={"Expected-Hash": _hash}
|
|
||||||
)
|
)
|
||||||
if resp.status_code == 204:
|
if resp.status_code == 204:
|
||||||
print(f"Deleted: {name}")
|
print(f"Deleted: {name}")
|
||||||
|
|||||||
@@ -25,6 +25,7 @@ def _(mo):
|
|||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _():
|
def _():
|
||||||
import polars as pl
|
import polars as pl
|
||||||
|
|
||||||
from conf import connect
|
from conf import connect
|
||||||
|
|
||||||
con = connect.duckdb()
|
con = connect.duckdb()
|
||||||
@@ -51,10 +52,7 @@ def _(con, mo):
|
|||||||
"""
|
"""
|
||||||
).fetchall()
|
).fetchall()
|
||||||
|
|
||||||
_loc_options = {
|
_loc_options = {f"{name} ({loc})": f"{mac}|{loc}" for mac, loc, name in _localities}
|
||||||
f"{name} ({loc})": f"{mac}|{loc}"
|
|
||||||
for mac, loc, name in _localities
|
|
||||||
}
|
|
||||||
_first_key = next(iter(_loc_options))
|
_first_key = next(iter(_loc_options))
|
||||||
|
|
||||||
year_picker = mo.ui.dropdown(
|
year_picker = mo.ui.dropdown(
|
||||||
@@ -139,7 +137,7 @@ def _(hcpcs_input, locality_picker, pl, q, year_picker):
|
|||||||
_cf = RULES[_year].conversion_factor
|
_cf = RULES[_year].conversion_factor
|
||||||
|
|
||||||
rvu_df = q(f"""
|
rvu_df = q(f"""
|
||||||
SELECT *, {_cf} as conv_factor, '{_locality}' as locality, '{_mac}' as mac
|
SELECT *, '{_locality}' as locality, '{_mac}' as mac
|
||||||
FROM pfs.rvu
|
FROM pfs.rvu
|
||||||
WHERE year = {_year} AND hcpcs = '{_hcpcs}'
|
WHERE year = {_year} AND hcpcs = '{_hcpcs}'
|
||||||
""")
|
""")
|
||||||
@@ -149,14 +147,21 @@ def _(hcpcs_input, locality_picker, pl, q, year_picker):
|
|||||||
WHERE year = {_year} AND mac = '{_mac}' AND locality = '{_locality}'
|
WHERE year = {_year} AND mac = '{_mac}' AND locality = '{_locality}'
|
||||||
""")
|
""")
|
||||||
|
|
||||||
calc_nf = payment(rvu_df, gpci_df, facility=False).select(
|
calc_nf = payment(rvu_df, gpci_df, cf=_cf, facility=False).select(
|
||||||
"hcpcs", "mod", "work_rvu", "non_fac_pe_rvu", "mp_rvu",
|
"hcpcs",
|
||||||
"work_gpci", "pe_gpci", "mp_gpci", "conv_factor",
|
"mod",
|
||||||
|
"work_rvu",
|
||||||
|
"non_fac_pe_rvu",
|
||||||
|
"mp_rvu",
|
||||||
|
"work_gpci",
|
||||||
|
"pe_gpci",
|
||||||
|
"mp_gpci",
|
||||||
pl.col("payment_amount").round(2).alias("non_fac_payment"),
|
pl.col("payment_amount").round(2).alias("non_fac_payment"),
|
||||||
)
|
)
|
||||||
|
|
||||||
calc_f = payment(rvu_df, gpci_df, facility=True).select(
|
calc_f = payment(rvu_df, gpci_df, cf=_cf, facility=True).select(
|
||||||
"hcpcs", "mod",
|
"hcpcs",
|
||||||
|
"mod",
|
||||||
pl.col("payment_amount").round(2).alias("fac_payment"),
|
pl.col("payment_amount").round(2).alias("fac_payment"),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -253,7 +258,9 @@ def _(carrier_result, mo, pl, calc_result):
|
|||||||
indicates a carrier-priced or status-indicator edge case.
|
indicates a carrier-priced or status-indicator edge case.
|
||||||
"""
|
"""
|
||||||
else:
|
else:
|
||||||
_comparison = "*Select a valid year/locality/code combination to see comparison.*"
|
_comparison = (
|
||||||
|
"*Select a valid year/locality/code combination to see comparison.*"
|
||||||
|
)
|
||||||
|
|
||||||
mo.md(_comparison)
|
mo.md(_comparison)
|
||||||
return
|
return
|
||||||
|
|||||||
@@ -13,15 +13,13 @@ def _():
|
|||||||
|
|
||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(mo):
|
def _(mo):
|
||||||
mo.md(
|
mo.md("""
|
||||||
"""
|
|
||||||
# PFS Reconciliation
|
# PFS Reconciliation
|
||||||
|
|
||||||
Runs `rec.pricers.pfs.PfsPricer` against `pfs.carrier_locality`
|
Runs `rec.pricers.pfs.PfsPricer` against `pfs.carrier_locality`
|
||||||
for a chosen year and reports the per-row delta. Goal is perfect
|
for a chosen year and reports the per-row delta. Goal is perfect
|
||||||
1:1 concordance. Tracks **homelab/stack#340**.
|
1:1 concordance. Tracks **homelab/stack#340**.
|
||||||
"""
|
""")
|
||||||
)
|
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
||||||
@@ -58,7 +56,7 @@ def _(con, mo, pricer):
|
|||||||
return tolerance, year_picker
|
return tolerance, year_picker
|
||||||
|
|
||||||
|
|
||||||
@app.cell
|
@app.cell(hide_code=True)
|
||||||
def _(con, mo, pricer, reconcile, tolerance, year_picker):
|
def _(con, mo, pricer, reconcile, tolerance, year_picker):
|
||||||
_year = int(year_picker.value) if year_picker.value else 0
|
_year = int(year_picker.value) if year_picker.value else 0
|
||||||
if _year == 0:
|
if _year == 0:
|
||||||
@@ -72,20 +70,18 @@ def _(con, mo, pricer, reconcile, tolerance, year_picker):
|
|||||||
|
|
||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(mo):
|
def _(mo):
|
||||||
mo.md(
|
mo.md("""
|
||||||
"""
|
|
||||||
## Delta table
|
## Delta table
|
||||||
|
|
||||||
Every row in the outer join, sorted by the largest absolute delta.
|
Every row in the outer join, sorted by the largest absolute delta.
|
||||||
``is_exact`` uses the tolerance above; ``is_near`` is always a 1¢
|
``is_exact`` uses the tolerance above; ``is_near`` is always a 1¢
|
||||||
window. Null ``fee_gt`` means the row is calculated-only; null
|
window. Null ``fee_gt`` means the row is calculated-only; null
|
||||||
``fee_calc`` means ground-truth-only.
|
``fee_calc`` means ground-truth-only.
|
||||||
"""
|
""")
|
||||||
)
|
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
||||||
@app.cell
|
@app.cell(hide_code=True)
|
||||||
def _(result):
|
def _(result):
|
||||||
_deltas = result.deltas if result is not None else None
|
_deltas = result.deltas if result is not None else None
|
||||||
_deltas
|
_deltas
|
||||||
@@ -94,18 +90,16 @@ def _(result):
|
|||||||
|
|
||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(mo):
|
def _(mo):
|
||||||
mo.md(
|
mo.md("""
|
||||||
"""
|
|
||||||
## Warnings
|
## Warnings
|
||||||
|
|
||||||
Non-fatal issues surfaced by the engine — duplicate join keys,
|
Non-fatal issues surfaced by the engine — duplicate join keys,
|
||||||
missing columns, rule lookup failures, etc.
|
missing columns, rule lookup failures, etc.
|
||||||
"""
|
""")
|
||||||
)
|
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
||||||
@app.cell
|
@app.cell(hide_code=True)
|
||||||
def _(mo, result):
|
def _(mo, result):
|
||||||
if result is None or not result.warnings:
|
if result is None or not result.warnings:
|
||||||
mo.md("*(none)*")
|
mo.md("*(none)*")
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ app = marimo.App(width="medium")
|
|||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _():
|
def _():
|
||||||
import marimo as mo
|
import marimo as mo
|
||||||
|
|
||||||
mo.md("""
|
mo.md("""
|
||||||
# Apache Polaris Tutorial - Iceberg Catalog with Governance
|
# Apache Polaris Tutorial - Iceberg Catalog with Governance
|
||||||
|
|
||||||
@@ -20,10 +21,11 @@ def _():
|
|||||||
|
|
||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _():
|
def _():
|
||||||
import requests
|
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
|
||||||
|
import requests
|
||||||
|
|
||||||
from conf import cfg
|
from conf import cfg
|
||||||
|
|
||||||
POLARIS_API = cfg.services.polaris
|
POLARIS_API = cfg.services.polaris
|
||||||
@@ -51,8 +53,8 @@ def _(POLARIS_API, CLIENT_ID, CLIENT_SECRET, requests):
|
|||||||
"grant_type": "client_credentials",
|
"grant_type": "client_credentials",
|
||||||
"client_id": CLIENT_ID,
|
"client_id": CLIENT_ID,
|
||||||
"client_secret": CLIENT_SECRET,
|
"client_secret": CLIENT_SECRET,
|
||||||
"scope": "PRINCIPAL_ROLE:ALL"
|
"scope": "PRINCIPAL_ROLE:ALL",
|
||||||
}
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
if token_resp.status_code == 200:
|
if token_resp.status_code == 200:
|
||||||
@@ -81,8 +83,7 @@ def _(mo):
|
|||||||
def _(POLARIS_API, access_token, requests):
|
def _(POLARIS_API, access_token, requests):
|
||||||
headers = {"Authorization": f"Bearer {access_token}"}
|
headers = {"Authorization": f"Bearer {access_token}"}
|
||||||
catalogs_resp = requests.get(
|
catalogs_resp = requests.get(
|
||||||
f"{POLARIS_API}/api/management/v1/catalogs",
|
f"{POLARIS_API}/api/management/v1/catalogs", headers=headers
|
||||||
headers=headers
|
|
||||||
)
|
)
|
||||||
|
|
||||||
if catalogs_resp.status_code == 200:
|
if catalogs_resp.status_code == 200:
|
||||||
@@ -114,22 +115,24 @@ def _(POLARIS_API, headers, json, requests):
|
|||||||
create_resp = requests.post(
|
create_resp = requests.post(
|
||||||
f"{POLARIS_API}/api/management/v1/catalogs",
|
f"{POLARIS_API}/api/management/v1/catalogs",
|
||||||
headers={**headers, "Content-Type": "application/json"},
|
headers={**headers, "Content-Type": "application/json"},
|
||||||
data=json.dumps({
|
data=json.dumps(
|
||||||
"name": catalog_name,
|
{
|
||||||
"type": "INTERNAL",
|
"name": catalog_name,
|
||||||
"properties": {
|
"type": "INTERNAL",
|
||||||
"default-base-location": f"s3://polaris/{catalog_name}/"
|
"properties": {
|
||||||
},
|
"default-base-location": f"s3://polaris/{catalog_name}/"
|
||||||
"storageConfigInfo": {
|
},
|
||||||
"storageType": "S3",
|
"storageConfigInfo": {
|
||||||
"allowedLocations": ["s3://polaris/", "s3://lakehouse/"],
|
"storageType": "S3",
|
||||||
"s3": {
|
"allowedLocations": ["s3://polaris/", "s3://lakehouse/"],
|
||||||
"region": "us-east-1",
|
"s3": {
|
||||||
"endpoint": cfg.services.rustfs,
|
"region": "us-east-1",
|
||||||
"pathStyleAccess": True
|
"endpoint": cfg.services.rustfs,
|
||||||
}
|
"pathStyleAccess": True,
|
||||||
|
},
|
||||||
|
},
|
||||||
}
|
}
|
||||||
})
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
if create_resp.status_code == 200:
|
if create_resp.status_code == 200:
|
||||||
@@ -153,8 +156,7 @@ def _(mo):
|
|||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(POLARIS_API, catalog_name, headers, requests):
|
def _(POLARIS_API, catalog_name, headers, requests):
|
||||||
cat_resp = requests.get(
|
cat_resp = requests.get(
|
||||||
f"{POLARIS_API}/api/management/v1/catalogs/{catalog_name}",
|
f"{POLARIS_API}/api/management/v1/catalogs/{catalog_name}", headers=headers
|
||||||
headers=headers
|
|
||||||
)
|
)
|
||||||
|
|
||||||
if cat_resp.status_code == 200:
|
if cat_resp.status_code == 200:
|
||||||
@@ -180,8 +182,7 @@ def _(mo):
|
|||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(POLARIS_API, headers, requests):
|
def _(POLARIS_API, headers, requests):
|
||||||
principals_resp = requests.get(
|
principals_resp = requests.get(
|
||||||
f"{POLARIS_API}/api/management/v1/principals",
|
f"{POLARIS_API}/api/management/v1/principals", headers=headers
|
||||||
headers=headers
|
|
||||||
)
|
)
|
||||||
|
|
||||||
if principals_resp.status_code == 200:
|
if principals_resp.status_code == 200:
|
||||||
@@ -207,8 +208,7 @@ def _(mo):
|
|||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(POLARIS_API, headers, requests):
|
def _(POLARIS_API, headers, requests):
|
||||||
roles_resp = requests.get(
|
roles_resp = requests.get(
|
||||||
f"{POLARIS_API}/api/management/v1/principal-roles",
|
f"{POLARIS_API}/api/management/v1/principal-roles", headers=headers
|
||||||
headers=headers
|
|
||||||
)
|
)
|
||||||
|
|
||||||
if roles_resp.status_code == 200:
|
if roles_resp.status_code == 200:
|
||||||
@@ -239,7 +239,7 @@ def _(POLARIS_API, catalog_name, headers, requests):
|
|||||||
config_resp = requests.get(
|
config_resp = requests.get(
|
||||||
f"{POLARIS_API}/api/catalog/v1/config",
|
f"{POLARIS_API}/api/catalog/v1/config",
|
||||||
headers=headers,
|
headers=headers,
|
||||||
params={"warehouse": catalog_name}
|
params={"warehouse": catalog_name},
|
||||||
)
|
)
|
||||||
|
|
||||||
if config_resp.status_code == 200:
|
if config_resp.status_code == 200:
|
||||||
@@ -266,8 +266,7 @@ def _(mo):
|
|||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(POLARIS_API, catalog_name, headers, requests):
|
def _(POLARIS_API, catalog_name, headers, requests):
|
||||||
ns_resp = requests.get(
|
ns_resp = requests.get(
|
||||||
f"{POLARIS_API}/api/catalog/v1/{catalog_name}/namespaces",
|
f"{POLARIS_API}/api/catalog/v1/{catalog_name}/namespaces", headers=headers
|
||||||
headers=headers
|
|
||||||
)
|
)
|
||||||
|
|
||||||
if ns_resp.status_code == 200:
|
if ns_resp.status_code == 200:
|
||||||
@@ -298,12 +297,12 @@ def _(POLARIS_API, catalog_name, headers, json, requests):
|
|||||||
create_ns_resp = requests.post(
|
create_ns_resp = requests.post(
|
||||||
f"{POLARIS_API}/api/catalog/v1/{catalog_name}/namespaces",
|
f"{POLARIS_API}/api/catalog/v1/{catalog_name}/namespaces",
|
||||||
headers={**headers, "Content-Type": "application/json"},
|
headers={**headers, "Content-Type": "application/json"},
|
||||||
data=json.dumps({
|
data=json.dumps(
|
||||||
"namespace": [ns_name],
|
{
|
||||||
"properties": {
|
"namespace": [ns_name],
|
||||||
"description": "Tutorial namespace"
|
"properties": {"description": "Tutorial namespace"},
|
||||||
}
|
}
|
||||||
})
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
if create_ns_resp.status_code == 200:
|
if create_ns_resp.status_code == 200:
|
||||||
@@ -329,8 +328,7 @@ def _(mo):
|
|||||||
def _(POLARIS_API, headers, requests):
|
def _(POLARIS_API, headers, requests):
|
||||||
def delete_catalog(name):
|
def delete_catalog(name):
|
||||||
resp = requests.delete(
|
resp = requests.delete(
|
||||||
f"{POLARIS_API}/api/management/v1/catalogs/{name}",
|
f"{POLARIS_API}/api/management/v1/catalogs/{name}", headers=headers
|
||||||
headers=headers
|
|
||||||
)
|
)
|
||||||
if resp.status_code == 204:
|
if resp.status_code == 204:
|
||||||
print(f"Deleted: {name}")
|
print(f"Deleted: {name}")
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
import marimo
|
import marimo
|
||||||
|
|
||||||
__generated_with = "0.19.9"
|
__generated_with = "0.21.1"
|
||||||
app = marimo.App()
|
app = marimo.App()
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -37,6 +37,7 @@ def _(mo):
|
|||||||
def _():
|
def _():
|
||||||
import altair as alt
|
import altair as alt
|
||||||
import polars as pl
|
import polars as pl
|
||||||
|
|
||||||
from conf import connect
|
from conf import connect
|
||||||
from pfs.rules import RULES
|
from pfs.rules import RULES
|
||||||
|
|
||||||
@@ -102,7 +103,11 @@ def _(SKIN_CODES, alt, mo, q):
|
|||||||
y=alt.Y("skin_rvu:Q", title="Skin Sub Total NF RVUs (8 codes)"),
|
y=alt.Y("skin_rvu:Q", title="Skin Sub Total NF RVUs (8 codes)"),
|
||||||
tooltip=["year", "skin_rvu", "total_rvu", "skin_pct", "skin_delta"],
|
tooltip=["year", "skin_rvu", "total_rvu", "skin_pct", "skin_delta"],
|
||||||
)
|
)
|
||||||
.properties(title="Skin Sub Application Codes — Total NF RVUs by Year", width=700, height=300)
|
.properties(
|
||||||
|
title="Skin Sub Application Codes — Total NF RVUs by Year",
|
||||||
|
width=700,
|
||||||
|
height=300,
|
||||||
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
mo.vstack([share_chart, pool_share])
|
mo.vstack([share_chart, pool_share])
|
||||||
@@ -153,7 +158,14 @@ def _(alt, q):
|
|||||||
x=alt.X("year:O", title="Year"),
|
x=alt.X("year:O", title="Year"),
|
||||||
y=alt.Y("non_fac_pe_rvu:Q", title="Non-Facility PE RVU"),
|
y=alt.Y("non_fac_pe_rvu:Q", title="Non-Facility PE RVU"),
|
||||||
color=alt.Color("label:N", title="Code"),
|
color=alt.Color("label:N", title="Code"),
|
||||||
tooltip=["year", "hcpcs", "label", "non_fac_pe_rvu", "work_rvu", "total_nf_rvu"],
|
tooltip=[
|
||||||
|
"year",
|
||||||
|
"hcpcs",
|
||||||
|
"label",
|
||||||
|
"non_fac_pe_rvu",
|
||||||
|
"work_rvu",
|
||||||
|
"total_nf_rvu",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
.properties(title="Practice Expense RVU Trajectory", width=700, height=350)
|
.properties(title="Practice Expense RVU Trajectory", width=700, height=350)
|
||||||
)
|
)
|
||||||
@@ -293,27 +305,42 @@ def _(SKIN_CODES, alt, mo, q):
|
|||||||
alt.Chart(category_impact.to_pandas())
|
alt.Chart(category_impact.to_pandas())
|
||||||
.mark_bar()
|
.mark_bar()
|
||||||
.encode(
|
.encode(
|
||||||
x=alt.X("implied_pe_loss:Q", title=f"Implied PE RVU Loss (from {delta_val:+.2f} skin sub PE delta)"),
|
x=alt.X(
|
||||||
|
"implied_pe_loss:Q",
|
||||||
|
title=f"Implied PE RVU Loss (from {delta_val:+.2f} skin sub PE delta)",
|
||||||
|
),
|
||||||
y=alt.Y("category:N", title="", sort="-x"),
|
y=alt.Y("category:N", title="", sort="-x"),
|
||||||
color=alt.Color("pe_share_pct:Q", title="PE Pool Share %",
|
color=alt.Color(
|
||||||
scale=alt.Scale(scheme="reds")),
|
"pe_share_pct:Q",
|
||||||
tooltip=["category", "codes", "category_pe", "pe_share_pct", "implied_pe_loss"],
|
title="PE Pool Share %",
|
||||||
|
scale=alt.Scale(scheme="reds"),
|
||||||
|
),
|
||||||
|
tooltip=[
|
||||||
|
"category",
|
||||||
|
"codes",
|
||||||
|
"category_pe",
|
||||||
|
"pe_share_pct",
|
||||||
|
"implied_pe_loss",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
.properties(
|
.properties(
|
||||||
title=f"Budget Neutrality Burden by Service Category (CY{delta_year})",
|
title=f"Budget Neutrality Burden by Service Category (CY{delta_year})",
|
||||||
width=700, height=350,
|
width=700,
|
||||||
|
height=350,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
mo.vstack([
|
mo.vstack(
|
||||||
mo.md(f"""
|
[
|
||||||
|
mo.md(f"""
|
||||||
**CY{delta_year}:** Skin sub application codes gained **{delta_val:+.2f} PE RVUs**.
|
**CY{delta_year}:** Skin sub application codes gained **{delta_val:+.2f} PE RVUs**.
|
||||||
Under budget neutrality, this is redistributed across ~{category_impact.select('codes').sum().item():,} other codes
|
Under budget neutrality, this is redistributed across ~{category_impact.select("codes").sum().item():,} other codes
|
||||||
proportional to their PE share.
|
proportional to their PE share.
|
||||||
"""),
|
"""),
|
||||||
impact_chart,
|
impact_chart,
|
||||||
category_impact,
|
category_impact,
|
||||||
])
|
]
|
||||||
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
||||||
@@ -339,10 +366,12 @@ def _(mo):
|
|||||||
|
|
||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(RULES, alt, pl, q):
|
def _(RULES, alt, pl, q):
|
||||||
cf_data = pl.DataFrame({
|
cf_data = pl.DataFrame(
|
||||||
"year": list(RULES.keys()),
|
{
|
||||||
"conversion_factor": [r.conversion_factor for r in RULES.values()],
|
"year": list(RULES.keys()),
|
||||||
})
|
"conversion_factor": [r.conversion_factor for r in RULES.values()],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
pool_growth = q("""
|
pool_growth = q("""
|
||||||
SELECT year, round(sum(work_rvu + non_fac_pe_rvu + mp_rvu), 0) as total_rvu
|
SELECT year, round(sum(work_rvu + non_fac_pe_rvu + mp_rvu), 0) as total_rvu
|
||||||
@@ -357,8 +386,11 @@ def _(RULES, alt, pl, q):
|
|||||||
.mark_line(point=True, color="#1f77b4")
|
.mark_line(point=True, color="#1f77b4")
|
||||||
.encode(
|
.encode(
|
||||||
x=alt.X("year:O", title="Year"),
|
x=alt.X("year:O", title="Year"),
|
||||||
y=alt.Y("conversion_factor:Q", title="Conversion Factor ($)",
|
y=alt.Y(
|
||||||
scale=alt.Scale(zero=False)),
|
"conversion_factor:Q",
|
||||||
|
title="Conversion Factor ($)",
|
||||||
|
scale=alt.Scale(zero=False),
|
||||||
|
),
|
||||||
tooltip=["year", "conversion_factor", "total_rvu"],
|
tooltip=["year", "conversion_factor", "total_rvu"],
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
@@ -368,15 +400,20 @@ def _(RULES, alt, pl, q):
|
|||||||
.mark_line(point=True, color="#d62728", strokeDash=[4, 4])
|
.mark_line(point=True, color="#d62728", strokeDash=[4, 4])
|
||||||
.encode(
|
.encode(
|
||||||
x=alt.X("year:O"),
|
x=alt.X("year:O"),
|
||||||
y=alt.Y("total_rvu:Q", title="Total Unweighted RVU Pool",
|
y=alt.Y(
|
||||||
scale=alt.Scale(zero=False)),
|
"total_rvu:Q",
|
||||||
|
title="Total Unweighted RVU Pool",
|
||||||
|
scale=alt.Scale(zero=False),
|
||||||
|
),
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
cf_chart = (
|
cf_chart = (
|
||||||
alt.layer(cf_line, rvu_line)
|
alt.layer(cf_line, rvu_line)
|
||||||
.resolve_scale(y="independent")
|
.resolve_scale(y="independent")
|
||||||
.properties(title="Conversion Factor vs. RVU Pool Growth", width=700, height=350)
|
.properties(
|
||||||
|
title="Conversion Factor vs. RVU Pool Growth", width=700, height=350
|
||||||
|
)
|
||||||
)
|
)
|
||||||
cf_chart
|
cf_chart
|
||||||
return
|
return
|
||||||
@@ -449,16 +486,18 @@ def _(RULES, SKIN_CODES, mo, q):
|
|||||||
ORDER BY dollar_loss
|
ORDER BY dollar_loss
|
||||||
""")
|
""")
|
||||||
|
|
||||||
mo.vstack([
|
mo.vstack(
|
||||||
mo.md(f"""
|
[
|
||||||
|
mo.md(f"""
|
||||||
**CY{latest_year}** parameters:
|
**CY{latest_year}** parameters:
|
||||||
- Skin sub PE delta: **{skin_delta:+.2f} RVUs**
|
- Skin sub PE delta: **{skin_delta:+.2f} RVUs**
|
||||||
- Other-code PE pool: **{total_pe:,.1f} RVUs**
|
- Other-code PE pool: **{total_pe:,.1f} RVUs**
|
||||||
- Implied tax rate: **{tax_rate*100:.4f}%** of each code's PE
|
- Implied tax rate: **{tax_rate * 100:.4f}%** of each code's PE
|
||||||
- CF: **${cf}**
|
- CF: **${cf}**
|
||||||
"""),
|
"""),
|
||||||
common_codes,
|
common_codes,
|
||||||
])
|
]
|
||||||
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
||||||
@@ -502,8 +541,10 @@ def _(SKIN_CODES, alt, q):
|
|||||||
.mark_area(opacity=0.3, color="#d62728")
|
.mark_area(opacity=0.3, color="#d62728")
|
||||||
.encode(
|
.encode(
|
||||||
x=alt.X("year:O", title="Year"),
|
x=alt.X("year:O", title="Year"),
|
||||||
y=alt.Y("cumulative_pe_growth:Q",
|
y=alt.Y(
|
||||||
title="Cumulative Skin Sub PE Growth (RVUs above 2015 baseline)"),
|
"cumulative_pe_growth:Q",
|
||||||
|
title="Cumulative Skin Sub PE Growth (RVUs above 2015 baseline)",
|
||||||
|
),
|
||||||
tooltip=["year", "skin_pe", "cumulative_pe_growth", "total_pe"],
|
tooltip=["year", "skin_pe", "cumulative_pe_growth", "total_pe"],
|
||||||
)
|
)
|
||||||
) + (
|
) + (
|
||||||
@@ -517,7 +558,8 @@ def _(SKIN_CODES, alt, q):
|
|||||||
|
|
||||||
cum_chart_final = cum_chart.properties(
|
cum_chart_final = cum_chart.properties(
|
||||||
title="Cumulative PE RVU Growth — Skin Sub Application Codes vs. 2015 Baseline",
|
title="Cumulative PE RVU Growth — Skin Sub Application Codes vs. 2015 Baseline",
|
||||||
width=700, height=300,
|
width=700,
|
||||||
|
height=300,
|
||||||
)
|
)
|
||||||
cum_chart_final
|
cum_chart_final
|
||||||
return
|
return
|
||||||
|
|||||||
@@ -38,6 +38,7 @@ def _(mo):
|
|||||||
def _():
|
def _():
|
||||||
import altair as alt
|
import altair as alt
|
||||||
import polars as pl
|
import polars as pl
|
||||||
|
|
||||||
from conf import connect
|
from conf import connect
|
||||||
from pfs.rules import RULES
|
from pfs.rules import RULES
|
||||||
|
|
||||||
@@ -50,9 +51,18 @@ def _():
|
|||||||
|
|
||||||
# Part B deductible history (published by CMS annually)
|
# Part B deductible history (published by CMS annually)
|
||||||
DEDUCTIBLES = {
|
DEDUCTIBLES = {
|
||||||
2015: 147.00, 2016: 166.00, 2017: 183.00, 2018: 183.00,
|
2015: 147.00,
|
||||||
2019: 185.00, 2020: 198.00, 2021: 203.00, 2022: 233.00,
|
2016: 166.00,
|
||||||
2023: 226.00, 2024: 240.00, 2025: 257.00, 2026: 257.00,
|
2017: 183.00,
|
||||||
|
2018: 183.00,
|
||||||
|
2019: 185.00,
|
||||||
|
2020: 198.00,
|
||||||
|
2021: 203.00,
|
||||||
|
2022: 233.00,
|
||||||
|
2023: 226.00,
|
||||||
|
2024: 240.00,
|
||||||
|
2025: 257.00,
|
||||||
|
2026: 257.00,
|
||||||
}
|
}
|
||||||
|
|
||||||
return DEDUCTIBLES, RULES, SKIN_CODES, alt, con, pl, q
|
return DEDUCTIBLES, RULES, SKIN_CODES, alt, con, pl, q
|
||||||
@@ -75,8 +85,11 @@ def _(mo):
|
|||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(alt, q):
|
def _(alt, q):
|
||||||
regions = [
|
regions = [
|
||||||
"MANHATTAN", "REST OF FLORIDA", "REST OF TEXAS",
|
"MANHATTAN",
|
||||||
"REST OF CALIFORNIA", "SOUTH CAROLINA",
|
"REST OF FLORIDA",
|
||||||
|
"REST OF TEXAS",
|
||||||
|
"REST OF CALIFORNIA",
|
||||||
|
"SOUTH CAROLINA",
|
||||||
]
|
]
|
||||||
region_list = ", ".join(f"'{r}'" for r in regions)
|
region_list = ", ".join(f"'{r}'" for r in regions)
|
||||||
|
|
||||||
@@ -100,12 +113,18 @@ def _(alt, q):
|
|||||||
x=alt.X("year:O", title="Year"),
|
x=alt.X("year:O", title="Year"),
|
||||||
y=alt.Y("bene_coinsurance:Q", title="Beneficiary Coinsurance ($)"),
|
y=alt.Y("bene_coinsurance:Q", title="Beneficiary Coinsurance ($)"),
|
||||||
color=alt.Color("locality_name:N", title="Region"),
|
color=alt.Color("locality_name:N", title="Region"),
|
||||||
tooltip=["year", "locality_name", "non_fac_fee",
|
tooltip=[
|
||||||
"bene_coinsurance", "non_fac_limiting_charge"],
|
"year",
|
||||||
|
"locality_name",
|
||||||
|
"non_fac_fee",
|
||||||
|
"bene_coinsurance",
|
||||||
|
"non_fac_limiting_charge",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
.properties(
|
.properties(
|
||||||
title="15271 Application — Beneficiary 20% Coinsurance by Region",
|
title="15271 Application — Beneficiary 20% Coinsurance by Region",
|
||||||
width=700, height=350,
|
width=700,
|
||||||
|
height=350,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
app_chart
|
app_chart
|
||||||
@@ -151,21 +170,27 @@ def _(alt, pl, q):
|
|||||||
alt.Chart(product_coins.to_pandas())
|
alt.Chart(product_coins.to_pandas())
|
||||||
.mark_line()
|
.mark_line()
|
||||||
.encode(
|
.encode(
|
||||||
x=alt.X("quarter:O", title="Quarter",
|
x=alt.X(
|
||||||
axis=alt.Axis(labelAngle=-45, labelFontSize=8)),
|
"quarter:O",
|
||||||
|
title="Quarter",
|
||||||
|
axis=alt.Axis(labelAngle=-45, labelFontSize=8),
|
||||||
|
),
|
||||||
y=alt.Y("bene_per_unit:Q", title="Beneficiary Coinsurance per cm² ($)"),
|
y=alt.Y("bene_per_unit:Q", title="Beneficiary Coinsurance per cm² ($)"),
|
||||||
color=alt.Color("short_description:N", title="Product"),
|
color=alt.Color("short_description:N", title="Product"),
|
||||||
tooltip=["quarter", "hcpcs_code", "short_description",
|
tooltip=[
|
||||||
"payment_limit", "bene_per_unit"],
|
"quarter",
|
||||||
|
"hcpcs_code",
|
||||||
|
"short_description",
|
||||||
|
"payment_limit",
|
||||||
|
"bene_per_unit",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
product_chart = (
|
product_chart = (product_lines + flat_coins).properties(
|
||||||
(product_lines + flat_coins)
|
title="Product Coinsurance per Unit (20% of ASP + 6%)",
|
||||||
.properties(
|
width=700,
|
||||||
title="Product Coinsurance per Unit (20% of ASP + 6%)",
|
height=400,
|
||||||
width=700, height=400,
|
|
||||||
)
|
|
||||||
)
|
)
|
||||||
product_chart
|
product_chart
|
||||||
return
|
return
|
||||||
@@ -234,21 +259,24 @@ def _(alt, pl, q):
|
|||||||
.mark_line(point=True)
|
.mark_line(point=True)
|
||||||
.encode(
|
.encode(
|
||||||
x=alt.X("year:O", title="Year"),
|
x=alt.X("year:O", title="Year"),
|
||||||
y=alt.Y("total_bene_cost:Q",
|
y=alt.Y("total_bene_cost:Q", title="Beneficiary Total Cost Sharing ($)"),
|
||||||
title="Beneficiary Total Cost Sharing ($)"),
|
|
||||||
color=alt.Color("short_description:N", title="Product"),
|
color=alt.Color("short_description:N", title="Product"),
|
||||||
tooltip=["year", "hcpcs_code", "short_description",
|
tooltip=[
|
||||||
"asp_per_unit", "app_coinsurance",
|
"year",
|
||||||
"product_coinsurance_25cm", "total_bene_cost"],
|
"hcpcs_code",
|
||||||
|
"short_description",
|
||||||
|
"asp_per_unit",
|
||||||
|
"app_coinsurance",
|
||||||
|
"product_coinsurance_25cm",
|
||||||
|
"total_bene_cost",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
episode_chart = (
|
episode_chart = (episode_lines + flat_line).properties(
|
||||||
(episode_lines + flat_line)
|
title="Total Beneficiary Cost Sharing — 25cm² Wound Episode",
|
||||||
.properties(
|
width=700,
|
||||||
title="Total Beneficiary Cost Sharing — 25cm² Wound Episode",
|
height=400,
|
||||||
width=700, height=400,
|
|
||||||
)
|
|
||||||
)
|
)
|
||||||
episode_chart
|
episode_chart
|
||||||
return
|
return
|
||||||
@@ -302,12 +330,19 @@ def _(alt, q):
|
|||||||
alt.value("#2ca02c"),
|
alt.value("#2ca02c"),
|
||||||
alt.value("#1f77b4"),
|
alt.value("#1f77b4"),
|
||||||
),
|
),
|
||||||
tooltip=["year", "products", "avg_copay", "min_copay",
|
tooltip=[
|
||||||
"max_copay", "copay_spread"],
|
"year",
|
||||||
|
"products",
|
||||||
|
"avg_copay",
|
||||||
|
"min_copay",
|
||||||
|
"max_copay",
|
||||||
|
"copay_spread",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
.properties(
|
.properties(
|
||||||
title="OPPS Minimum Unadjusted Copayment — Skin Substitutes",
|
title="OPPS Minimum Unadjusted Copayment — Skin Substitutes",
|
||||||
width=700, height=300,
|
width=700,
|
||||||
|
height=300,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -357,19 +392,28 @@ def _(alt, q):
|
|||||||
alt.Chart(bene_impact.to_pandas())
|
alt.Chart(bene_impact.to_pandas())
|
||||||
.mark_bar()
|
.mark_bar()
|
||||||
.encode(
|
.encode(
|
||||||
x=alt.X("bene_delta_per_unit:Q",
|
x=alt.X(
|
||||||
title="Change in Beneficiary Coinsurance per Unit ($)"),
|
"bene_delta_per_unit:Q",
|
||||||
y=alt.Y("short_description:N", title="", sort="x",
|
title="Change in Beneficiary Coinsurance per Unit ($)",
|
||||||
axis=alt.Axis(labelLimit=300)),
|
),
|
||||||
color=alt.Color("bene_impact:N",
|
y=alt.Y(
|
||||||
scale=alt.Scale(
|
"short_description:N", title="", sort="x", axis=alt.Axis(labelLimit=300)
|
||||||
domain=["bene saves", "bene pays more", "neutral"],
|
),
|
||||||
range=["#2ca02c", "#d62728", "#7f7f7f"],
|
color=alt.Color(
|
||||||
),
|
"bene_impact:N",
|
||||||
title="Impact"),
|
scale=alt.Scale(
|
||||||
tooltip=["hcpcs_code", "short_description",
|
domain=["bene saves", "bene pays more", "neutral"],
|
||||||
"current_bene_per_unit", "flat_bene_per_unit",
|
range=["#2ca02c", "#d62728", "#7f7f7f"],
|
||||||
"bene_delta_per_unit"],
|
),
|
||||||
|
title="Impact",
|
||||||
|
),
|
||||||
|
tooltip=[
|
||||||
|
"hcpcs_code",
|
||||||
|
"short_description",
|
||||||
|
"current_bene_per_unit",
|
||||||
|
"flat_bene_per_unit",
|
||||||
|
"bene_delta_per_unit",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
.properties(
|
.properties(
|
||||||
title="CY2026 Beneficiary Coinsurance Change per Unit",
|
title="CY2026 Beneficiary Coinsurance Change per Unit",
|
||||||
@@ -405,27 +449,40 @@ def _(pl, q):
|
|||||||
WHERE year = 2025 AND hcpcs = '15271'
|
WHERE year = 2025 AND hcpcs = '15271'
|
||||||
""").item()
|
""").item()
|
||||||
|
|
||||||
scenarios = pl.DataFrame({
|
scenarios = (
|
||||||
"scenario": [
|
pl.DataFrame(
|
||||||
"Small wound (10cm²) — Q4101 Apligraf",
|
{
|
||||||
"Medium wound (25cm²) — Q4186 EpiFix",
|
"scenario": [
|
||||||
"Large wound (50cm²) — Q4132 GrafixCore",
|
"Small wound (10cm²) — Q4101 Apligraf",
|
||||||
],
|
"Medium wound (25cm²) — Q4186 EpiFix",
|
||||||
"units": [10, 25, 50],
|
"Large wound (50cm²) — Q4132 GrafixCore",
|
||||||
"asp_per_unit": [30.23, 151.17, 106.70],
|
],
|
||||||
"product_name": ["Apligraf", "EpiFix", "GrafixCore"],
|
"units": [10, 25, 50],
|
||||||
}).with_columns(
|
"asp_per_unit": [30.23, 151.17, 106.70],
|
||||||
# Pre-2026: ASP + 6%
|
"product_name": ["Apligraf", "EpiFix", "GrafixCore"],
|
||||||
(pl.col("asp_per_unit") * pl.col("units") * 0.20).round(2).alias("pre_product_coins"),
|
}
|
||||||
pl.lit(avg_fee * 0.20).round(2).alias("pre_app_coins"),
|
)
|
||||||
# Post-2026: flat $127.28
|
.with_columns(
|
||||||
(pl.lit(127.28) * pl.col("units") * 0.20).round(2).alias("post_product_coins"),
|
# Pre-2026: ASP + 6%
|
||||||
pl.lit(avg_fee * 0.20).round(2).alias("post_app_coins"),
|
(pl.col("asp_per_unit") * pl.col("units") * 0.20)
|
||||||
).with_columns(
|
.round(2)
|
||||||
(pl.col("pre_product_coins") + pl.col("pre_app_coins")).alias("pre_total"),
|
.alias("pre_product_coins"),
|
||||||
(pl.col("post_product_coins") + pl.col("post_app_coins")).alias("post_total"),
|
pl.lit(avg_fee * 0.20).round(2).alias("pre_app_coins"),
|
||||||
).with_columns(
|
# Post-2026: flat $127.28
|
||||||
(pl.col("post_total") - pl.col("pre_total")).round(2).alias("delta"),
|
(pl.lit(127.28) * pl.col("units") * 0.20)
|
||||||
|
.round(2)
|
||||||
|
.alias("post_product_coins"),
|
||||||
|
pl.lit(avg_fee * 0.20).round(2).alias("post_app_coins"),
|
||||||
|
)
|
||||||
|
.with_columns(
|
||||||
|
(pl.col("pre_product_coins") + pl.col("pre_app_coins")).alias("pre_total"),
|
||||||
|
(pl.col("post_product_coins") + pl.col("post_app_coins")).alias(
|
||||||
|
"post_total"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
.with_columns(
|
||||||
|
(pl.col("post_total") - pl.col("pre_total")).round(2).alias("delta"),
|
||||||
|
)
|
||||||
)
|
)
|
||||||
scenarios
|
scenarios
|
||||||
return
|
return
|
||||||
@@ -449,10 +506,12 @@ def _(mo):
|
|||||||
|
|
||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(DEDUCTIBLES, alt, pl):
|
def _(DEDUCTIBLES, alt, pl):
|
||||||
deductible_df = pl.DataFrame({
|
deductible_df = pl.DataFrame(
|
||||||
"year": list(DEDUCTIBLES.keys()),
|
{
|
||||||
"deductible": list(DEDUCTIBLES.values()),
|
"year": list(DEDUCTIBLES.keys()),
|
||||||
})
|
"deductible": list(DEDUCTIBLES.values()),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
ded_chart = (
|
ded_chart = (
|
||||||
alt.Chart(deductible_df.to_pandas())
|
alt.Chart(deductible_df.to_pandas())
|
||||||
@@ -464,7 +523,8 @@ def _(DEDUCTIBLES, alt, pl):
|
|||||||
)
|
)
|
||||||
.properties(
|
.properties(
|
||||||
title="Medicare Part B Annual Deductible",
|
title="Medicare Part B Annual Deductible",
|
||||||
width=700, height=250,
|
width=700,
|
||||||
|
height=250,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
ded_chart
|
ded_chart
|
||||||
|
|||||||
@@ -35,6 +35,7 @@ def _(mo):
|
|||||||
def _():
|
def _():
|
||||||
import altair as alt
|
import altair as alt
|
||||||
import polars as pl
|
import polars as pl
|
||||||
|
|
||||||
from conf import connect
|
from conf import connect
|
||||||
|
|
||||||
con = connect.duckdb()
|
con = connect.duckdb()
|
||||||
@@ -87,8 +88,15 @@ def _(alt, q):
|
|||||||
x=alt.X("year:O", title="Year"),
|
x=alt.X("year:O", title="Year"),
|
||||||
y=alt.Y("non_fac_total:Q", title="Total Non-Facility RVUs"),
|
y=alt.Y("non_fac_total:Q", title="Total Non-Facility RVUs"),
|
||||||
color=alt.Color("description:N", title="Code"),
|
color=alt.Color("description:N", title="Code"),
|
||||||
tooltip=["year", "hcpcs", "description",
|
tooltip=[
|
||||||
"work_rvu", "non_fac_pe_rvu", "mp_rvu", "non_fac_total"],
|
"year",
|
||||||
|
"hcpcs",
|
||||||
|
"description",
|
||||||
|
"work_rvu",
|
||||||
|
"non_fac_pe_rvu",
|
||||||
|
"mp_rvu",
|
||||||
|
"non_fac_total",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
.properties(title="Application Code RVUs (Non-Facility)", width=700, height=400)
|
.properties(title="Application Code RVUs (Non-Facility)", width=700, height=400)
|
||||||
)
|
)
|
||||||
@@ -153,12 +161,19 @@ def _(alt, q, region_picker):
|
|||||||
x=alt.X("year:O", title="Year"),
|
x=alt.X("year:O", title="Year"),
|
||||||
y=alt.Y("non_fac_fee:Q", title="Non-Facility Fee ($)"),
|
y=alt.Y("non_fac_fee:Q", title="Non-Facility Fee ($)"),
|
||||||
color=alt.Color("description:N", title="Code"),
|
color=alt.Color("description:N", title="Code"),
|
||||||
tooltip=["year", "hcpcs", "description", "non_fac_fee",
|
tooltip=[
|
||||||
"fac_fee", "locality_name"],
|
"year",
|
||||||
|
"hcpcs",
|
||||||
|
"description",
|
||||||
|
"non_fac_fee",
|
||||||
|
"fac_fee",
|
||||||
|
"locality_name",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
.properties(
|
.properties(
|
||||||
title=f"Application Code Fees — {_region}",
|
title=f"Application Code Fees — {_region}",
|
||||||
width=700, height=400,
|
width=700,
|
||||||
|
height=400,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
carrier_chart
|
carrier_chart
|
||||||
@@ -179,9 +194,16 @@ def _(mo):
|
|||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(alt, q):
|
def _(alt, q):
|
||||||
regions = [
|
regions = [
|
||||||
"MANHATTAN", "ALASKA*", "REST OF FLORIDA", "REST OF TEXAS",
|
"MANHATTAN",
|
||||||
"REST OF CALIFORNIA", "CHICAGO", "DETROIT", "ATLANTA",
|
"ALASKA*",
|
||||||
"HAWAII, GUAM", "SOUTH CAROLINA",
|
"REST OF FLORIDA",
|
||||||
|
"REST OF TEXAS",
|
||||||
|
"REST OF CALIFORNIA",
|
||||||
|
"CHICAGO",
|
||||||
|
"DETROIT",
|
||||||
|
"ATLANTA",
|
||||||
|
"HAWAII, GUAM",
|
||||||
|
"SOUTH CAROLINA",
|
||||||
]
|
]
|
||||||
region_list = ", ".join(f"'{r}'" for r in regions)
|
region_list = ", ".join(f"'{r}'" for r in regions)
|
||||||
|
|
||||||
@@ -206,7 +228,8 @@ def _(alt, q):
|
|||||||
)
|
)
|
||||||
.properties(
|
.properties(
|
||||||
title="15271 Non-Facility Fee — Regional Comparison",
|
title="15271 Non-Facility Fee — Regional Comparison",
|
||||||
width=700, height=400,
|
width=700,
|
||||||
|
height=400,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
regional_chart
|
regional_chart
|
||||||
@@ -259,22 +282,31 @@ def _(alt, pl, q):
|
|||||||
alt.Chart(asp_ts.to_pandas())
|
alt.Chart(asp_ts.to_pandas())
|
||||||
.mark_line()
|
.mark_line()
|
||||||
.encode(
|
.encode(
|
||||||
x=alt.X("quarter:O", title="Quarter",
|
x=alt.X(
|
||||||
axis=alt.Axis(labelAngle=-45, labelFontSize=8)),
|
"quarter:O",
|
||||||
y=alt.Y("payment_limit:Q", title="Payment Limit (ASP + 6%) $",
|
title="Quarter",
|
||||||
scale=alt.Scale(domainMax=800)),
|
axis=alt.Axis(labelAngle=-45, labelFontSize=8),
|
||||||
|
),
|
||||||
|
y=alt.Y(
|
||||||
|
"payment_limit:Q",
|
||||||
|
title="Payment Limit (ASP + 6%) $",
|
||||||
|
scale=alt.Scale(domainMax=800),
|
||||||
|
),
|
||||||
color=alt.Color("short_description:N", title="Product"),
|
color=alt.Color("short_description:N", title="Product"),
|
||||||
tooltip=["quarter", "hcpcs_code", "short_description",
|
tooltip=[
|
||||||
"payment_limit", "asp_per_unit"],
|
"quarter",
|
||||||
|
"hcpcs_code",
|
||||||
|
"short_description",
|
||||||
|
"payment_limit",
|
||||||
|
"asp_per_unit",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
asp_chart = (
|
asp_chart = (asp_lines + flat_rate_rule).properties(
|
||||||
(asp_lines + flat_rate_rule)
|
title="ASP Quarterly Payment Limits — Top 15 Products",
|
||||||
.properties(
|
width=700,
|
||||||
title="ASP Quarterly Payment Limits — Top 15 Products",
|
height=450,
|
||||||
width=700, height=450,
|
|
||||||
)
|
|
||||||
)
|
)
|
||||||
asp_chart
|
asp_chart
|
||||||
return
|
return
|
||||||
@@ -356,11 +388,9 @@ def _(alt, pl, q, region_picker):
|
|||||||
""")
|
""")
|
||||||
|
|
||||||
# Join to get total episode cost
|
# Join to get total episode cost
|
||||||
combined = (
|
combined = episode_data.join(app_fee, on="year", how="left").with_columns(
|
||||||
episode_data.join(app_fee, on="year", how="left")
|
(pl.col("product_cost_25cm2") + pl.col("application_fee").fill_null(0)).alias(
|
||||||
.with_columns(
|
"total_episode_cost"
|
||||||
(pl.col("product_cost_25cm2") + pl.col("application_fee").fill_null(0))
|
|
||||||
.alias("total_episode_cost")
|
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -377,18 +407,21 @@ def _(alt, pl, q, region_picker):
|
|||||||
x=alt.X("year:O", title="Year"),
|
x=alt.X("year:O", title="Year"),
|
||||||
y=alt.Y("total_episode_cost:Q", title="Total Episode Cost ($)"),
|
y=alt.Y("total_episode_cost:Q", title="Total Episode Cost ($)"),
|
||||||
color=alt.Color("short_description:N", title="Product"),
|
color=alt.Color("short_description:N", title="Product"),
|
||||||
tooltip=["year", "hcpcs_code", "short_description",
|
tooltip=[
|
||||||
"product_cost_25cm2", "application_fee",
|
"year",
|
||||||
"total_episode_cost"],
|
"hcpcs_code",
|
||||||
|
"short_description",
|
||||||
|
"product_cost_25cm2",
|
||||||
|
"application_fee",
|
||||||
|
"total_episode_cost",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
episode_chart = (
|
episode_chart = (episode_lines + flat_line).properties(
|
||||||
(episode_lines + flat_line)
|
title=f"Total Episode Cost (25cm² wound) — {_region}",
|
||||||
.properties(
|
width=700,
|
||||||
title=f"Total Episode Cost (25cm² wound) — {_region}",
|
height=400,
|
||||||
width=700, height=400,
|
|
||||||
)
|
|
||||||
)
|
)
|
||||||
episode_chart
|
episode_chart
|
||||||
return
|
return
|
||||||
@@ -428,16 +461,27 @@ def _(alt, q):
|
|||||||
.mark_bar()
|
.mark_bar()
|
||||||
.encode(
|
.encode(
|
||||||
x=alt.X("delta:Q", title="Payment Limit − $127.28 Flat Rate"),
|
x=alt.X("delta:Q", title="Payment Limit − $127.28 Flat Rate"),
|
||||||
y=alt.Y("short_description:N", title="", sort="-x",
|
y=alt.Y(
|
||||||
axis=alt.Axis(labelLimit=300)),
|
"short_description:N",
|
||||||
color=alt.Color("impact:N",
|
title="",
|
||||||
scale=alt.Scale(
|
sort="-x",
|
||||||
domain=["loses", "gains", "neutral"],
|
axis=alt.Axis(labelLimit=300),
|
||||||
range=["#d62728", "#2ca02c", "#7f7f7f"],
|
),
|
||||||
),
|
color=alt.Color(
|
||||||
title="Impact"),
|
"impact:N",
|
||||||
tooltip=["hcpcs_code", "short_description",
|
scale=alt.Scale(
|
||||||
"payment_limit", "flat_rate", "delta"],
|
domain=["loses", "gains", "neutral"],
|
||||||
|
range=["#d62728", "#2ca02c", "#7f7f7f"],
|
||||||
|
),
|
||||||
|
title="Impact",
|
||||||
|
),
|
||||||
|
tooltip=[
|
||||||
|
"hcpcs_code",
|
||||||
|
"short_description",
|
||||||
|
"payment_limit",
|
||||||
|
"flat_rate",
|
||||||
|
"delta",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
.properties(title="CY2026 Flat-Rate Impact by Product", width=700)
|
.properties(title="CY2026 Flat-Rate Impact by Product", width=700)
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -31,6 +31,7 @@ def _(mo):
|
|||||||
def _():
|
def _():
|
||||||
import altair as alt
|
import altair as alt
|
||||||
import polars as pl
|
import polars as pl
|
||||||
|
|
||||||
from conf import connect
|
from conf import connect
|
||||||
|
|
||||||
con = connect.duckdb()
|
con = connect.duckdb()
|
||||||
@@ -85,7 +86,9 @@ def _(alt, asp, mo, pl, product_select):
|
|||||||
filtered.select(["hcpcs_code", "product_name"])
|
filtered.select(["hcpcs_code", "product_name"])
|
||||||
.unique()
|
.unique()
|
||||||
.with_columns(
|
.with_columns(
|
||||||
(pl.col("hcpcs_code") + " " + pl.col("product_name").fill_null("")).alias("label")
|
(pl.col("hcpcs_code") + " " + pl.col("product_name").fill_null("")).alias(
|
||||||
|
"label"
|
||||||
|
)
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
chart_data = filtered.join(labels, on="hcpcs_code").to_pandas()
|
chart_data = filtered.join(labels, on="hcpcs_code").to_pandas()
|
||||||
@@ -97,8 +100,15 @@ def _(alt, asp, mo, pl, product_select):
|
|||||||
x=alt.X("quarter:N", title="Quarter", axis=alt.Axis(labelAngle=-45)),
|
x=alt.X("quarter:N", title="Quarter", axis=alt.Axis(labelAngle=-45)),
|
||||||
y=alt.Y("payment_limit:Q", title="Payment Limit ($/cm²)"),
|
y=alt.Y("payment_limit:Q", title="Payment Limit ($/cm²)"),
|
||||||
color=alt.Color("label:N", title="Product"),
|
color=alt.Color("label:N", title="Product"),
|
||||||
tooltip=["quarter", "hcpcs_code", "product_name", "manufacturer",
|
tooltip=[
|
||||||
"asp_per_unit", "payment_limit", "qoq_pct_change"],
|
"quarter",
|
||||||
|
"hcpcs_code",
|
||||||
|
"product_name",
|
||||||
|
"manufacturer",
|
||||||
|
"asp_per_unit",
|
||||||
|
"payment_limit",
|
||||||
|
"qoq_pct_change",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
.properties(width=700, height=400, title="ASP Payment Limit Over Time")
|
.properties(width=700, height=400, title="ASP Payment Limit Over Time")
|
||||||
)
|
)
|
||||||
@@ -140,9 +150,17 @@ def _(alt, con, mo, pl):
|
|||||||
.encode(
|
.encode(
|
||||||
x=alt.X("total_revenue:Q", title="Total Revenue ($)"),
|
x=alt.X("total_revenue:Q", title="Total Revenue ($)"),
|
||||||
y=alt.Y("manufacturer:N", sort="-x", title=None),
|
y=alt.Y("manufacturer:N", sort="-x", title=None),
|
||||||
color=alt.Color("avg_asp:Q", scale=alt.Scale(scheme="reds"), title="Avg ASP ($/cm²)"),
|
color=alt.Color(
|
||||||
tooltip=["manufacturer", "product_count", "active_products",
|
"avg_asp:Q", scale=alt.Scale(scheme="reds"), title="Avg ASP ($/cm²)"
|
||||||
"total_revenue", "avg_asp", "categories"],
|
),
|
||||||
|
tooltip=[
|
||||||
|
"manufacturer",
|
||||||
|
"product_count",
|
||||||
|
"active_products",
|
||||||
|
"total_revenue",
|
||||||
|
"avg_asp",
|
||||||
|
"categories",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
.properties(width=600, height=350, title="Top Manufacturers by Revenue")
|
.properties(width=600, height=350, title="Top Manufacturers by Revenue")
|
||||||
)
|
)
|
||||||
@@ -165,7 +183,9 @@ def _(alt, con, mo, pl):
|
|||||||
.encode(
|
.encode(
|
||||||
x=alt.X("products:Q", title="Product Count"),
|
x=alt.X("products:Q", title="Product Count"),
|
||||||
y=alt.Y("category:N", sort="-x", title=None),
|
y=alt.Y("category:N", sort="-x", title=None),
|
||||||
color=alt.Color("avg_asp:Q", scale=alt.Scale(scheme="blues"), title="Avg ASP"),
|
color=alt.Color(
|
||||||
|
"avg_asp:Q", scale=alt.Scale(scheme="blues"), title="Avg ASP"
|
||||||
|
),
|
||||||
tooltip=["category", "products", "avg_asp", "claims"],
|
tooltip=["category", "products", "avg_asp", "claims"],
|
||||||
)
|
)
|
||||||
.properties(width=600, height=300, title="Products by Category")
|
.properties(width=600, height=300, title="Products by Category")
|
||||||
@@ -197,14 +217,23 @@ def _(alt, con, mo, pl):
|
|||||||
x=alt.X("paid_per_bene:Q", title="Paid per Beneficiary ($)"),
|
x=alt.X("paid_per_bene:Q", title="Paid per Beneficiary ($)"),
|
||||||
y=alt.Y("state:N", sort="-x", title=None),
|
y=alt.Y("state:N", sort="-x", title=None),
|
||||||
color=alt.Color("mac_jurisdiction:N", title="MAC Jurisdiction"),
|
color=alt.Color("mac_jurisdiction:N", title="MAC Jurisdiction"),
|
||||||
tooltip=["state", "mac_jurisdiction", "unique_benes", "total_paid",
|
tooltip=[
|
||||||
"paid_per_bene", "pct_office", "pct_podiatry"],
|
"state",
|
||||||
|
"mac_jurisdiction",
|
||||||
|
"unique_benes",
|
||||||
|
"total_paid",
|
||||||
|
"paid_per_bene",
|
||||||
|
"pct_office",
|
||||||
|
"pct_podiatry",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
.properties(width=600, height=400, title="Spend per Beneficiary by State")
|
.properties(width=600, height=400, title="Spend per Beneficiary by State")
|
||||||
)
|
)
|
||||||
|
|
||||||
mac = pl.from_pandas(
|
mac = pl.from_pandas(
|
||||||
con.execute("SELECT * FROM skin_subs.mac_summary ORDER BY total_paid DESC").fetchdf()
|
con.execute(
|
||||||
|
"SELECT * FROM skin_subs.mac_summary ORDER BY total_paid DESC"
|
||||||
|
).fetchdf()
|
||||||
)
|
)
|
||||||
|
|
||||||
mac_chart = (
|
mac_chart = (
|
||||||
@@ -213,10 +242,20 @@ def _(alt, con, mo, pl):
|
|||||||
.encode(
|
.encode(
|
||||||
x=alt.X("total_paid:Q", title="Total Paid ($)"),
|
x=alt.X("total_paid:Q", title="Total Paid ($)"),
|
||||||
y=alt.Y("mac_jurisdiction:N", sort="-x", title=None),
|
y=alt.Y("mac_jurisdiction:N", sort="-x", title=None),
|
||||||
color=alt.Color("avg_pct_podiatry:Q", scale=alt.Scale(scheme="oranges"),
|
color=alt.Color(
|
||||||
title="% Podiatry"),
|
"avg_pct_podiatry:Q",
|
||||||
tooltip=["mac_jurisdiction", "states", "claim_lines", "total_paid",
|
scale=alt.Scale(scheme="oranges"),
|
||||||
"avg_paid_per_bene", "avg_pct_office", "avg_pct_podiatry"],
|
title="% Podiatry",
|
||||||
|
),
|
||||||
|
tooltip=[
|
||||||
|
"mac_jurisdiction",
|
||||||
|
"states",
|
||||||
|
"claim_lines",
|
||||||
|
"total_paid",
|
||||||
|
"avg_paid_per_bene",
|
||||||
|
"avg_pct_office",
|
||||||
|
"avg_pct_podiatry",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
.properties(width=600, height=250, title="Total Paid by MAC Jurisdiction")
|
.properties(width=600, height=250, title="Total Paid by MAC Jurisdiction")
|
||||||
)
|
)
|
||||||
@@ -278,16 +317,27 @@ def _(alt, con, mo, pl):
|
|||||||
.encode(
|
.encode(
|
||||||
x=alt.X("volume_zscore:Q", title="Volume Z-Score (vs specialty peers)"),
|
x=alt.X("volume_zscore:Q", title="Volume Z-Score (vs specialty peers)"),
|
||||||
y=alt.Y("intensity_zscore:Q", title="Intensity Z-Score ($/patient)"),
|
y=alt.Y("intensity_zscore:Q", title="Intensity Z-Score ($/patient)"),
|
||||||
color=alt.Color("risk_tier:N",
|
color=alt.Color(
|
||||||
scale=alt.Scale(
|
"risk_tier:N",
|
||||||
domain=["critical", "high", "moderate", "low"],
|
scale=alt.Scale(
|
||||||
range=["#d62728", "#ff7f0e", "#bcbd22", "#2ca02c"]),
|
domain=["critical", "high", "moderate", "low"],
|
||||||
title="Risk Tier"),
|
range=["#d62728", "#ff7f0e", "#bcbd22", "#2ca02c"],
|
||||||
|
),
|
||||||
|
title="Risk Tier",
|
||||||
|
),
|
||||||
size=alt.Size("total_paid:Q", title="Total Paid"),
|
size=alt.Size("total_paid:Q", title="Total Paid"),
|
||||||
shape=alt.Shape("primary_setting:N", title="Setting"),
|
shape=alt.Shape("primary_setting:N", title="Setting"),
|
||||||
tooltip=["rendering_npi", "provider_specialty", "state",
|
tooltip=[
|
||||||
"primary_setting", "total_paid", "paid_per_patient",
|
"rendering_npi",
|
||||||
"composite_risk", "risk_tier", "fraud_pattern_match"],
|
"provider_specialty",
|
||||||
|
"state",
|
||||||
|
"primary_setting",
|
||||||
|
"total_paid",
|
||||||
|
"paid_per_patient",
|
||||||
|
"composite_risk",
|
||||||
|
"risk_tier",
|
||||||
|
"fraud_pattern_match",
|
||||||
|
],
|
||||||
)
|
)
|
||||||
.properties(width=650, height=450, title="Provider Risk: Volume vs Intensity")
|
.properties(width=650, height=450, title="Provider Risk: Volume vs Intensity")
|
||||||
)
|
)
|
||||||
@@ -300,7 +350,9 @@ def _(alt, con, mo, pl):
|
|||||||
|
|
||||||
tier_table = mo.ui.table(tier.to_pandas())
|
tier_table = mo.ui.table(tier.to_pandas())
|
||||||
|
|
||||||
mo.vstack([mo.ui.altair_chart(scatter), mo.md("### Risk Tier Distribution"), tier_table])
|
mo.vstack(
|
||||||
|
[mo.ui.altair_chart(scatter), mo.md("### Risk Tier Distribution"), tier_table]
|
||||||
|
)
|
||||||
return risk, scatter, tier, tier_table
|
return risk, scatter, tier, tier_table
|
||||||
|
|
||||||
|
|
||||||
@@ -329,13 +381,15 @@ def _(con, mo, pl):
|
|||||||
""").fetchdf()
|
""").fetchdf()
|
||||||
)
|
)
|
||||||
|
|
||||||
mo.vstack([
|
mo.vstack(
|
||||||
mo.md("### Evidence Base by Source"),
|
[
|
||||||
mo.ui.table(ev.to_pandas()),
|
mo.md("### Evidence Base by Source"),
|
||||||
mo.md("### Study Characteristics by Type"),
|
mo.ui.table(ev.to_pandas()),
|
||||||
mo.ui.table(study.to_pandas()),
|
mo.md("### Study Characteristics by Type"),
|
||||||
mo.md(f"**Total evidence items:** {ev['items'].sum():,}"),
|
mo.ui.table(study.to_pandas()),
|
||||||
])
|
mo.md(f"**Total evidence items:** {ev['items'].sum():,}"),
|
||||||
|
]
|
||||||
|
)
|
||||||
return ev, study
|
return ev, study
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ app = marimo.App(width="medium")
|
|||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _():
|
def _():
|
||||||
import marimo as mo
|
import marimo as mo
|
||||||
|
|
||||||
mo.md("""
|
mo.md("""
|
||||||
# Zotero Library Explorer
|
# Zotero Library Explorer
|
||||||
|
|
||||||
@@ -26,9 +27,12 @@ def _():
|
|||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(mo):
|
def _(mo):
|
||||||
import sqlite3
|
import sqlite3
|
||||||
|
|
||||||
import polars as pl
|
import polars as pl
|
||||||
from pyzotero import zotero
|
from pyzotero import zotero
|
||||||
from conf import connect, path as _conf_path
|
|
||||||
|
from conf import connect
|
||||||
|
from conf import path as _conf_path
|
||||||
|
|
||||||
ZOTERO_DB = _conf_path("db.zotero")
|
ZOTERO_DB = _conf_path("db.zotero")
|
||||||
|
|
||||||
@@ -47,9 +51,12 @@ def _(connect, sqlite3):
|
|||||||
db.row_factory = sqlite3.Row
|
db.row_factory = sqlite3.Row
|
||||||
|
|
||||||
# Quick health check
|
# Quick health check
|
||||||
tables = [r[0] for r in db.execute(
|
tables = [
|
||||||
"SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"
|
r[0]
|
||||||
).fetchall()]
|
for r in db.execute(
|
||||||
|
"SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"
|
||||||
|
).fetchall()
|
||||||
|
]
|
||||||
print(f"Connected — {len(tables)} tables found")
|
print(f"Connected — {len(tables)} tables found")
|
||||||
print(f"Tables: {', '.join(tables[:15])}{'...' if len(tables) > 15 else ''}")
|
print(f"Tables: {', '.join(tables[:15])}{'...' if len(tables) > 15 else ''}")
|
||||||
return (db,)
|
return (db,)
|
||||||
@@ -114,17 +121,25 @@ def _(db, mo, pl):
|
|||||||
|
|
||||||
def _build_tree(df, parent_id=None, depth=0):
|
def _build_tree(df, parent_id=None, depth=0):
|
||||||
_rows = df.filter(
|
_rows = df.filter(
|
||||||
pl.col("parentCollectionID") == parent_id if parent_id else pl.col("parentCollectionID").is_null()
|
pl.col("parentCollectionID") == parent_id
|
||||||
|
if parent_id
|
||||||
|
else pl.col("parentCollectionID").is_null()
|
||||||
)
|
)
|
||||||
_lines = []
|
_lines = []
|
||||||
for _row in _rows.iter_rows(named=True):
|
for _row in _rows.iter_rows(named=True):
|
||||||
_indent = " " * depth
|
_indent = " " * depth
|
||||||
_lines.append(f"{_indent}- **{_row['collectionName']}** ({_row['itemCount']} items)")
|
_lines.append(
|
||||||
|
f"{_indent}- **{_row['collectionName']}** ({_row['itemCount']} items)"
|
||||||
|
)
|
||||||
_lines.extend(_build_tree(df, _row["collectionID"], depth + 1))
|
_lines.extend(_build_tree(df, _row["collectionID"], depth + 1))
|
||||||
return _lines
|
return _lines
|
||||||
|
|
||||||
_tree = _build_tree(collections_df) if collections_df.height > 0 else []
|
_tree = _build_tree(collections_df) if collections_df.height > 0 else []
|
||||||
mo.md("\n".join(_tree) if _tree else "No collections found. Add some in the Zotero desktop app.")
|
mo.md(
|
||||||
|
"\n".join(_tree)
|
||||||
|
if _tree
|
||||||
|
else "No collections found. Add some in the Zotero desktop app."
|
||||||
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
||||||
@@ -155,7 +170,9 @@ def _(db, mo, pl):
|
|||||||
connection=db,
|
connection=db,
|
||||||
)
|
)
|
||||||
|
|
||||||
mo.ui.table(item_types_df, label="Item Types") if item_types_df.height > 0 else mo.md("No library items found yet.")
|
mo.ui.table(
|
||||||
|
item_types_df, label="Item Types"
|
||||||
|
) if item_types_df.height > 0 else mo.md("No library items found yet.")
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
||||||
@@ -195,7 +212,9 @@ def _(db, mo, pl):
|
|||||||
connection=db,
|
connection=db,
|
||||||
)
|
)
|
||||||
|
|
||||||
mo.ui.table(recent_df, label="Recent Items") if recent_df.height > 0 else mo.md("No items found. Add references through the Zotero desktop app (VNC).")
|
mo.ui.table(recent_df, label="Recent Items") if recent_df.height > 0 else mo.md(
|
||||||
|
"No items found. Add references through the Zotero desktop app (VNC)."
|
||||||
|
)
|
||||||
return (recent_df,)
|
return (recent_df,)
|
||||||
|
|
||||||
|
|
||||||
@@ -211,7 +230,9 @@ def _(mo):
|
|||||||
|
|
||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(mo):
|
def _(mo):
|
||||||
search_input = mo.ui.text(placeholder="Enter search terms...", label="Search", full_width=True)
|
search_input = mo.ui.text(
|
||||||
|
placeholder="Enter search terms...", label="Search", full_width=True
|
||||||
|
)
|
||||||
search_input
|
search_input
|
||||||
return (search_input,)
|
return (search_input,)
|
||||||
|
|
||||||
@@ -257,7 +278,9 @@ def _(db, mo, pl, search_input):
|
|||||||
execute_options={"parameters": {"q": _query}},
|
execute_options={"parameters": {"q": _query}},
|
||||||
)
|
)
|
||||||
|
|
||||||
mo.ui.table(_search_results, label=f"Results for '{_query}'") if _search_results.height > 0 else mo.md(f"No results for **{_query}**")
|
mo.ui.table(
|
||||||
|
_search_results, label=f"Results for '{_query}'"
|
||||||
|
) if _search_results.height > 0 else mo.md(f"No results for **{_query}**")
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
||||||
@@ -351,7 +374,9 @@ def _(db, item_id_input, mo):
|
|||||||
if _creators:
|
if _creators:
|
||||||
_sections.append("\n**Creators**\n")
|
_sections.append("\n**Creators**\n")
|
||||||
for _c in _creators:
|
for _c in _creators:
|
||||||
_sections.append(f"- {_c['firstName']} {_c['lastName']} ({_c['creatorType']})")
|
_sections.append(
|
||||||
|
f"- {_c['firstName']} {_c['lastName']} ({_c['creatorType']})"
|
||||||
|
)
|
||||||
|
|
||||||
if _tags:
|
if _tags:
|
||||||
_sections.append("\n**Tags**\n")
|
_sections.append("\n**Tags**\n")
|
||||||
@@ -392,7 +417,9 @@ def _(db, mo, pl):
|
|||||||
connection=db,
|
connection=db,
|
||||||
)
|
)
|
||||||
|
|
||||||
mo.ui.table(tags_df, label="Tags") if tags_df.height > 0 else mo.md("No tags found.")
|
mo.ui.table(tags_df, label="Tags") if tags_df.height > 0 else mo.md(
|
||||||
|
"No tags found."
|
||||||
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
||||||
@@ -426,7 +453,9 @@ def _(db, mo, pl):
|
|||||||
connection=db,
|
connection=db,
|
||||||
)
|
)
|
||||||
|
|
||||||
mo.ui.table(creators_df, label="Top Creators") if creators_df.height > 0 else mo.md("No creators found.")
|
mo.ui.table(creators_df, label="Top Creators") if creators_df.height > 0 else mo.md(
|
||||||
|
"No creators found."
|
||||||
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
||||||
@@ -501,9 +530,15 @@ def _(mo):
|
|||||||
|
|
||||||
@app.cell(hide_code=True)
|
@app.cell(hide_code=True)
|
||||||
def _(mo):
|
def _(mo):
|
||||||
api_key_input = mo.ui.text(placeholder="Zotero API key", label="API Key", kind="password", full_width=True)
|
api_key_input = mo.ui.text(
|
||||||
library_id_input = mo.ui.text(placeholder="Library ID (numeric)", label="Library ID", full_width=True)
|
placeholder="Zotero API key", label="API Key", kind="password", full_width=True
|
||||||
lib_type_input = mo.ui.dropdown(options=["user", "group"], value="user", label="Library Type")
|
)
|
||||||
|
library_id_input = mo.ui.text(
|
||||||
|
placeholder="Library ID (numeric)", label="Library ID", full_width=True
|
||||||
|
)
|
||||||
|
lib_type_input = mo.ui.dropdown(
|
||||||
|
options=["user", "group"], value="user", label="Library Type"
|
||||||
|
)
|
||||||
mo.hstack([library_id_input, lib_type_input, api_key_input], widths=[1, 1, 2])
|
mo.hstack([library_id_input, lib_type_input, api_key_input], widths=[1, 1, 2])
|
||||||
return api_key_input, lib_type_input, library_id_input
|
return api_key_input, lib_type_input, library_id_input
|
||||||
|
|
||||||
@@ -515,7 +550,9 @@ def _(api_key_input, lib_type_input, library_id_input, mo, zotero):
|
|||||||
lib_type = lib_type_input.value
|
lib_type = lib_type_input.value
|
||||||
|
|
||||||
if not api_key or not lib_id:
|
if not api_key or not lib_id:
|
||||||
mo.md("Enter your Zotero API key and library ID above to enable Web API access.")
|
mo.md(
|
||||||
|
"Enter your Zotero API key and library ID above to enable Web API access."
|
||||||
|
)
|
||||||
mo.stop(True)
|
mo.stop(True)
|
||||||
|
|
||||||
zot = zotero.Zotero(lib_id, lib_type, api_key)
|
zot = zotero.Zotero(lib_id, lib_type, api_key)
|
||||||
@@ -534,9 +571,9 @@ def _(api_key_input, lib_type_input, library_id_input, mo, zotero):
|
|||||||
mo.md(f"""
|
mo.md(f"""
|
||||||
Connected to Zotero Web API
|
Connected to Zotero Web API
|
||||||
|
|
||||||
- **User**: {_key_info.get('userID', 'N/A')}
|
- **User**: {_key_info.get("userID", "N/A")}
|
||||||
- **Key name**: {_key_info.get('key', 'N/A')[:8]}...
|
- **Key name**: {_key_info.get("key", "N/A")[:8]}...
|
||||||
- **Access**: {'read/write' if _key_info.get('access', {}).get('user', {}).get('library') else 'read-only'}
|
- **Access**: {"read/write" if _key_info.get("access", {}).get("user", {}).get("library") else "read-only"}
|
||||||
""")
|
""")
|
||||||
return (zot,)
|
return (zot,)
|
||||||
|
|
||||||
@@ -552,7 +589,8 @@ def _(mo, zot):
|
|||||||
_title = _data.get("title", "(untitled)")
|
_title = _data.get("title", "(untitled)")
|
||||||
_item_type = _data.get("itemType", "?")
|
_item_type = _data.get("itemType", "?")
|
||||||
_authors = ", ".join(
|
_authors = ", ".join(
|
||||||
_cr.get("lastName", _cr.get("name", "?")) for _cr in _data.get("creators", [])
|
_cr.get("lastName", _cr.get("name", "?"))
|
||||||
|
for _cr in _data.get("creators", [])
|
||||||
)
|
)
|
||||||
print(f"[{_item_type}] {_title}")
|
print(f"[{_item_type}] {_title}")
|
||||||
if _authors:
|
if _authors:
|
||||||
|
|||||||
@@ -52,9 +52,15 @@ cli = [
|
|||||||
"stack[aco]",
|
"stack[aco]",
|
||||||
"stack[api]",
|
"stack[api]",
|
||||||
"stack[bib]",
|
"stack[bib]",
|
||||||
|
"stack[mail]",
|
||||||
"typer>=0.24.1",
|
"typer>=0.24.1",
|
||||||
"uvicorn>=0.41.0",
|
"uvicorn>=0.41.0",
|
||||||
]
|
]
|
||||||
|
mail = [
|
||||||
|
"httpx>=0.28.1",
|
||||||
|
"pydo>=0.29.0",
|
||||||
|
"resend>=2.0.0",
|
||||||
|
]
|
||||||
cms = [
|
cms = [
|
||||||
"narwhals>=2.17.0",
|
"narwhals>=2.17.0",
|
||||||
"pydantic>=2.0.0",
|
"pydantic>=2.0.0",
|
||||||
@@ -84,6 +90,16 @@ rex = [
|
|||||||
"pyarrow>=23.0.0",
|
"pyarrow>=23.0.0",
|
||||||
"fsspec>=2024.1.0",
|
"fsspec>=2024.1.0",
|
||||||
]
|
]
|
||||||
|
prisma = [
|
||||||
|
"stack[conf]",
|
||||||
|
"stack[bib]",
|
||||||
|
"anthropic>=0.40.0",
|
||||||
|
"httpx[socks]>=0.28.1",
|
||||||
|
"pyyaml>=6.0.0",
|
||||||
|
"pydo>=0.29.0",
|
||||||
|
"pdfminer.six>=20221105",
|
||||||
|
"resend>=2.0.0",
|
||||||
|
]
|
||||||
perf = [
|
perf = [
|
||||||
"stack[conf]",
|
"stack[conf]",
|
||||||
"opentelemetry-api>=1.25.0",
|
"opentelemetry-api>=1.25.0",
|
||||||
@@ -148,6 +164,7 @@ dev = [
|
|||||||
"obstore>=0.9.2",
|
"obstore>=0.9.2",
|
||||||
"s3fs>=2026.2.0",
|
"s3fs>=2026.2.0",
|
||||||
"jinja2>=3.1.0",
|
"jinja2>=3.1.0",
|
||||||
|
"fastexcel>=0.19.0",
|
||||||
]
|
]
|
||||||
|
|
||||||
[build-system]
|
[build-system]
|
||||||
|
|||||||
219
src/bib/email_ingest.py
Normal file
@@ -0,0 +1,219 @@
|
|||||||
|
"""IMAP → bib ingest. Pulls mail from a mailbox and upserts each message
|
||||||
|
as a ``Source`` item with attachments, suitable for downstream Zotero
|
||||||
|
sync.
|
||||||
|
|
||||||
|
Dedup uses two layers:
|
||||||
|
* IMAP ``\\Seen`` flag — set on each successful ingest so re-runs only
|
||||||
|
pull UNSEEN messages.
|
||||||
|
* ``store.upsert`` keys items by ``url`` and we encode each message's
|
||||||
|
``Message-ID`` as ``email:<message-id>`` — re-importing the same
|
||||||
|
message updates instead of duplicating, even if Seen state was lost.
|
||||||
|
|
||||||
|
Attachments are extracted via ``email.message.iter_attachments``
|
||||||
|
(Python 3.6+) and pushed through ``store.attach_file`` so they end up
|
||||||
|
in the bib storage tree alongside the parent Source row.
|
||||||
|
|
||||||
|
CLI: ``stack bib ingest-mail [--user <addr>] [--limit N]``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import email
|
||||||
|
import email.utils
|
||||||
|
import imaplib
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
import ssl
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from email.policy import default as default_policy
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
from bib.item import Source
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from bib.store import Store
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Mailbox:
|
||||||
|
host: str
|
||||||
|
port: int
|
||||||
|
username: str
|
||||||
|
password: str
|
||||||
|
folder: str = "INBOX"
|
||||||
|
|
||||||
|
|
||||||
|
def ingest(
|
||||||
|
store: "Store",
|
||||||
|
mailbox: Mailbox,
|
||||||
|
*,
|
||||||
|
scratch_root: Path = Path(".state/email-ingest"),
|
||||||
|
limit: int | None = None,
|
||||||
|
) -> dict[str, int]:
|
||||||
|
"""Pull UNSEEN messages from *mailbox*, upsert each as a Source.
|
||||||
|
|
||||||
|
Returns counts of {seen, ingested, errors, attached}. ``seen``
|
||||||
|
is the count of UIDs the IMAP server returned as UNSEEN at the
|
||||||
|
start of the run; the others are subsets.
|
||||||
|
"""
|
||||||
|
stats = {"seen": 0, "ingested": 0, "errors": 0, "attached": 0}
|
||||||
|
|
||||||
|
ctx = ssl.create_default_context()
|
||||||
|
conn = imaplib.IMAP4_SSL(mailbox.host, mailbox.port, ssl_context=ctx)
|
||||||
|
try:
|
||||||
|
conn.login(mailbox.username, mailbox.password)
|
||||||
|
conn.select(mailbox.folder)
|
||||||
|
typ, data = conn.uid("SEARCH", None, "UNSEEN")
|
||||||
|
if typ != "OK":
|
||||||
|
log.warning("UNSEEN search failed: %s %s", typ, data)
|
||||||
|
return stats
|
||||||
|
uids = (data[0] or b"").split()
|
||||||
|
if limit:
|
||||||
|
uids = uids[: int(limit)]
|
||||||
|
stats["seen"] = len(uids)
|
||||||
|
|
||||||
|
for uid in uids:
|
||||||
|
try:
|
||||||
|
typ, msg_data = conn.uid("FETCH", uid, "(RFC822)")
|
||||||
|
if typ != "OK" or not msg_data or not msg_data[0]:
|
||||||
|
stats["errors"] += 1
|
||||||
|
continue
|
||||||
|
raw = msg_data[0][1]
|
||||||
|
msg = email.message_from_bytes(raw, policy=default_policy)
|
||||||
|
attached = _ingest_message(store, mailbox, msg, scratch_root)
|
||||||
|
stats["ingested"] += 1
|
||||||
|
stats["attached"] += attached
|
||||||
|
conn.uid("STORE", uid, "+FLAGS", r"(\Seen)")
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
log.warning("ingest uid=%s failed: %s", uid.decode(), e)
|
||||||
|
stats["errors"] += 1
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
conn.close()
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
conn.logout()
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
pass
|
||||||
|
return stats
|
||||||
|
|
||||||
|
|
||||||
|
# ── Per-message ────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def _ingest_message(
|
||||||
|
store: "Store",
|
||||||
|
mailbox: Mailbox,
|
||||||
|
msg: email.message.EmailMessage,
|
||||||
|
scratch_root: Path,
|
||||||
|
) -> int:
|
||||||
|
"""Upsert one message + its attachments. Returns attachment count."""
|
||||||
|
raw_mid = (msg.get("Message-ID") or "").strip().strip("<>")
|
||||||
|
if not raw_mid:
|
||||||
|
# Synthesize from headers — same input always yields same key.
|
||||||
|
raw_mid = "synth:" + _slug(
|
||||||
|
f"{msg.get('Date', '')}|{msg.get('From', '')}|{msg.get('Subject', '')[:60]}"
|
||||||
|
)
|
||||||
|
url = f"email:{raw_mid}"
|
||||||
|
|
||||||
|
sender = (msg.get("From") or "").strip()
|
||||||
|
subject = (msg.get("Subject") or "(no subject)").strip()
|
||||||
|
sender_domain = _sender_domain(sender)
|
||||||
|
body = _extract_body(msg)
|
||||||
|
|
||||||
|
item = Source(title=subject[:255], url=url)
|
||||||
|
item.doc_type = "Email"
|
||||||
|
item.institution = sender_domain or "Email"
|
||||||
|
item.date_published = _date_iso(msg.get("Date", ""))
|
||||||
|
item.abstract = (body or "").strip()[:4000]
|
||||||
|
|
||||||
|
# Tags. mailbox tag uses the local-part so multi-mailbox setups can
|
||||||
|
# filter at the store level (mailbox:cmsupdates, mailbox:postmaster).
|
||||||
|
item.add_tag("source:email")
|
||||||
|
mailbox_local = mailbox.username.split("@", 1)[0]
|
||||||
|
item.add_tag(f"mailbox:{_slug(mailbox_local)}")
|
||||||
|
if sender_domain:
|
||||||
|
item.add_tag(f"sender:{_slug(sender_domain)}")
|
||||||
|
list_id = (msg.get("List-ID") or msg.get("List-Id") or "").strip().strip("<>")
|
||||||
|
if list_id:
|
||||||
|
item.add_tag(f"list:{_slug(list_id)}")
|
||||||
|
if item.date_published:
|
||||||
|
item.add_tag(f"year:{item.date_published[:4]}")
|
||||||
|
|
||||||
|
key = store.upsert(item)
|
||||||
|
|
||||||
|
# Attachments → temp dir → attach_file
|
||||||
|
attached = 0
|
||||||
|
msg_scratch = scratch_root / _slug(raw_mid)[:60]
|
||||||
|
for part in msg.iter_attachments():
|
||||||
|
try:
|
||||||
|
filename = (part.get_filename() or "").strip()
|
||||||
|
if not filename:
|
||||||
|
# Use a content-type-derived filename so multi-attachment
|
||||||
|
# messages without filenames still distinct.
|
||||||
|
ext = (part.get_content_subtype() or "bin").split(";")[0]
|
||||||
|
filename = f"part-{attached + 1}.{ext}"
|
||||||
|
payload = part.get_payload(decode=True)
|
||||||
|
if not payload:
|
||||||
|
continue
|
||||||
|
msg_scratch.mkdir(parents=True, exist_ok=True)
|
||||||
|
dest = msg_scratch / _safe_filename(filename)
|
||||||
|
dest.write_bytes(payload)
|
||||||
|
store.attach_file(key, dest, title=filename)
|
||||||
|
attached += 1
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
log.warning("attachment '%s' for %s: %s", part.get_filename(), raw_mid, e)
|
||||||
|
return attached
|
||||||
|
|
||||||
|
|
||||||
|
# ── Helpers ────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_body(msg: email.message.EmailMessage) -> str:
|
||||||
|
"""Prefer text/plain; fall back to HTML→text."""
|
||||||
|
for part in msg.walk():
|
||||||
|
if part.get_content_type() == "text/plain":
|
||||||
|
try:
|
||||||
|
return part.get_content()
|
||||||
|
except (KeyError, LookupError):
|
||||||
|
continue
|
||||||
|
for part in msg.walk():
|
||||||
|
if part.get_content_type() == "text/html":
|
||||||
|
try:
|
||||||
|
html = part.get_content()
|
||||||
|
# Crude tag-strip; real HTML→text would pull bs4, not
|
||||||
|
# worth the dep for body previews.
|
||||||
|
return re.sub(r"<[^>]+>", " ", html)
|
||||||
|
except (KeyError, LookupError):
|
||||||
|
continue
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def _sender_domain(sender: str) -> str:
|
||||||
|
m = re.search(r"<([^@<>]+)@([^>]+)>", sender)
|
||||||
|
if m:
|
||||||
|
return m.group(2)
|
||||||
|
m = re.search(r"@([^\s]+)", sender)
|
||||||
|
return m.group(1) if m else ""
|
||||||
|
|
||||||
|
|
||||||
|
def _slug(s: str) -> str:
|
||||||
|
return re.sub(r"[^a-z0-9]+", "-", (s or "").lower()).strip("-")[:60]
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_filename(s: str) -> str:
|
||||||
|
return re.sub(r"[^A-Za-z0-9._-]+", "_", s)[:200] or "attachment"
|
||||||
|
|
||||||
|
|
||||||
|
def _date_iso(date_str: str) -> str:
|
||||||
|
if not date_str:
|
||||||
|
return ""
|
||||||
|
try:
|
||||||
|
dt = email.utils.parsedate_to_datetime(date_str)
|
||||||
|
return dt.date().isoformat()
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
return ""
|
||||||
193
src/bib/federalregister.py
Normal file
@@ -0,0 +1,193 @@
|
|||||||
|
"""Federal Register API client — rule discovery.
|
||||||
|
|
||||||
|
Thin wrapper over ``federalregister.gov/api/v1/documents``. Use it to
|
||||||
|
enumerate every rule matching a pattern (e.g. all PFS proposed + final
|
||||||
|
rules since 2017), then hand the document numbers off to
|
||||||
|
:func:`bib.translate.federal_register` or to
|
||||||
|
:mod:`bib.regulations_gov` for comment retrieval.
|
||||||
|
|
||||||
|
API docs: https://www.federalregister.gov/reader-aids/developer-resources/rest-api-documentation
|
||||||
|
|
||||||
|
No auth required. Rate limit: ~60 req/min. We batch with ``per_page=100``
|
||||||
|
which is the max the API returns.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import Iterator
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
_BASE = "https://www.federalregister.gov/api/v1"
|
||||||
|
|
||||||
|
# PFS (CY NNNN Physician Fee Schedule, Medicare Part B) proposed and
|
||||||
|
# final rules. Title wording shifted over the years:
|
||||||
|
#
|
||||||
|
# 2017–2019 — "Revisions to Payment Policies Under the Physician Fee Schedule"
|
||||||
|
# 2020+ — "CY <YYYY> Payment Policies Under the Physician Fee Schedule"
|
||||||
|
#
|
||||||
|
# Both forms share "Payment Policies Under the Physician Fee Schedule".
|
||||||
|
# Use that substring to match every era without sweeping in unrelated
|
||||||
|
# rules that only mention the phrase "physician fee schedule" in passing.
|
||||||
|
PFS_TITLE_PATTERN = "Payment Policies Under the Physician Fee Schedule"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class FRDocument:
|
||||||
|
"""Subset of a Federal Register document we care about for rule
|
||||||
|
discovery + comment retrieval."""
|
||||||
|
|
||||||
|
document_number: str
|
||||||
|
title: str
|
||||||
|
type: str # "Rule" (final) | "Proposed Rule"
|
||||||
|
publication_date: str # YYYY-MM-DD
|
||||||
|
dockets: list[str] # e.g. ["CMS-1676-P"]
|
||||||
|
volume: str = ""
|
||||||
|
start_page: str = ""
|
||||||
|
regulation_id_numbers: list[str] | None = None
|
||||||
|
html_url: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
# ── Search ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def search(
|
||||||
|
*,
|
||||||
|
term: str = "",
|
||||||
|
agencies: list[str] | None = None,
|
||||||
|
types: list[str] | None = None,
|
||||||
|
since: str | None = None,
|
||||||
|
until: str | None = None,
|
||||||
|
per_page: int = 100,
|
||||||
|
client: httpx.Client | None = None,
|
||||||
|
) -> Iterator[FRDocument]:
|
||||||
|
"""Yield every FR document matching the given filters, paginating
|
||||||
|
through every page.
|
||||||
|
|
||||||
|
Parameters mirror the API's ``conditions[…]`` keys. ``types`` values
|
||||||
|
use the API's enum: ``"RULE"`` (final rules) and ``"PRORULE"``
|
||||||
|
(proposed rules).
|
||||||
|
"""
|
||||||
|
own_client = client is None
|
||||||
|
client = client or httpx.Client(timeout=30)
|
||||||
|
try:
|
||||||
|
params: list[tuple[str, str]] = [
|
||||||
|
("per_page", str(per_page)),
|
||||||
|
("order", "oldest"),
|
||||||
|
*_fields(),
|
||||||
|
]
|
||||||
|
if term:
|
||||||
|
params.append(("conditions[term]", term))
|
||||||
|
for a in agencies or []:
|
||||||
|
params.append(("conditions[agencies][]", a))
|
||||||
|
for t in types or []:
|
||||||
|
params.append(("conditions[type][]", t))
|
||||||
|
if since:
|
||||||
|
params.append(("conditions[publication_date][gte]", since))
|
||||||
|
if until:
|
||||||
|
params.append(("conditions[publication_date][lte]", until))
|
||||||
|
|
||||||
|
page = 1
|
||||||
|
while True:
|
||||||
|
paged = params + [("page", str(page))]
|
||||||
|
r = client.get(f"{_BASE}/documents", params=paged)
|
||||||
|
r.raise_for_status()
|
||||||
|
data = r.json()
|
||||||
|
for result in data.get("results", []):
|
||||||
|
yield _to_doc(result)
|
||||||
|
if page >= data.get("total_pages", 1):
|
||||||
|
break
|
||||||
|
page += 1
|
||||||
|
finally:
|
||||||
|
if own_client:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
def split_docket_ids(raw: list[str]) -> list[str]:
|
||||||
|
"""Clean the FR API's docket_ids list into individual CMS-XXXX-Y ids.
|
||||||
|
|
||||||
|
The API sometimes returns a single-element list whose only entry is
|
||||||
|
a human-written aggregate like::
|
||||||
|
|
||||||
|
["CMS-1693-F, CMS-1693-IFC, CMS-5522-F3, and CMS-1701-F"]
|
||||||
|
|
||||||
|
Split on commas + "and" to recover the atoms. Dedup preserving order.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
|
||||||
|
out: list[str] = []
|
||||||
|
seen: set[str] = set()
|
||||||
|
for entry in raw:
|
||||||
|
for atom in re.split(r",\s*|\s+and\s+", entry):
|
||||||
|
atom = atom.strip()
|
||||||
|
if atom and atom not in seen:
|
||||||
|
seen.add(atom)
|
||||||
|
out.append(atom)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def pfs_rules(
|
||||||
|
*,
|
||||||
|
since: str = "2017-01-01",
|
||||||
|
until: str | None = None,
|
||||||
|
client: httpx.Client | None = None,
|
||||||
|
) -> list[FRDocument]:
|
||||||
|
"""Every PFS proposed/final rule since *since* (inclusive).
|
||||||
|
|
||||||
|
Title-filtered so we get the true annual PFS rules, not every CMS
|
||||||
|
rule that mentions PFS in passing.
|
||||||
|
"""
|
||||||
|
found: list[FRDocument] = []
|
||||||
|
seen: set[str] = set()
|
||||||
|
for doc in search(
|
||||||
|
term="physician fee schedule",
|
||||||
|
agencies=["centers-for-medicare-medicaid-services"],
|
||||||
|
types=["RULE", "PRORULE"],
|
||||||
|
since=since,
|
||||||
|
until=until,
|
||||||
|
client=client,
|
||||||
|
):
|
||||||
|
if PFS_TITLE_PATTERN.lower() not in doc.title.lower():
|
||||||
|
continue
|
||||||
|
if doc.document_number in seen:
|
||||||
|
continue
|
||||||
|
seen.add(doc.document_number)
|
||||||
|
found.append(doc)
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
# ── Internals ───────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def _fields() -> list[tuple[str, str]]:
|
||||||
|
"""Fields we need from the API — smaller payloads, fewer timeouts."""
|
||||||
|
names = [
|
||||||
|
"document_number",
|
||||||
|
"title",
|
||||||
|
"type",
|
||||||
|
"publication_date",
|
||||||
|
"docket_ids",
|
||||||
|
"volume",
|
||||||
|
"start_page",
|
||||||
|
"regulation_id_numbers",
|
||||||
|
"html_url",
|
||||||
|
]
|
||||||
|
return [("fields[]", n) for n in names]
|
||||||
|
|
||||||
|
|
||||||
|
def _to_doc(raw: dict) -> FRDocument:
|
||||||
|
return FRDocument(
|
||||||
|
document_number=raw.get("document_number", ""),
|
||||||
|
title=raw.get("title", ""),
|
||||||
|
type=raw.get("type", ""),
|
||||||
|
publication_date=raw.get("publication_date", ""),
|
||||||
|
dockets=list(raw.get("docket_ids") or []),
|
||||||
|
volume=str(raw.get("volume") or ""),
|
||||||
|
start_page=str(raw.get("start_page") or ""),
|
||||||
|
regulation_id_numbers=list(raw.get("regulation_id_numbers") or []),
|
||||||
|
html_url=raw.get("html_url", ""),
|
||||||
|
)
|
||||||
446
src/bib/iom.py
Normal file
@@ -0,0 +1,446 @@
|
|||||||
|
"""CMS Internet-Only Manual (IOM) index crawler.
|
||||||
|
|
||||||
|
Walks the public CMS IOM index page, enumerates every manual, and for
|
||||||
|
each one discovers its chapter PDFs from the corresponding landing
|
||||||
|
page. Produces :class:`bib.item.Manual` records and idempotently
|
||||||
|
upserts them via the bibliography store.
|
||||||
|
|
||||||
|
The scraper is HTML-regex based — CMS's IOM pages have very stable
|
||||||
|
structure (every chapter sits inside a ``<ul class="field__items">``
|
||||||
|
with an anchor pointing to ``…/manuals/downloads/<prefix><pub>c<NN>.pdf``)
|
||||||
|
— so the heavier HTML parsers pull their weight elsewhere.
|
||||||
|
|
||||||
|
Usage::
|
||||||
|
|
||||||
|
from bib import connect
|
||||||
|
from bib.iom import ingest_all
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
store = connect()
|
||||||
|
with httpx.Client() as client:
|
||||||
|
summary = ingest_all(store, client)
|
||||||
|
# {"100-02": 18, "100-04": 38, ...}
|
||||||
|
|
||||||
|
Or from the CLI: ``uv run stack bib ingest-iom``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import html
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import TYPE_CHECKING, Iterable
|
||||||
|
|
||||||
|
from bib.item import Download
|
||||||
|
from bib.tag import Tag
|
||||||
|
from bib.translate import cms_manual
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
from bib.store import Store
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
INDEX_URL = "https://www.cms.gov/medicare/regulations-guidance/manuals/internet-only-manuals-ioms"
|
||||||
|
BASE = "https://www.cms.gov"
|
||||||
|
# CMS publishes upcoming transmittal schedules here. We poll it to detect
|
||||||
|
# IOM changes between crawls — any checksum change is a signal to
|
||||||
|
# re-ingest affected manuals.
|
||||||
|
FUTURE_PDF_URL = "https://www.cms.gov/regulations-and-guidance/guidance/manuals/downloads/futurepdf.pdf"
|
||||||
|
|
||||||
|
# Publication-number → filename prefix used by CMS download URLs.
|
||||||
|
# Discovered by inspecting each IOM's landing page. Not authoritative
|
||||||
|
# — the scraper doesn't depend on this map (it trusts the anchor hrefs
|
||||||
|
# directly) but it makes post-hoc URL parsing possible elsewhere.
|
||||||
|
PUB_PREFIXES: dict[str, str] = {
|
||||||
|
"100-01": "ge101",
|
||||||
|
"100-02": "bp102",
|
||||||
|
"100-03": "ncd103",
|
||||||
|
"100-04": "clm104",
|
||||||
|
"100-05": "msp105",
|
||||||
|
"100-06": "fin106",
|
||||||
|
"100-07": "som107",
|
||||||
|
"100-08": "pim83", # CMS uses pim83c, not pim108
|
||||||
|
"100-09": "qio109",
|
||||||
|
"100-10": "bp102", # Hospice shares file stem in practice; varies
|
||||||
|
"100-11": "hha111",
|
||||||
|
# Newer pubs (100-12..100-25) vary in prefix; we rely on anchor
|
||||||
|
# hrefs for those.
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class IOMEntry:
|
||||||
|
"""One row on the IOM index page."""
|
||||||
|
|
||||||
|
pub: str # e.g. "100-02"
|
||||||
|
title: str # e.g. "Medicare Benefit Policy Manual"
|
||||||
|
landing_url: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ChapterEntry:
|
||||||
|
"""One chapter PDF discovered on an IOM landing page."""
|
||||||
|
|
||||||
|
pub: str
|
||||||
|
manual: str # display name from index page
|
||||||
|
chapter: str # "1", "6A", …
|
||||||
|
title: str # "Inpatient Hospital Services Covered Under Part A"
|
||||||
|
url: str
|
||||||
|
|
||||||
|
|
||||||
|
# ── Scraping ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
_INDEX_ROW = re.compile(
|
||||||
|
r'<a\s+href="(?P<href>/regulations-and-guidance/guidance/manuals/'
|
||||||
|
r'internet-only-manuals-ioms-items/cms\d+)"[^>]*>\s*'
|
||||||
|
r"(?P<pub>100(?:-\d+)?)\s*</a>"
|
||||||
|
r".*?Title</label>\s*(?P<title>[^<]+?)\s*</div>",
|
||||||
|
re.DOTALL | re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_index(client: httpx.Client) -> list[IOMEntry]:
|
||||||
|
"""Scrape the IOM index table."""
|
||||||
|
r = client.get(INDEX_URL, follow_redirects=True, timeout=30)
|
||||||
|
r.raise_for_status()
|
||||||
|
seen: set[str] = set()
|
||||||
|
out: list[IOMEntry] = []
|
||||||
|
for m in _INDEX_ROW.finditer(r.text):
|
||||||
|
pub = m.group("pub").strip()
|
||||||
|
if pub in seen:
|
||||||
|
continue
|
||||||
|
seen.add(pub)
|
||||||
|
title = html.unescape(re.sub(r"\s+", " ", m.group("title"))).strip()
|
||||||
|
out.append(
|
||||||
|
IOMEntry(
|
||||||
|
pub=pub,
|
||||||
|
title=title,
|
||||||
|
landing_url=BASE + m.group("href"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
_CHAPTER_LINK = re.compile(
|
||||||
|
r'<a[^>]*href="(?P<href>/regulations-and-guidance/guidance/manuals/'
|
||||||
|
r'downloads/[^"]+\.pdf)"[^>]*>\s*(?P<label>[^<]+?)\s*</a>',
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
_CHAPTER_HEAD = re.compile(
|
||||||
|
r"""^(?:Chapter|Ch\.)\s+
|
||||||
|
(?P<num>\d+[A-Za-z]?) # chapter number, optional trailing letter
|
||||||
|
(?:\s*[-–—:.,]\s*(?P<rest>.*))?
|
||||||
|
$""",
|
||||||
|
re.IGNORECASE | re.VERBOSE,
|
||||||
|
)
|
||||||
|
# Some manuals (e.g. NCD 100-03) split a single chapter across multiple
|
||||||
|
# "Part N" PDFs. Capture the part so we can disambiguate the dedup key.
|
||||||
|
_CHAPTER_PART = re.compile(r"\bPart\s+(?P<part>\d+)\b", re.IGNORECASE)
|
||||||
|
_SUPPLEMENT_HINTS = re.compile(
|
||||||
|
r"crosswalk|addendum|appendix|transmittal|revision\s+history|old\s+version"
|
||||||
|
r"|table\s+of\s+contents|archive|transition|change\s+request|redline",
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
# "Pub 100-18 - Medicare Prescription Drug Benefit Manual" — single-file
|
||||||
|
# manuals that don't split into chapters. We ingest them as one record
|
||||||
|
# with chapter="" so the whole-PDF content is still discoverable.
|
||||||
|
_WHOLE_PUB = re.compile(
|
||||||
|
r"^(?:Pub\s*100[_-]?\d+|Pub100_\d+)\b\s*[-–—:.]?\s*(?P<rest>.*)$",
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_chapters(client: httpx.Client, entry: IOMEntry) -> list[ChapterEntry]:
|
||||||
|
"""Extract the chapter list from one IOM's landing page.
|
||||||
|
|
||||||
|
Skips crosswalks, addenda, transmittals — the chapter PDFs are the
|
||||||
|
source-of-truth for content; supplements get their own issues if
|
||||||
|
they're needed later. Single-file manuals (no per-chapter PDFs) are
|
||||||
|
captured as one ChapterEntry with chapter="".
|
||||||
|
"""
|
||||||
|
r = client.get(entry.landing_url, follow_redirects=True, timeout=30)
|
||||||
|
r.raise_for_status()
|
||||||
|
chapters: list[ChapterEntry] = []
|
||||||
|
whole: list[ChapterEntry] = []
|
||||||
|
seen: set[str] = set()
|
||||||
|
for m in _CHAPTER_LINK.finditer(r.text):
|
||||||
|
label = html.unescape(re.sub(r"\s+", " ", m.group("label"))).strip()
|
||||||
|
if _SUPPLEMENT_HINTS.search(label):
|
||||||
|
continue
|
||||||
|
url = BASE + m.group("href")
|
||||||
|
head = _CHAPTER_HEAD.match(label)
|
||||||
|
if head:
|
||||||
|
num = head.group("num")
|
||||||
|
rest = (head.group("rest") or "").strip()
|
||||||
|
part_match = _CHAPTER_PART.search(rest)
|
||||||
|
chapter_id = f"{num}P{part_match.group('part')}" if part_match else num
|
||||||
|
key = chapter_id.upper()
|
||||||
|
if key in seen:
|
||||||
|
continue
|
||||||
|
seen.add(key)
|
||||||
|
chapters.append(
|
||||||
|
ChapterEntry(
|
||||||
|
pub=entry.pub,
|
||||||
|
manual=entry.title,
|
||||||
|
chapter=chapter_id,
|
||||||
|
title=rest or label,
|
||||||
|
url=url,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
whole_m = _WHOLE_PUB.match(label)
|
||||||
|
if whole_m:
|
||||||
|
whole.append(
|
||||||
|
ChapterEntry(
|
||||||
|
pub=entry.pub,
|
||||||
|
manual=entry.title,
|
||||||
|
chapter="",
|
||||||
|
title=whole_m.group("rest").strip() or entry.title,
|
||||||
|
url=url,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
# Chapters take precedence; fall back to a single whole-manual
|
||||||
|
# record when the landing page doesn't expose per-chapter PDFs.
|
||||||
|
return chapters if chapters else whole
|
||||||
|
|
||||||
|
|
||||||
|
# ── Orchestration ────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def _short_manual_name(full_title: str) -> str:
|
||||||
|
"""Drop "Medicare" prefix and "Manual" suffix for the stored name."""
|
||||||
|
s = re.sub(r"^Medicare\s+", "", full_title.strip())
|
||||||
|
s = re.sub(r"\s+Manual\s*$", "", s)
|
||||||
|
return s.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def ingest_entry(
|
||||||
|
store: Store,
|
||||||
|
client: httpx.Client,
|
||||||
|
entry: IOMEntry,
|
||||||
|
*,
|
||||||
|
prune: bool = True,
|
||||||
|
) -> list[str]:
|
||||||
|
"""Ingest a single IOM. Returns the store keys of upserted items.
|
||||||
|
|
||||||
|
With ``prune=True`` (default), any prior Manual records for this
|
||||||
|
pub whose URLs don't appear in the freshly-discovered set are
|
||||||
|
deleted. That removes legacy dupes caused by CMS renaming chapter
|
||||||
|
filenames (e.g. ``clm104c18pdf.pdf`` → ``clm104c18.pdf``).
|
||||||
|
"""
|
||||||
|
keys: list[str] = []
|
||||||
|
short = _short_manual_name(entry.title)
|
||||||
|
fresh = fetch_chapters(client, entry)
|
||||||
|
fresh_urls = {c.url for c in fresh}
|
||||||
|
for ch in fresh:
|
||||||
|
title = (
|
||||||
|
f"{entry.title} — Chapter {ch.chapter}: {ch.title}"
|
||||||
|
if ch.chapter
|
||||||
|
else f"{entry.title}: {ch.title}"
|
||||||
|
)
|
||||||
|
item = cms_manual(ch.url, manual=short, chapter=ch.chapter)
|
||||||
|
item.title = title
|
||||||
|
item.pub_number = entry.pub
|
||||||
|
item.add_tag(Tag.source("iom").label)
|
||||||
|
item.add_tag(f"pub:{entry.pub}")
|
||||||
|
keys.append(store.upsert(item))
|
||||||
|
if prune:
|
||||||
|
_prune_stale(store, entry.pub, fresh_urls)
|
||||||
|
return keys
|
||||||
|
|
||||||
|
|
||||||
|
def _prune_stale(store: Store, pub: str, live_urls: set[str]) -> int:
|
||||||
|
"""Delete Manual rows for this pub whose URL isn't in the live set.
|
||||||
|
|
||||||
|
Uses the store's sqlite connection directly because the public API
|
||||||
|
doesn't expose a "list by pub_number" helper.
|
||||||
|
"""
|
||||||
|
con = store._con() # noqa: SLF001 — internal access by design here
|
||||||
|
rows = con.execute(
|
||||||
|
"""
|
||||||
|
SELECT key, url FROM items
|
||||||
|
WHERE item_type='manual'
|
||||||
|
AND json_extract(extra_json, '$.pub_number') = ?
|
||||||
|
""",
|
||||||
|
(pub,),
|
||||||
|
).fetchall()
|
||||||
|
stale = [r["key"] for r in rows if r["url"] and r["url"] not in live_urls]
|
||||||
|
for key in stale:
|
||||||
|
store.delete(key)
|
||||||
|
if stale:
|
||||||
|
log.info("pruned %d stale record(s) for pub %s", len(stale), pub)
|
||||||
|
return len(stale)
|
||||||
|
|
||||||
|
|
||||||
|
def ingest_all(
|
||||||
|
store: Store,
|
||||||
|
client: httpx.Client,
|
||||||
|
*,
|
||||||
|
pubs: Iterable[str] | None = None,
|
||||||
|
prune: bool = True,
|
||||||
|
) -> dict[str, int]:
|
||||||
|
"""Walk the IOM index and upsert every chapter (or just the filter).
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
pubs : iterable of pub numbers (e.g. ``["100-02", "100-08"]``) to
|
||||||
|
limit ingestion to a subset. Defaults to every IOM on the index.
|
||||||
|
|
||||||
|
Returns
|
||||||
|
-------
|
||||||
|
dict
|
||||||
|
Mapping of pub number → count of chapters upserted.
|
||||||
|
"""
|
||||||
|
wanted = set(pubs) if pubs else None
|
||||||
|
summary: dict[str, int] = {}
|
||||||
|
for entry in fetch_index(client):
|
||||||
|
if wanted is not None and entry.pub not in wanted:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
keys = ingest_entry(store, client, entry, prune=prune)
|
||||||
|
except Exception as e: # noqa: BLE001 — one manual failing shouldn't stop the others
|
||||||
|
log.warning("IOM %s (%s) failed: %s", entry.pub, entry.title, e)
|
||||||
|
summary[entry.pub] = 0
|
||||||
|
continue
|
||||||
|
log.info("IOM %s (%s): %d chapters", entry.pub, entry.title, len(keys))
|
||||||
|
summary[entry.pub] = len(keys)
|
||||||
|
return summary
|
||||||
|
|
||||||
|
|
||||||
|
# ── Attachments ──────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def _sha256(path: Path) -> str:
|
||||||
|
h = hashlib.sha256()
|
||||||
|
with open(path, "rb") as f:
|
||||||
|
for chunk in iter(lambda: f.read(1 << 20), b""):
|
||||||
|
h.update(chunk)
|
||||||
|
return h.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _existing_attachment_hash(store: Store, item_key: str) -> str | None:
|
||||||
|
"""SHA-256 of an item's first already-attached file, or None."""
|
||||||
|
con = store._con() # noqa: SLF001
|
||||||
|
row = con.execute(
|
||||||
|
"""SELECT a.storage_path FROM attachments a
|
||||||
|
JOIN items i ON a.item_id = i.id
|
||||||
|
WHERE i.key = ? ORDER BY a.id LIMIT 1""",
|
||||||
|
(item_key,),
|
||||||
|
).fetchone()
|
||||||
|
if not row or not row["storage_path"]:
|
||||||
|
return None
|
||||||
|
p = Path(row["storage_path"])
|
||||||
|
return _sha256(p) if p.is_file() else None
|
||||||
|
|
||||||
|
|
||||||
|
def download_attachments(
|
||||||
|
store: Store,
|
||||||
|
client: httpx.Client,
|
||||||
|
*,
|
||||||
|
pubs: Iterable[str] | None = None,
|
||||||
|
force: bool = False,
|
||||||
|
tmp_dir: Path | None = None,
|
||||||
|
) -> dict[str, int]:
|
||||||
|
"""Download every Manual's PDF and attach it via Store.attach_file.
|
||||||
|
|
||||||
|
Idempotent: the remote file's SHA-256 is compared against the
|
||||||
|
currently-attached copy, and the download is skipped if unchanged.
|
||||||
|
Returns ``{pub: count_attached}``.
|
||||||
|
"""
|
||||||
|
wanted = set(pubs) if pubs else None
|
||||||
|
tmp = tmp_dir or (Path(store._db_path).parent / "tmp-iom") # noqa: SLF001
|
||||||
|
tmp.mkdir(parents=True, exist_ok=True)
|
||||||
|
results: dict[str, int] = {}
|
||||||
|
|
||||||
|
items = store.list_items(item_type="manual")
|
||||||
|
for item in items:
|
||||||
|
ej = json.loads(item.to_row().get("extra_json", "{}"))
|
||||||
|
pub = ej.get("pub_number", "")
|
||||||
|
if wanted is not None and pub not in wanted:
|
||||||
|
continue
|
||||||
|
url = item.url
|
||||||
|
if not url:
|
||||||
|
continue
|
||||||
|
filename = url.rsplit("/", 1)[-1]
|
||||||
|
tmp_path = tmp / filename
|
||||||
|
try:
|
||||||
|
r = client.get(url, follow_redirects=True, timeout=60)
|
||||||
|
r.raise_for_status()
|
||||||
|
tmp_path.write_bytes(r.content)
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
log.warning("download failed: %s — %s", url, e)
|
||||||
|
continue
|
||||||
|
|
||||||
|
remote_hash = _sha256(tmp_path)
|
||||||
|
if not force and _existing_attachment_hash(store, item.key) == remote_hash:
|
||||||
|
tmp_path.unlink(missing_ok=True)
|
||||||
|
continue
|
||||||
|
|
||||||
|
store.attach_file(item.key, tmp_path, title=item.title)
|
||||||
|
tmp_path.unlink(missing_ok=True)
|
||||||
|
results[pub] = results.get(pub, 0) + 1
|
||||||
|
log.info("attached: %s (%s)", filename, pub or "?")
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
# ── Future-updates watcher ──────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
_FUTURE_ITEM_KEY_TAG = "iom:futurepdf"
|
||||||
|
|
||||||
|
|
||||||
|
def _futurepdf_anchor(store: Store) -> str:
|
||||||
|
"""Return the key of the anchor Download item tracking futurepdf.pdf.
|
||||||
|
|
||||||
|
Creates one on first call. We use a tag rather than a stable key so
|
||||||
|
the item can live alongside the normal collection hierarchy.
|
||||||
|
"""
|
||||||
|
existing = store.list_items(tag=_FUTURE_ITEM_KEY_TAG)
|
||||||
|
if existing:
|
||||||
|
return existing[0].key
|
||||||
|
item = Download(
|
||||||
|
title="CMS IOM — Future Transmittal Schedule (futurepdf.pdf)",
|
||||||
|
url=FUTURE_PDF_URL,
|
||||||
|
)
|
||||||
|
item.add_tag(_FUTURE_ITEM_KEY_TAG)
|
||||||
|
item.add_tag(Tag.source("iom").label)
|
||||||
|
item.add_tag("pub:futurepdf")
|
||||||
|
return store.upsert(item)
|
||||||
|
|
||||||
|
|
||||||
|
def check_future_updates(
|
||||||
|
store: Store,
|
||||||
|
client: httpx.Client,
|
||||||
|
*,
|
||||||
|
tmp_dir: Path | None = None,
|
||||||
|
) -> tuple[bool, str]:
|
||||||
|
"""Poll CMS futurepdf.pdf; attach if changed; return (changed, sha256).
|
||||||
|
|
||||||
|
Pass this into a cron/daemon. When ``changed`` is True, run
|
||||||
|
``ingest_all`` + ``download_attachments`` to pick up the new chapters
|
||||||
|
CMS is advertising on its schedule.
|
||||||
|
"""
|
||||||
|
anchor_key = _futurepdf_anchor(store)
|
||||||
|
tmp = tmp_dir or (Path(store._db_path).parent / "tmp-iom") # noqa: SLF001
|
||||||
|
tmp.mkdir(parents=True, exist_ok=True)
|
||||||
|
path = tmp / "futurepdf.pdf"
|
||||||
|
|
||||||
|
r = client.get(FUTURE_PDF_URL, follow_redirects=True, timeout=60)
|
||||||
|
r.raise_for_status()
|
||||||
|
path.write_bytes(r.content)
|
||||||
|
new_hash = _sha256(path)
|
||||||
|
|
||||||
|
prior = _existing_attachment_hash(store, anchor_key)
|
||||||
|
changed = prior != new_hash
|
||||||
|
if changed:
|
||||||
|
store.attach_file(anchor_key, path, title=f"futurepdf.pdf @ {new_hash[:8]}")
|
||||||
|
log.info("futurepdf.pdf CHANGED — prior=%s new=%s", prior, new_hash)
|
||||||
|
path.unlink(missing_ok=True)
|
||||||
|
return changed, new_hash
|
||||||
309
src/bib/oig.py
Normal file
@@ -0,0 +1,309 @@
|
|||||||
|
"""HHS OIG compliance guidance + fraud alert crawler.
|
||||||
|
|
||||||
|
Walks OIG's public CPG and alerts indexes, extracts every PDF/HTML
|
||||||
|
guidance document, and upserts them as :class:`bib.item.Source` records
|
||||||
|
tagged with ``agency:oig`` plus the specific guidance type.
|
||||||
|
|
||||||
|
Covered sources:
|
||||||
|
|
||||||
|
- ``/compliance/compliance-guidance/`` — sector-specific CPGs, supplemental
|
||||||
|
CPGs, Industry-Specific CPGs (ICPGs), drafts.
|
||||||
|
- ``/compliance/alerts/`` — Special Fraud Alerts, Advisory Bulletins,
|
||||||
|
Enforcement Alerts, Advisory Opinions that live on the alerts page.
|
||||||
|
|
||||||
|
Usage mirrors :mod:`bib.iom`::
|
||||||
|
|
||||||
|
from bib import connect
|
||||||
|
from bib.oig import ingest_all, download_attachments
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
store = connect()
|
||||||
|
with httpx.Client() as client:
|
||||||
|
ingest_all(store, client)
|
||||||
|
download_attachments(store, client)
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import html
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import TYPE_CHECKING, Iterable
|
||||||
|
|
||||||
|
from bib.item import Source
|
||||||
|
from bib.tag import Tag
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
from bib.store import Store
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
BASE = "https://oig.hhs.gov"
|
||||||
|
CPG_URL = f"{BASE}/compliance/compliance-guidance/"
|
||||||
|
ALERTS_URL = f"{BASE}/compliance/alerts/"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class OIGDoc:
|
||||||
|
"""One scraped OIG guidance record."""
|
||||||
|
|
||||||
|
title: str # human-readable title from anchor text
|
||||||
|
url: str # absolute doc URL
|
||||||
|
guidance_type: str # cpg | sfa | sab | ea | enforcement | other
|
||||||
|
sector: str = "" # "hospitals", "dme", "nursing", …
|
||||||
|
|
||||||
|
|
||||||
|
# ── Heuristics ───────────────────────────────────────────────────
|
||||||
|
|
||||||
|
# Classifier: title → guidance_type. Order matters (most specific first).
|
||||||
|
_TYPE_RULES: list[tuple[re.Pattern[str], str]] = [
|
||||||
|
(re.compile(r"special\s+fraud\s+alert", re.I), "sfa"),
|
||||||
|
(re.compile(r"special\s+advisory\s+bulletin", re.I), "sab"),
|
||||||
|
(re.compile(r"advisory\s+bulletin", re.I), "sab"),
|
||||||
|
(re.compile(r"enforcement\s+alert", re.I), "ea"),
|
||||||
|
(re.compile(r"\b(?:compliance\s+program\s+guidance|cpg|icpg|gcpg)\b", re.I), "cpg"),
|
||||||
|
(re.compile(r"open\s+letter", re.I), "open_letter"),
|
||||||
|
]
|
||||||
|
|
||||||
|
# Sector tagger: crude keyword match on titles. Unknown → "".
|
||||||
|
_SECTOR_KEYWORDS: dict[str, str] = {
|
||||||
|
"hospital": "hospitals",
|
||||||
|
"nursing": "nursing",
|
||||||
|
"hospice": "hospice",
|
||||||
|
"ambulance": "ambulance",
|
||||||
|
"home health": "home_health",
|
||||||
|
"home care": "home_health",
|
||||||
|
"pharmaceutical": "pharmaceutical",
|
||||||
|
"clinical laborator": "labs",
|
||||||
|
"durable medical equipment": "dme",
|
||||||
|
"dme": "dme",
|
||||||
|
"physician": "physicians",
|
||||||
|
"medicare advantage": "medicare_advantage",
|
||||||
|
"medicare+choice": "medicare_advantage",
|
||||||
|
"third-party medical billing": "billing",
|
||||||
|
"pharmacy benefit": "pbm",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Skip hints: labels that point at navigation, category pages, or
|
||||||
|
# non-document links.
|
||||||
|
_SKIP_HINTS = re.compile(
|
||||||
|
r"^(?:download|view|click\s+here|\s*-\s*|more\s+information|read\s+more)\s*$",
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _classify_type(title: str, href: str) -> str:
|
||||||
|
for pat, kind in _TYPE_RULES:
|
||||||
|
if pat.search(title):
|
||||||
|
return kind
|
||||||
|
if "/special-fraud-alerts/" in href:
|
||||||
|
return "sfa"
|
||||||
|
if "/special-advisory-bulletins/" in href:
|
||||||
|
return "sab"
|
||||||
|
if "/compliance-guidance/" in href:
|
||||||
|
return "cpg"
|
||||||
|
return "other"
|
||||||
|
|
||||||
|
|
||||||
|
def _classify_sector(title: str) -> str:
|
||||||
|
low = title.lower()
|
||||||
|
for kw, tag in _SECTOR_KEYWORDS.items():
|
||||||
|
if kw in low:
|
||||||
|
return tag
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
# ── Scraping ─────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
# OIG PDFs live under /documents/... (compliance-guidance, special-fraud-alerts,
|
||||||
|
# special-advisory-bulletins, root/…). HTML articles under /documents/... .html
|
||||||
|
# also count — we ingest them the same way.
|
||||||
|
_DOC_LINK = re.compile(
|
||||||
|
r'<a[^>]*href="(?P<href>/documents/[^"]+?\.(?:pdf|html?))"[^>]*>'
|
||||||
|
r"\s*(?P<label>[^<]+?)\s*</a>",
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _fetch(client: httpx.Client, url: str) -> str:
|
||||||
|
r = client.get(url, follow_redirects=True, timeout=30)
|
||||||
|
r.raise_for_status()
|
||||||
|
return r.text
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_docs(page_html: str) -> list[tuple[str, str]]:
|
||||||
|
"""Return [(href, clean_title), …] for every doc link on a page."""
|
||||||
|
out: list[tuple[str, str]] = []
|
||||||
|
seen_hrefs: set[str] = set()
|
||||||
|
for m in _DOC_LINK.finditer(page_html):
|
||||||
|
href = m.group("href")
|
||||||
|
if href in seen_hrefs:
|
||||||
|
continue
|
||||||
|
label = html.unescape(re.sub(r"\s+", " ", m.group("label"))).strip()
|
||||||
|
if not label or _SKIP_HINTS.match(label):
|
||||||
|
continue
|
||||||
|
seen_hrefs.add(href)
|
||||||
|
out.append((href, label))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_cpgs(client: httpx.Client) -> list[OIGDoc]:
|
||||||
|
"""Scrape the OIG Compliance Program Guidance index."""
|
||||||
|
docs: list[OIGDoc] = []
|
||||||
|
for href, title in _extract_docs(_fetch(client, CPG_URL)):
|
||||||
|
docs.append(
|
||||||
|
OIGDoc(
|
||||||
|
title=title,
|
||||||
|
url=BASE + href,
|
||||||
|
guidance_type=_classify_type(title, href),
|
||||||
|
sector=_classify_sector(title),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return docs
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_alerts(client: httpx.Client) -> list[OIGDoc]:
|
||||||
|
"""Scrape the OIG Special Fraud Alerts / Advisory Bulletins index."""
|
||||||
|
docs: list[OIGDoc] = []
|
||||||
|
for href, title in _extract_docs(_fetch(client, ALERTS_URL)):
|
||||||
|
docs.append(
|
||||||
|
OIGDoc(
|
||||||
|
title=title,
|
||||||
|
url=BASE + href,
|
||||||
|
guidance_type=_classify_type(title, href),
|
||||||
|
sector=_classify_sector(title),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return docs
|
||||||
|
|
||||||
|
|
||||||
|
# ── Ingestion ────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def _build_item(doc: OIGDoc) -> Source:
|
||||||
|
item = Source(title=doc.title, url=doc.url)
|
||||||
|
item.doc_type = {
|
||||||
|
"cpg": "Compliance Program Guidance",
|
||||||
|
"sfa": "Special Fraud Alert",
|
||||||
|
"sab": "Special Advisory Bulletin",
|
||||||
|
"ea": "Enforcement Alert",
|
||||||
|
"open_letter": "Open Letter",
|
||||||
|
"other": "OIG Guidance",
|
||||||
|
}.get(doc.guidance_type, "OIG Guidance")
|
||||||
|
item.institution = (
|
||||||
|
"U.S. Department of Health and Human Services, Office of Inspector General"
|
||||||
|
)
|
||||||
|
item.add_tag("agency:oig")
|
||||||
|
item.add_tag(f"guidance:{doc.guidance_type}")
|
||||||
|
if doc.sector:
|
||||||
|
item.add_tag(f"sector:{doc.sector}")
|
||||||
|
item.add_tag(Tag.source("oig").label)
|
||||||
|
return item
|
||||||
|
|
||||||
|
|
||||||
|
def ingest_all(
|
||||||
|
store: Store,
|
||||||
|
client: httpx.Client,
|
||||||
|
*,
|
||||||
|
kinds: Iterable[str] | None = None,
|
||||||
|
) -> dict[str, int]:
|
||||||
|
"""Crawl OIG CPG + alerts indexes and upsert every document.
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
kinds : iterable, optional
|
||||||
|
Restrict to a subset of sources. Accepted values: ``"cpg"``,
|
||||||
|
``"alerts"``. Defaults to both.
|
||||||
|
|
||||||
|
Returns
|
||||||
|
-------
|
||||||
|
dict
|
||||||
|
``{"cpg": n, "alerts": n}`` counts of upserted records.
|
||||||
|
"""
|
||||||
|
wanted = set(kinds) if kinds else {"cpg", "alerts"}
|
||||||
|
summary = {"cpg": 0, "alerts": 0}
|
||||||
|
|
||||||
|
if "cpg" in wanted:
|
||||||
|
for doc in fetch_cpgs(client):
|
||||||
|
store.upsert(_build_item(doc))
|
||||||
|
summary["cpg"] += 1
|
||||||
|
log.info("OIG CPG: %d docs", summary["cpg"])
|
||||||
|
|
||||||
|
if "alerts" in wanted:
|
||||||
|
for doc in fetch_alerts(client):
|
||||||
|
store.upsert(_build_item(doc))
|
||||||
|
summary["alerts"] += 1
|
||||||
|
log.info("OIG alerts: %d docs", summary["alerts"])
|
||||||
|
|
||||||
|
return summary
|
||||||
|
|
||||||
|
|
||||||
|
# ── Attachments (PDF/HTML) ──────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def _sha256(path: Path) -> str:
|
||||||
|
h = hashlib.sha256()
|
||||||
|
with open(path, "rb") as f:
|
||||||
|
for chunk in iter(lambda: f.read(1 << 20), b""):
|
||||||
|
h.update(chunk)
|
||||||
|
return h.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _existing_attachment_hash(store: Store, item_key: str) -> str | None:
|
||||||
|
con = store._con() # noqa: SLF001
|
||||||
|
row = con.execute(
|
||||||
|
"""SELECT a.storage_path FROM attachments a
|
||||||
|
JOIN items i ON a.item_id = i.id
|
||||||
|
WHERE i.key = ? ORDER BY a.id LIMIT 1""",
|
||||||
|
(item_key,),
|
||||||
|
).fetchone()
|
||||||
|
if not row or not row["storage_path"]:
|
||||||
|
return None
|
||||||
|
p = Path(row["storage_path"])
|
||||||
|
return _sha256(p) if p.is_file() else None
|
||||||
|
|
||||||
|
|
||||||
|
def download_attachments(
|
||||||
|
store: Store,
|
||||||
|
client: httpx.Client,
|
||||||
|
*,
|
||||||
|
force: bool = False,
|
||||||
|
) -> int:
|
||||||
|
"""Download and attach every OIG document we've ingested.
|
||||||
|
|
||||||
|
Idempotent on SHA-256. Returns count of new/changed attachments.
|
||||||
|
"""
|
||||||
|
tmp = Path(store._db_path).parent / "tmp-oig" # noqa: SLF001
|
||||||
|
tmp.mkdir(parents=True, exist_ok=True)
|
||||||
|
items = store.list_items(tag="agency:oig")
|
||||||
|
changed = 0
|
||||||
|
for item in items:
|
||||||
|
url = item.url
|
||||||
|
if not url:
|
||||||
|
continue
|
||||||
|
filename = url.rsplit("/", 1)[-1]
|
||||||
|
tmp_path = tmp / filename
|
||||||
|
try:
|
||||||
|
r = client.get(url, follow_redirects=True, timeout=60)
|
||||||
|
r.raise_for_status()
|
||||||
|
tmp_path.write_bytes(r.content)
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
log.warning("download failed: %s — %s", url, e)
|
||||||
|
continue
|
||||||
|
|
||||||
|
remote_hash = _sha256(tmp_path)
|
||||||
|
if not force and _existing_attachment_hash(store, item.key) == remote_hash:
|
||||||
|
tmp_path.unlink(missing_ok=True)
|
||||||
|
continue
|
||||||
|
|
||||||
|
store.attach_file(item.key, tmp_path, title=item.title)
|
||||||
|
tmp_path.unlink(missing_ok=True)
|
||||||
|
changed += 1
|
||||||
|
log.info("attached: %s", filename)
|
||||||
|
return changed
|
||||||
547
src/bib/regulations_gov.py
Normal file
@@ -0,0 +1,547 @@
|
|||||||
|
"""regulations.gov v4 API client — public comments + attachments.
|
||||||
|
|
||||||
|
Walks every comment posted to a docket and (optionally) fetches each
|
||||||
|
comment's attached files (PDF / DOCX). Results are upserted to the bib
|
||||||
|
store as ``Source`` records tagged ``source:regulations-gov``,
|
||||||
|
``doctype:comment``, ``docket:<id>``, and ``rule:<cms-id>``.
|
||||||
|
|
||||||
|
API docs: https://open.gsa.gov/api/regulationsgov/
|
||||||
|
Rate limit: 1000 req/hr, 50 req/min. Keep ``--sleep`` above 1.25s for
|
||||||
|
safety; bursty short-runs are fine.
|
||||||
|
|
||||||
|
Reference endpoints::
|
||||||
|
|
||||||
|
GET /v4/documents?filter[docketId]=CMS-1676-P
|
||||||
|
GET /v4/comments?filter[commentOnId]=<documentId>&page[size]=250
|
||||||
|
GET /v4/comments/{id}?include=attachments
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import TYPE_CHECKING, Iterator
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
from bib.item import Source
|
||||||
|
from bib.tag import Tag
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from bib.store import Store
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
_BASE = "https://api.regulations.gov/v4"
|
||||||
|
|
||||||
|
|
||||||
|
# ── Value objects ──────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Comment:
|
||||||
|
"""One row returned by /v4/comments — flattened."""
|
||||||
|
|
||||||
|
id: str
|
||||||
|
title: str
|
||||||
|
posted_date: str
|
||||||
|
received_date: str
|
||||||
|
docket_id: str
|
||||||
|
comment_on_id: str # FR document this comment replies to
|
||||||
|
first_name: str = ""
|
||||||
|
last_name: str = ""
|
||||||
|
organization: str = ""
|
||||||
|
comment_text: str = "" # inline text; large comments go to attachments
|
||||||
|
attachment_count: int = 0
|
||||||
|
raw: dict = field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Attachment:
|
||||||
|
url: str
|
||||||
|
filename: str
|
||||||
|
content_type: str = ""
|
||||||
|
size: int | None = None
|
||||||
|
|
||||||
|
|
||||||
|
# ── Client ─────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
class Client:
|
||||||
|
"""Thin httpx wrapper honoring the API key + burst rate limit."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
api_key: str | None = None,
|
||||||
|
*,
|
||||||
|
sleep: float = 1.3, # ~46 req/min — under the 50/min burst cap
|
||||||
|
client: httpx.Client | None = None,
|
||||||
|
) -> None:
|
||||||
|
key = api_key or os.environ.get("REGULATIONS_GOV_API_KEY")
|
||||||
|
if not key:
|
||||||
|
raise RuntimeError(
|
||||||
|
"REGULATIONS_GOV_API_KEY not set — stack/.env should carry it."
|
||||||
|
)
|
||||||
|
self._key = key
|
||||||
|
self._sleep = sleep
|
||||||
|
self._owned = client is None
|
||||||
|
self._client = client or httpx.Client(
|
||||||
|
timeout=30,
|
||||||
|
headers={"X-Api-Key": key, "User-Agent": "stack-bib/1.0"},
|
||||||
|
)
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
if self._owned:
|
||||||
|
self._client.close()
|
||||||
|
|
||||||
|
def __enter__(self): # noqa: D401
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *exc) -> None:
|
||||||
|
self.close()
|
||||||
|
|
||||||
|
def _get(self, path: str, **params) -> dict:
|
||||||
|
time.sleep(self._sleep)
|
||||||
|
r = self._client.get(f"{_BASE}{path}", params=params)
|
||||||
|
# Back off hard on rate-limit 429s; retry once.
|
||||||
|
if r.status_code == 429:
|
||||||
|
log.warning("rate limited — sleeping 60s and retrying once")
|
||||||
|
time.sleep(60)
|
||||||
|
r = self._client.get(f"{_BASE}{path}", params=params)
|
||||||
|
r.raise_for_status()
|
||||||
|
return r.json()
|
||||||
|
|
||||||
|
# ── Discovery ──────────────────────────────────────────────
|
||||||
|
|
||||||
|
def resolve_docket(self, cms_rule_id: str) -> str | None:
|
||||||
|
"""Map a CMS rule ID (e.g. ``CMS-1676-P``) to its regulations.gov
|
||||||
|
docket (e.g. ``CMS-2017-0092``).
|
||||||
|
|
||||||
|
Federal Register's ``docket_ids`` field returns the CMS rule
|
||||||
|
identifier — not the reg.gov tracking number. Reg.gov's
|
||||||
|
``/documents`` search with the CMS rule ID as ``searchTerm``
|
||||||
|
surfaces the matching document, whose ``attributes.docketId``
|
||||||
|
is the real docket we need for comment pulls.
|
||||||
|
|
||||||
|
Returns None when no document matches — happens for rules that
|
||||||
|
were never opened for public comment (corrections, IFCs).
|
||||||
|
"""
|
||||||
|
data = self._get(
|
||||||
|
"/documents",
|
||||||
|
**{
|
||||||
|
"filter[searchTerm]": cms_rule_id,
|
||||||
|
"filter[agencyId]": "CMS",
|
||||||
|
"page[size]": 5,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
for row in data.get("data", []):
|
||||||
|
docket = (row.get("attributes") or {}).get("docketId")
|
||||||
|
if docket:
|
||||||
|
return docket
|
||||||
|
return None
|
||||||
|
|
||||||
|
def find_documents_in_docket(self, docket_id: str) -> list[dict]:
|
||||||
|
"""Return the FR documents the API indexes under *docket_id*.
|
||||||
|
|
||||||
|
Each document's ``id`` (the API's internal ID, not the FR doc
|
||||||
|
number) is the ``commentOnId`` you need to list its comments.
|
||||||
|
"""
|
||||||
|
out: list[dict] = []
|
||||||
|
page = 1
|
||||||
|
while True:
|
||||||
|
data = self._get(
|
||||||
|
"/documents",
|
||||||
|
**{
|
||||||
|
"filter[docketId]": docket_id,
|
||||||
|
"page[size]": 250,
|
||||||
|
"page[number]": page,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
out.extend(data.get("data", []))
|
||||||
|
meta = data.get("meta", {})
|
||||||
|
if page >= meta.get("totalPages", 1):
|
||||||
|
break
|
||||||
|
page += 1
|
||||||
|
return out
|
||||||
|
|
||||||
|
# ── Comment iteration ──────────────────────────────────────
|
||||||
|
|
||||||
|
def iter_comments(self, object_id: str) -> Iterator[Comment]:
|
||||||
|
"""Yield every comment against a single FR document.
|
||||||
|
|
||||||
|
The ``commentOnId`` filter on ``/comments`` takes a document's
|
||||||
|
**objectId**, not its public ``id`` — the field lives under
|
||||||
|
``attributes.objectId`` and looks like ``0900006482921ba1``.
|
||||||
|
Getting this wrong silently returns zero results.
|
||||||
|
|
||||||
|
Two pagination quirks we've been burned by:
|
||||||
|
|
||||||
|
1. The ``page[number]`` counter caps at 20 (≈5K items at page
|
||||||
|
size 250). Dockets with more comments need a date-based
|
||||||
|
cursor via ``filter[lastModifiedDate][ge]``.
|
||||||
|
2. That filter rejects ISO-8601 timestamps with ``T``/``Z`` —
|
||||||
|
returns 400. It wants ``YYYY-MM-DD HH:MM:SS`` (space-
|
||||||
|
separated, no timezone suffix). Normalize before sending.
|
||||||
|
"""
|
||||||
|
cursor: str | None = None
|
||||||
|
page = 1
|
||||||
|
while True:
|
||||||
|
params: dict[str, str] = {
|
||||||
|
"filter[commentOnId]": object_id,
|
||||||
|
"page[size]": 250,
|
||||||
|
"page[number]": page,
|
||||||
|
"sort": "lastModifiedDate,documentId",
|
||||||
|
}
|
||||||
|
if cursor:
|
||||||
|
params["filter[lastModifiedDate][ge]"] = _reg_date(cursor)
|
||||||
|
try:
|
||||||
|
data = self._get("/comments", **params)
|
||||||
|
except httpx.HTTPStatusError as e:
|
||||||
|
log.warning(
|
||||||
|
"iter_comments: %s on object %s page %d cursor %r — stopping this object",
|
||||||
|
e.response.status_code,
|
||||||
|
object_id,
|
||||||
|
page,
|
||||||
|
cursor,
|
||||||
|
)
|
||||||
|
break
|
||||||
|
rows = data.get("data", [])
|
||||||
|
if not rows:
|
||||||
|
break
|
||||||
|
for row in rows:
|
||||||
|
yield _parse_comment(row)
|
||||||
|
if len(rows) < 250:
|
||||||
|
break
|
||||||
|
meta = data.get("meta", {})
|
||||||
|
total_pages = meta.get("totalPages") or 1
|
||||||
|
if page < total_pages:
|
||||||
|
page += 1
|
||||||
|
continue
|
||||||
|
# Ran out of pages — advance the date cursor past the last
|
||||||
|
# row we've seen and restart pagination from page 1.
|
||||||
|
last_mod = rows[-1].get("attributes", {}).get("lastModifiedDate")
|
||||||
|
if not last_mod:
|
||||||
|
break
|
||||||
|
new_cursor = _reg_date(last_mod)
|
||||||
|
if new_cursor == cursor:
|
||||||
|
break
|
||||||
|
cursor = new_cursor
|
||||||
|
page = 1
|
||||||
|
|
||||||
|
def get_comment_detail(self, comment_id: str) -> dict:
|
||||||
|
"""Full comment including attachments relationship."""
|
||||||
|
return self._get(f"/comments/{comment_id}", **{"include": "attachments"})
|
||||||
|
|
||||||
|
# ── Attachments ────────────────────────────────────────────
|
||||||
|
|
||||||
|
def attachments_for(self, comment_id: str) -> list[Attachment]:
|
||||||
|
data = self.get_comment_detail(comment_id)
|
||||||
|
attachments: list[Attachment] = []
|
||||||
|
for inc in data.get("included", []) or []:
|
||||||
|
if inc.get("type") != "attachments":
|
||||||
|
continue
|
||||||
|
attrs = inc.get("attributes", {})
|
||||||
|
for f in attrs.get("fileFormats") or []:
|
||||||
|
attachments.append(
|
||||||
|
Attachment(
|
||||||
|
url=f.get("fileUrl", ""),
|
||||||
|
filename=_filename_from(f.get("fileUrl", "")),
|
||||||
|
content_type=f.get("format", ""),
|
||||||
|
size=f.get("size"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return attachments
|
||||||
|
|
||||||
|
# Attachment CDN lives on downloads.regulations.gov and rejects the
|
||||||
|
# API-key header used by the v4 API host. A browser-shaped UA +
|
||||||
|
# Referer is what it expects; otherwise every GET returns 403 with
|
||||||
|
# an HTML error page.
|
||||||
|
_DL_HEADERS = {
|
||||||
|
"User-Agent": (
|
||||||
|
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
|
||||||
|
"(KHTML, like Gecko) Firefox/122.0"
|
||||||
|
),
|
||||||
|
"Accept": "application/pdf,application/octet-stream,*/*;q=0.8",
|
||||||
|
"Accept-Language": "en-US,en;q=0.9",
|
||||||
|
"Referer": "https://www.regulations.gov/",
|
||||||
|
}
|
||||||
|
|
||||||
|
def download_attachment(
|
||||||
|
self,
|
||||||
|
url: str,
|
||||||
|
dest_dir: Path,
|
||||||
|
*,
|
||||||
|
overwrite: bool = False,
|
||||||
|
) -> Path | None:
|
||||||
|
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
filename = _filename_from(url) or _hash(url) + ".bin"
|
||||||
|
dest = dest_dir / filename
|
||||||
|
if dest.is_file() and not overwrite:
|
||||||
|
return dest
|
||||||
|
time.sleep(self._sleep)
|
||||||
|
try:
|
||||||
|
with self._client.stream(
|
||||||
|
"GET",
|
||||||
|
url,
|
||||||
|
timeout=60,
|
||||||
|
follow_redirects=True,
|
||||||
|
headers=self._DL_HEADERS,
|
||||||
|
) as r:
|
||||||
|
if r.status_code != 200:
|
||||||
|
log.warning(
|
||||||
|
"attachment %s → HTTP %s",
|
||||||
|
url.rsplit("/", 1)[-1],
|
||||||
|
r.status_code,
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
with open(dest, "wb") as f:
|
||||||
|
for chunk in r.iter_bytes(1 << 16):
|
||||||
|
f.write(chunk)
|
||||||
|
except httpx.HTTPError as e:
|
||||||
|
log.warning("attachment %s failed: %s", url, e)
|
||||||
|
return None
|
||||||
|
return dest
|
||||||
|
|
||||||
|
|
||||||
|
# ── Bib integration ────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def backfill_details(
|
||||||
|
store: Store,
|
||||||
|
client: Client,
|
||||||
|
*,
|
||||||
|
limit: int | None = None,
|
||||||
|
log_path: Path | None = None,
|
||||||
|
commit_every: int = 25,
|
||||||
|
scratch_root: Path = Path(".state/comments"),
|
||||||
|
) -> dict[str, int]:
|
||||||
|
"""Enrich every ``source:regulations-gov`` stub with body text,
|
||||||
|
organization, and any attached files.
|
||||||
|
|
||||||
|
Background: ``iter_comments`` pulls from ``/v4/comments`` which only
|
||||||
|
returns surface metadata — no body, no attachment URLs, no org
|
||||||
|
name. Full text + attachments live on the per-comment detail
|
||||||
|
endpoint, one round-trip per item. With 164K comments against
|
||||||
|
reg.gov's 1000-req/hr cap this runs for days; this helper is
|
||||||
|
structured so it can be restarted without losing progress.
|
||||||
|
|
||||||
|
Resume rule: we treat ``items.abstract`` as the "already enriched"
|
||||||
|
flag. Any reg-gov item with an empty abstract is retried on next
|
||||||
|
run. A comment whose detail returns 404 is tagged ``enriched:gone``
|
||||||
|
so we stop hammering it. Commits land every ``commit_every`` items
|
||||||
|
so a crash loses at most that many items of work.
|
||||||
|
"""
|
||||||
|
con = store._con() # noqa: SLF001
|
||||||
|
# Resume rule: treat ``enriched:ok`` as the completion marker.
|
||||||
|
# Using a tag (not just the abstract) lets us handle "see attached"
|
||||||
|
# comments with near-empty body AND still drives distinct retry
|
||||||
|
# states (``enriched:gone`` for 404s, untagged for transient fails).
|
||||||
|
rows = con.execute(
|
||||||
|
"""
|
||||||
|
SELECT i.id, i.key, i.url
|
||||||
|
FROM items i
|
||||||
|
WHERE i.id IN (
|
||||||
|
SELECT item_id FROM item_tags
|
||||||
|
WHERE tag_id IN (SELECT id FROM tags WHERE name='source:regulations-gov')
|
||||||
|
)
|
||||||
|
AND i.id NOT IN (
|
||||||
|
SELECT item_id FROM item_tags
|
||||||
|
WHERE tag_id IN (SELECT id FROM tags WHERE name IN ('enriched:ok','enriched:gone'))
|
||||||
|
)
|
||||||
|
ORDER BY i.id
|
||||||
|
"""
|
||||||
|
+ (f" LIMIT {int(limit)}" if limit else "")
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
stats = {"enriched": 0, "attached": 0, "gone": 0, "errors": 0, "seen": len(rows)}
|
||||||
|
|
||||||
|
def _write_log(msg: str) -> None:
|
||||||
|
log.info(msg)
|
||||||
|
print(msg, flush=True)
|
||||||
|
if log_path:
|
||||||
|
log_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
with open(log_path, "a") as f:
|
||||||
|
f.write(msg + "\n")
|
||||||
|
|
||||||
|
_write_log(f"backfill start: {len(rows)} items pending")
|
||||||
|
|
||||||
|
for i, row in enumerate(rows, 1):
|
||||||
|
iid, key, url = row["id"], row["key"], row["url"]
|
||||||
|
cid = (url or "").rsplit("/", 1)[-1]
|
||||||
|
if not cid:
|
||||||
|
stats["errors"] += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
data = client.get_comment_detail(cid)
|
||||||
|
except httpx.HTTPStatusError as e:
|
||||||
|
if e.response.status_code == 404:
|
||||||
|
try:
|
||||||
|
store.add_tag(key, "enriched:gone")
|
||||||
|
stats["gone"] += 1
|
||||||
|
except Exception as add_err: # noqa: BLE001
|
||||||
|
log.warning("add_tag enriched:gone failed for %s: %s", key, add_err)
|
||||||
|
stats["errors"] += 1
|
||||||
|
else:
|
||||||
|
log.warning("HTTP %s on %s — skipping", e.response.status_code, cid)
|
||||||
|
stats["errors"] += 1
|
||||||
|
continue
|
||||||
|
except httpx.HTTPError as e:
|
||||||
|
log.warning("transport error on %s: %s", cid, e)
|
||||||
|
stats["errors"] += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
attrs = (data.get("data") or {}).get("attributes") or {}
|
||||||
|
body = (attrs.get("comment") or "")[:4000]
|
||||||
|
org = (attrs.get("organization") or "").strip()
|
||||||
|
|
||||||
|
if body:
|
||||||
|
con.execute("UPDATE items SET abstract=? WHERE id=?", (body, iid))
|
||||||
|
if org:
|
||||||
|
try:
|
||||||
|
store.add_tag(key, f"org:{_slug(org)}")
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
log.warning("add_tag org failed for %s: %s", key, e)
|
||||||
|
|
||||||
|
# Download any attached files into per-comment dir and register.
|
||||||
|
docket = _docket_from_comment_id(cid)
|
||||||
|
dest_dir = scratch_root / docket / cid
|
||||||
|
for inc in data.get("included") or []:
|
||||||
|
if inc.get("type") != "attachments":
|
||||||
|
continue
|
||||||
|
for f in inc.get("attributes", {}).get("fileFormats") or []:
|
||||||
|
att_url = f.get("fileUrl") or ""
|
||||||
|
if not att_url:
|
||||||
|
continue
|
||||||
|
path = client.download_attachment(att_url, dest_dir)
|
||||||
|
if not path:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
store.attach_file(key, path, title=path.name)
|
||||||
|
stats["attached"] += 1
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
# Usually a dup filename; fine to skip.
|
||||||
|
log.debug("attach_file skipped for %s/%s: %s", key, path.name, e)
|
||||||
|
|
||||||
|
try:
|
||||||
|
store.add_tag(key, "enriched:ok")
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
log.warning("add_tag enriched:ok failed for %s: %s", key, e)
|
||||||
|
stats["enriched"] += 1
|
||||||
|
if i % commit_every == 0:
|
||||||
|
con.commit()
|
||||||
|
_write_log(
|
||||||
|
f" {i}/{len(rows)} "
|
||||||
|
f"enriched={stats['enriched']} "
|
||||||
|
f"attached={stats['attached']} "
|
||||||
|
f"gone={stats['gone']} errors={stats['errors']}"
|
||||||
|
)
|
||||||
|
|
||||||
|
con.commit()
|
||||||
|
_write_log(f"backfill done: {stats}")
|
||||||
|
return stats
|
||||||
|
|
||||||
|
|
||||||
|
def _docket_from_comment_id(cid: str) -> str:
|
||||||
|
"""``CMS-2025-0304-14107`` → ``CMS-2025-0304``; safe fallback to cid."""
|
||||||
|
parts = cid.split("-")
|
||||||
|
return "-".join(parts[:3]) if len(parts) >= 3 else cid
|
||||||
|
|
||||||
|
|
||||||
|
def upsert_comment(
|
||||||
|
store: Store,
|
||||||
|
comment: Comment,
|
||||||
|
*,
|
||||||
|
cms_id: str = "",
|
||||||
|
extra_tags: list[str] | None = None,
|
||||||
|
) -> str:
|
||||||
|
"""Upsert the comment as a Source item, return bib key."""
|
||||||
|
title = comment.title or f"Comment on {comment.comment_on_id}"
|
||||||
|
byline = _byline(comment)
|
||||||
|
if byline:
|
||||||
|
title = f"{byline}: {title[:120]}"
|
||||||
|
|
||||||
|
url = f"https://www.regulations.gov/comment/{comment.id}"
|
||||||
|
item = Source(title=title, url=url)
|
||||||
|
item.doc_type = "Public Comment"
|
||||||
|
item.institution = "U.S. Government — regulations.gov"
|
||||||
|
item.date_published = comment.posted_date or comment.received_date
|
||||||
|
item.abstract = (comment.comment_text or "")[:4000]
|
||||||
|
|
||||||
|
tags = [
|
||||||
|
Tag.source("regulations-gov").label,
|
||||||
|
"doctype:comment",
|
||||||
|
f"docket:{comment.docket_id}",
|
||||||
|
]
|
||||||
|
if cms_id:
|
||||||
|
tags.append(f"rule:{cms_id}")
|
||||||
|
if comment.posted_date:
|
||||||
|
tags.append(f"year:{comment.posted_date[:4]}")
|
||||||
|
if comment.organization:
|
||||||
|
tags.append(f"org:{_slug(comment.organization)}")
|
||||||
|
for t in extra_tags or []:
|
||||||
|
tags.append(t)
|
||||||
|
for t in tags:
|
||||||
|
item.add_tag(t)
|
||||||
|
|
||||||
|
return store.upsert(item)
|
||||||
|
|
||||||
|
|
||||||
|
# ── Internals ──────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_comment(row: dict) -> Comment:
|
||||||
|
attrs = row.get("attributes", {})
|
||||||
|
return Comment(
|
||||||
|
id=row.get("id", ""),
|
||||||
|
title=attrs.get("title", ""),
|
||||||
|
posted_date=(attrs.get("postedDate") or "")[:10],
|
||||||
|
received_date=(attrs.get("receivedDate") or "")[:10],
|
||||||
|
docket_id=attrs.get("docketId", ""),
|
||||||
|
comment_on_id=attrs.get("commentOnId", ""),
|
||||||
|
first_name=attrs.get("firstName") or "",
|
||||||
|
last_name=attrs.get("lastName") or "",
|
||||||
|
organization=attrs.get("organization") or "",
|
||||||
|
comment_text=attrs.get("comment") or "",
|
||||||
|
attachment_count=(attrs.get("attachmentCount") or 0),
|
||||||
|
raw=row,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _byline(c: Comment) -> str:
|
||||||
|
if c.organization:
|
||||||
|
return c.organization
|
||||||
|
name = " ".join(p for p in (c.first_name, c.last_name) if p).strip()
|
||||||
|
return name
|
||||||
|
|
||||||
|
|
||||||
|
def _slug(s: str) -> str:
|
||||||
|
import re
|
||||||
|
|
||||||
|
s = (s or "").lower()
|
||||||
|
s = re.sub(r"[^a-z0-9]+", "-", s)
|
||||||
|
return s.strip("-")[:40]
|
||||||
|
|
||||||
|
|
||||||
|
def _filename_from(url: str) -> str:
|
||||||
|
return url.rsplit("/", 1)[-1].split("?")[0] if url else ""
|
||||||
|
|
||||||
|
|
||||||
|
def _hash(s: str) -> str:
|
||||||
|
return hashlib.sha1(s.encode()).hexdigest()[:10] # noqa: S324
|
||||||
|
|
||||||
|
|
||||||
|
def _reg_date(iso_ts: str) -> str:
|
||||||
|
"""Convert an ISO-8601 timestamp (what the API returns in payloads)
|
||||||
|
into the ``YYYY-MM-DD HH:MM:SS`` form the filter operators require.
|
||||||
|
|
||||||
|
``2017-08-30T18:35:48Z`` → ``2017-08-30 18:35:48``. Idempotent: if
|
||||||
|
the input is already space-separated we leave it alone.
|
||||||
|
"""
|
||||||
|
if not iso_ts:
|
||||||
|
return iso_ts
|
||||||
|
return iso_ts.replace("T", " ").rstrip("Z").strip()
|
||||||
207
src/bib/sync.py
@@ -11,26 +11,37 @@ Usage::
|
|||||||
|
|
||||||
s = Store()
|
s = Store()
|
||||||
items = s.list_items(tag="source:spider")
|
items = s.list_items(tag="source:spider")
|
||||||
stats = push_to_zotero(items)
|
stats = push_to_zotero(items, store=s)
|
||||||
print(stats)
|
print(stats)
|
||||||
# {'created': 5908, 'skipped': 12, 'tags': 23456, 'collections': 2}
|
# {'created': 5908, 'skipped': 12, 'tags': 23456, 'attachments': 240}
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
|
import shutil
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
from bib.item import Item
|
from bib.item import Item
|
||||||
from zot.db import Db, is_valid_key, now_iso
|
from zot.db import TYPE_MAP as ZOT_TYPE_MAP
|
||||||
|
from zot.db import Db, generate_key, is_valid_key, now_iso
|
||||||
|
|
||||||
# bib item_type → Zotero itemTypeID
|
if TYPE_CHECKING:
|
||||||
|
from bib.store import Store
|
||||||
|
|
||||||
|
# bib item_type → Zotero itemType *name*. Numeric IDs come from
|
||||||
|
# zot.db.TYPE_MAP, which is the single source of truth.
|
||||||
|
_TYPE_NAMES: dict[str, str] = {
|
||||||
|
"rule": "statute",
|
||||||
|
"regulation": "statute",
|
||||||
|
"manual": "report",
|
||||||
|
"download": "webpage",
|
||||||
|
"source": "document",
|
||||||
|
"journal-article": "journalArticle",
|
||||||
|
}
|
||||||
_TYPE_MAP: dict[str, int] = {
|
_TYPE_MAP: dict[str, int] = {
|
||||||
"rule": 20, # statute
|
bib_type: ZOT_TYPE_MAP[zot_name] for bib_type, zot_name in _TYPE_NAMES.items()
|
||||||
"regulation": 20, # statute
|
|
||||||
"manual": 15, # report
|
|
||||||
"download": 13, # webpage
|
|
||||||
"source": 34, # document
|
|
||||||
"journal-article": 4, # journalArticle
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -207,7 +218,9 @@ def _item_to_zotero_fields(item: Item) -> dict[str, str]:
|
|||||||
def push_to_zotero(
|
def push_to_zotero(
|
||||||
items: list[Item],
|
items: list[Item],
|
||||||
*,
|
*,
|
||||||
|
store: Store | None = None,
|
||||||
zotero_db: str | None = None,
|
zotero_db: str | None = None,
|
||||||
|
zotero_storage: str | None = None,
|
||||||
collection_key: str = "",
|
collection_key: str = "",
|
||||||
) -> dict[str, int]:
|
) -> dict[str, int]:
|
||||||
"""Push bib items into Zotero's SQLite database.
|
"""Push bib items into Zotero's SQLite database.
|
||||||
@@ -216,20 +229,30 @@ def push_to_zotero(
|
|||||||
----------
|
----------
|
||||||
items : list[Item]
|
items : list[Item]
|
||||||
Items to sync.
|
Items to sync.
|
||||||
|
store : bib.Store, optional
|
||||||
|
Source bib store. When provided, each item's attached PDFs are
|
||||||
|
copied into Zotero's storage dir and registered as child
|
||||||
|
``attachment`` items. Without it, attachments are skipped.
|
||||||
zotero_db : str
|
zotero_db : str
|
||||||
Path to Zotero's SQLite database.
|
Path to Zotero's SQLite database.
|
||||||
|
zotero_storage : str, optional
|
||||||
|
Zotero storage directory (``data/zotero/data/storage/``).
|
||||||
|
Defaults to ``<zotero_db>/../storage``.
|
||||||
collection_key : str
|
collection_key : str
|
||||||
Optional Zotero collection key to add items to.
|
Optional Zotero collection key to add items to.
|
||||||
|
|
||||||
Returns
|
Returns
|
||||||
-------
|
-------
|
||||||
dict
|
dict
|
||||||
Counts: created, skipped, tags, collections, creators.
|
Counts: created, skipped, tags, collections, creators, attachments.
|
||||||
"""
|
"""
|
||||||
if zotero_db is None:
|
if zotero_db is None:
|
||||||
from conf import path
|
from conf import path
|
||||||
|
|
||||||
zotero_db = str(path("db.zotero"))
|
zotero_db = str(path("db.zotero"))
|
||||||
|
storage_dir = (
|
||||||
|
Path(zotero_storage) if zotero_storage else Path(zotero_db).parent / "storage"
|
||||||
|
)
|
||||||
|
|
||||||
with Db(zotero_db) as db:
|
with Db(zotero_db) as db:
|
||||||
stats: dict[str, int] = {
|
stats: dict[str, int] = {
|
||||||
@@ -237,8 +260,23 @@ def push_to_zotero(
|
|||||||
"skipped": 0,
|
"skipped": 0,
|
||||||
"tags": 0,
|
"tags": 0,
|
||||||
"collections": 0,
|
"collections": 0,
|
||||||
|
"attachments": 0,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Resolve a tuple-path to a Zotero collection key, ensuring each
|
||||||
|
# level exists. Cached across items so we only touch the DB once
|
||||||
|
# per unique path per sync run.
|
||||||
|
path_cache: dict[tuple[str, ...], str] = {}
|
||||||
|
|
||||||
|
def _resolve_path(path: tuple[str, ...]) -> str:
|
||||||
|
if path in path_cache:
|
||||||
|
return path_cache[path]
|
||||||
|
parent = ""
|
||||||
|
for name in path:
|
||||||
|
parent = db.ensure_collection(name, parent_key=parent)
|
||||||
|
path_cache[path] = parent
|
||||||
|
return parent
|
||||||
|
|
||||||
ts = now_iso()
|
ts = now_iso()
|
||||||
|
|
||||||
# Resolve target collection
|
# Resolve target collection
|
||||||
@@ -247,11 +285,33 @@ def push_to_zotero(
|
|||||||
collection_id = db.find_collection(collection_key)
|
collection_id = db.find_collection(collection_key)
|
||||||
|
|
||||||
for item in items:
|
for item in items:
|
||||||
# Skip if URL already exists in Zotero
|
# Skip (update-in-place) if URL already exists in Zotero
|
||||||
if item.url:
|
if item.url:
|
||||||
existing_id = db.find_item_by_url(item.url)
|
existing_id = db.find_item_by_url(item.url)
|
||||||
if existing_id is not None:
|
if existing_id is not None:
|
||||||
|
# Refresh fields from scratch — earlier syncs used
|
||||||
|
# a stale FIELD_MAP that landed values in wrongly-
|
||||||
|
# named columns (title→sessionTitle, seriesTitle→
|
||||||
|
# ISBN, etc.). replace_fields clears the item's
|
||||||
|
# entire itemData row set before rewriting with the
|
||||||
|
# corrected map.
|
||||||
|
db.replace_fields(existing_id, _item_to_zotero_fields(item))
|
||||||
db.sync_tags(existing_id, item.tags)
|
db.sync_tags(existing_id, item.tags)
|
||||||
|
if store is not None:
|
||||||
|
stats["attachments"] += _sync_attachments(
|
||||||
|
db,
|
||||||
|
store,
|
||||||
|
item,
|
||||||
|
existing_id,
|
||||||
|
storage_dir,
|
||||||
|
)
|
||||||
|
# Backfill collection membership for items that were
|
||||||
|
# created before sync learned about the hierarchy.
|
||||||
|
path = _zotero_collection_path(item)
|
||||||
|
if path:
|
||||||
|
key = _resolve_path(tuple(path))
|
||||||
|
if db.add_to_collection(existing_id, collection_key=key):
|
||||||
|
stats["collections"] += 1
|
||||||
stats["skipped"] += 1
|
stats["skipped"] += 1
|
||||||
continue
|
continue
|
||||||
|
|
||||||
@@ -287,13 +347,134 @@ def push_to_zotero(
|
|||||||
db.add_to_collection(item_id, collection_key=collection_key)
|
db.add_to_collection(item_id, collection_key=collection_key)
|
||||||
stats["collections"] += 1
|
stats["collections"] += 1
|
||||||
|
|
||||||
# Item's own collections
|
# Item's own collections (legacy: bib keys; usually no hits
|
||||||
|
# since bib/Zotero live in separate key spaces, but safe).
|
||||||
for col_key in item.collections:
|
for col_key in item.collections:
|
||||||
if col_key != collection_key:
|
if col_key != collection_key:
|
||||||
db.add_to_collection(item_id, collection_key=col_key)
|
db.add_to_collection(item_id, collection_key=col_key)
|
||||||
|
|
||||||
|
# Derived collection path (Manual → Healthcare Data Platform
|
||||||
|
# / Manuals / <name>; OIG → …/OIG Guidance/<type>, etc.).
|
||||||
|
path = _zotero_collection_path(item)
|
||||||
|
if path:
|
||||||
|
key = _resolve_path(tuple(path))
|
||||||
|
if db.add_to_collection(item_id, collection_key=key):
|
||||||
|
stats["collections"] += 1
|
||||||
|
|
||||||
|
# Attachments (PDFs etc.) — copy from bib storage into Zotero
|
||||||
|
# storage and register as child items.
|
||||||
|
if store is not None:
|
||||||
|
stats["attachments"] += _sync_attachments(
|
||||||
|
db,
|
||||||
|
store,
|
||||||
|
item,
|
||||||
|
item_id,
|
||||||
|
storage_dir,
|
||||||
|
)
|
||||||
|
|
||||||
stats["created"] += 1
|
stats["created"] += 1
|
||||||
|
|
||||||
db.commit()
|
db.commit()
|
||||||
|
|
||||||
return stats
|
return stats
|
||||||
|
|
||||||
|
|
||||||
|
def _zotero_collection_path(item: Item) -> list[str]:
|
||||||
|
"""Derive the Zotero collection hierarchy an item should live under.
|
||||||
|
|
||||||
|
Returns a path of human-readable names (``ensure_collection`` will
|
||||||
|
create missing levels on the fly). An empty list means no routing —
|
||||||
|
the item lands at the top of "My Library" only.
|
||||||
|
"""
|
||||||
|
ej = json.loads(item.to_row().get("extra_json", "{}"))
|
||||||
|
if item.item_type == "manual":
|
||||||
|
manual = (ej.get("manual_name") or "").strip() or "General"
|
||||||
|
return ["Healthcare Data Platform", "Manuals", manual]
|
||||||
|
if "source:regulations-gov" in item.tags:
|
||||||
|
# Every rulemaking comment lands flat in Rules/Comments. The
|
||||||
|
# rule:<id> and year:<Y> tags on each item carry the
|
||||||
|
# disambiguating citation, so a single bucket is readable.
|
||||||
|
return ["Rules", "Comments"]
|
||||||
|
if "source:email" in item.tags:
|
||||||
|
# IMAP-ingested email lands under Inbox/<Mailbox-Title>.
|
||||||
|
# Each mailbox tag (mailbox:cmsupdates) becomes its own
|
||||||
|
# subfolder so multi-mailbox setups don't intermingle.
|
||||||
|
for t in item.tags:
|
||||||
|
if t.startswith("mailbox:"):
|
||||||
|
label = t.split(":", 1)[1].replace("-", " ").title()
|
||||||
|
return ["Inbox", label]
|
||||||
|
return ["Inbox"]
|
||||||
|
if "agency:oig" in item.tags:
|
||||||
|
sector = ""
|
||||||
|
guidance = ""
|
||||||
|
for t in item.tags:
|
||||||
|
if t.startswith("sector:"):
|
||||||
|
sector = t.split(":", 1)[1].replace("_", " ").title()
|
||||||
|
elif t.startswith("guidance:"):
|
||||||
|
guidance = t.split(":", 1)[1].upper()
|
||||||
|
base = ["Healthcare Data Platform", "OIG Guidance"]
|
||||||
|
# Prefer sector when known; fall back to guidance type.
|
||||||
|
if sector:
|
||||||
|
return [*base, sector]
|
||||||
|
if guidance:
|
||||||
|
return [*base, guidance]
|
||||||
|
return base
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def _sync_attachments(
|
||||||
|
db: Db,
|
||||||
|
store: Store,
|
||||||
|
bib_item: Item,
|
||||||
|
zot_parent_id: int,
|
||||||
|
zot_storage: Path,
|
||||||
|
) -> int:
|
||||||
|
"""Copy every bib attachment for ``bib_item`` into Zotero's storage
|
||||||
|
and create the matching ``itemAttachments`` row. Idempotent: an
|
||||||
|
existing same-filename child attachment is left alone."""
|
||||||
|
con = store._con() # noqa: SLF001
|
||||||
|
rows = con.execute(
|
||||||
|
"""SELECT a.filename, a.content_type, a.storage_path
|
||||||
|
FROM attachments a
|
||||||
|
JOIN items i ON a.item_id = i.id
|
||||||
|
WHERE i.key = ?""",
|
||||||
|
(bib_item.key,),
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
if not rows:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
# Existing child attachments keyed by path so we don't duplicate.
|
||||||
|
existing = {
|
||||||
|
r[0]
|
||||||
|
for r in db.con.execute(
|
||||||
|
"SELECT ia.path FROM itemAttachments ia WHERE ia.parentItemID = ?",
|
||||||
|
(zot_parent_id,),
|
||||||
|
).fetchall()
|
||||||
|
if r[0]
|
||||||
|
}
|
||||||
|
|
||||||
|
count = 0
|
||||||
|
for row in rows:
|
||||||
|
src = Path(row["storage_path"])
|
||||||
|
if not src.is_file():
|
||||||
|
continue
|
||||||
|
# Bib's `filename` column is a display label, not a filesystem
|
||||||
|
# name; the actual PDF filename is the basename of storage_path.
|
||||||
|
disk_name = src.name
|
||||||
|
zot_path = f"storage:{disk_name}"
|
||||||
|
if zot_path in existing:
|
||||||
|
continue
|
||||||
|
att_key = generate_key()
|
||||||
|
dest_dir = zot_storage / att_key
|
||||||
|
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
shutil.copy2(src, dest_dir / disk_name)
|
||||||
|
db.add_attachment(
|
||||||
|
zot_parent_id,
|
||||||
|
link_mode=1, # Zotero.Attachments.LINK_MODE_IMPORTED_FILE
|
||||||
|
content_type=row["content_type"] or "application/pdf",
|
||||||
|
path=zot_path,
|
||||||
|
key=att_key,
|
||||||
|
)
|
||||||
|
count += 1
|
||||||
|
return count
|
||||||
|
|||||||
@@ -146,7 +146,7 @@ def federal_register(url: str) -> Rule:
|
|||||||
cms_id=cms_id,
|
cms_id=cms_id,
|
||||||
rule_type=rule_type,
|
rule_type=rule_type,
|
||||||
date_published=data.get("publication_date", ""),
|
date_published=data.get("publication_date", ""),
|
||||||
effective_date=data.get("effective_on", ""),
|
effective_date=data.get("effective_on") or "",
|
||||||
url=data.get("html_url", url),
|
url=data.get("html_url", url),
|
||||||
abstract=data.get("abstract", ""),
|
abstract=data.get("abstract", ""),
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -20,10 +20,13 @@ from cli.generate import app as generate_app
|
|||||||
from cli.health import health
|
from cli.health import health
|
||||||
from cli.lake import app as lake_app
|
from cli.lake import app as lake_app
|
||||||
from cli.load import app as load_app
|
from cli.load import app as load_app
|
||||||
|
from cli.mail import app as mail_app
|
||||||
from cli.perf import app as perf_app
|
from cli.perf import app as perf_app
|
||||||
|
from cli.prisma import app as prisma_app
|
||||||
from cli.rec import app as rec_app
|
from cli.rec import app as rec_app
|
||||||
from cli.run import run
|
from cli.run import run
|
||||||
from cli.validate import validate
|
from cli.validate import validate
|
||||||
|
from cli.zot import app as zot_app
|
||||||
|
|
||||||
app = typer.Typer(
|
app = typer.Typer(
|
||||||
name="stack",
|
name="stack",
|
||||||
@@ -47,6 +50,9 @@ app.add_typer(
|
|||||||
name="rec",
|
name="rec",
|
||||||
help="Reconcile calculated payments against CMS ground-truth files.",
|
help="Reconcile calculated payments against CMS ground-truth files.",
|
||||||
)
|
)
|
||||||
|
app.add_typer(zot_app, name="zot", help="Zotero database maintenance.")
|
||||||
|
app.add_typer(prisma_app, name="prisma", help="LLM-driven PRISMA screening workflow.")
|
||||||
|
app.add_typer(mail_app, name="mail", help="Maddy mail server (DO + Resend smarthost).")
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
|
|||||||
558
src/cli/bib.py
@@ -2,6 +2,8 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
import typer
|
import typer
|
||||||
|
|
||||||
app = typer.Typer(no_args_is_help=True)
|
app = typer.Typer(no_args_is_help=True)
|
||||||
@@ -71,3 +73,559 @@ def query(
|
|||||||
if len(items) > 20:
|
if len(items) > 20:
|
||||||
typer.echo(f" ... and {len(items) - 20} more")
|
typer.echo(f" ... and {len(items) - 20} more")
|
||||||
typer.echo(f"{len(items)} items found.")
|
typer.echo(f"{len(items)} items found.")
|
||||||
|
|
||||||
|
|
||||||
|
_UA = "fhirworx-bib/0.1"
|
||||||
|
|
||||||
|
|
||||||
|
def _http_client():
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
return httpx.Client(headers={"User-Agent": _UA})
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="discover-pfs-rules")
|
||||||
|
def discover_pfs_rules(
|
||||||
|
since: str = typer.Option("2017-01-01", "--since"),
|
||||||
|
until: str = typer.Option("", "--until"),
|
||||||
|
dry_run: bool = typer.Option(False, "--dry-run"),
|
||||||
|
) -> None:
|
||||||
|
"""Walk the Federal Register API for every PFS rule since a date.
|
||||||
|
|
||||||
|
Upserts each as a bib ``Rule`` record tagged ``module:pfs``,
|
||||||
|
``source:federal-register``, ``rule:<cms-id>``. Cheap — the FR API
|
||||||
|
has no auth and no rate charge we can trigger at this scale
|
||||||
|
(~20 rules since 2017).
|
||||||
|
"""
|
||||||
|
from bib import connect
|
||||||
|
from bib.federalregister import pfs_rules
|
||||||
|
from bib.tag import Tag
|
||||||
|
from bib.translate import federal_register
|
||||||
|
|
||||||
|
store = connect() if not dry_run else None
|
||||||
|
|
||||||
|
for doc in pfs_rules(since=since, until=until or None):
|
||||||
|
typer.echo(
|
||||||
|
f" {doc.publication_date} {doc.type:14s} "
|
||||||
|
f"{doc.document_number:14s} {','.join(doc.dockets) or '(no docket)'}"
|
||||||
|
)
|
||||||
|
if dry_run or store is None:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
rule = federal_register(
|
||||||
|
doc.html_url
|
||||||
|
or f"https://www.federalregister.gov/documents/{doc.document_number}"
|
||||||
|
)
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
typer.echo(f" skipped: {e.__class__.__name__}: {e}")
|
||||||
|
continue
|
||||||
|
rule.add_tag(Tag.source("federal-register").label)
|
||||||
|
rule.add_tag("module:pfs")
|
||||||
|
for d in doc.dockets:
|
||||||
|
rule.add_tag(f"docket:{d}")
|
||||||
|
store.upsert(rule)
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="fetch-docket-comments")
|
||||||
|
def fetch_docket_comments(
|
||||||
|
docket: str = typer.Argument(..., help="e.g. CMS-1676-P"),
|
||||||
|
cms_id: str = typer.Option(
|
||||||
|
"", "--cms-id", help="Associate comments with a specific CMS rule tag."
|
||||||
|
),
|
||||||
|
limit: int = typer.Option(
|
||||||
|
0, "--limit", "-n", help="Stop after N comments. 0 = all."
|
||||||
|
),
|
||||||
|
attachments: bool = typer.Option(
|
||||||
|
False,
|
||||||
|
"--attachments",
|
||||||
|
help="Also download each comment's PDF/DOCX attachments "
|
||||||
|
"(expensive — skim first without).",
|
||||||
|
),
|
||||||
|
sleep: float = typer.Option(
|
||||||
|
1.3, "--sleep", help="Seconds between API calls (rate budget)."
|
||||||
|
),
|
||||||
|
) -> None:
|
||||||
|
"""Walk every comment on one docket; upsert as Source items.
|
||||||
|
|
||||||
|
Policy: we iterate every FR document indexed under the docket and
|
||||||
|
pull their comments. For one PFS rule that usually means one parent
|
||||||
|
document with a few thousand to tens of thousands of comments.
|
||||||
|
"""
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from bib import connect
|
||||||
|
from bib.regulations_gov import Client, upsert_comment
|
||||||
|
|
||||||
|
store = connect()
|
||||||
|
scratch = Path(f".state/comments/{docket}")
|
||||||
|
|
||||||
|
with Client(sleep=sleep) as api:
|
||||||
|
docs = api.find_documents_in_docket(docket)
|
||||||
|
typer.echo(f" docket {docket}: {len(docs)} FR documents indexed")
|
||||||
|
total = 0
|
||||||
|
for fr_doc in docs:
|
||||||
|
attrs = fr_doc.get("attributes") or {}
|
||||||
|
fr_id = fr_doc["id"]
|
||||||
|
object_id = attrs.get("objectId")
|
||||||
|
if not object_id or not attrs.get("commentEndDate"):
|
||||||
|
continue
|
||||||
|
typer.echo(f" ↓ comments on {fr_id} (objectId={object_id})")
|
||||||
|
for c in api.iter_comments(object_id):
|
||||||
|
if limit and total >= limit:
|
||||||
|
return
|
||||||
|
key = upsert_comment(store, c, cms_id=cms_id)
|
||||||
|
if attachments and c.attachment_count:
|
||||||
|
for att in api.attachments_for(c.id):
|
||||||
|
path = api.download_attachment(att.url, scratch / c.id)
|
||||||
|
if path:
|
||||||
|
store.attach_file(key, path, title=att.filename)
|
||||||
|
total += 1
|
||||||
|
if total % 50 == 0:
|
||||||
|
typer.echo(f" processed {total} comments")
|
||||||
|
store._con().commit() # noqa: SLF001
|
||||||
|
typer.echo(f" total: {total} comments upserted")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="fetch-pfs-comments")
|
||||||
|
def fetch_pfs_comments(
|
||||||
|
since: str = typer.Option("2017-01-01", "--since"),
|
||||||
|
until: str = typer.Option("", "--until"),
|
||||||
|
attachments: bool = typer.Option(False, "--attachments"),
|
||||||
|
per_docket_limit: int = typer.Option(
|
||||||
|
0,
|
||||||
|
"--per-docket-limit",
|
||||||
|
help="Cap comments fetched per docket. 0 = unlimited.",
|
||||||
|
),
|
||||||
|
sleep: float = typer.Option(1.3, "--sleep"),
|
||||||
|
) -> None:
|
||||||
|
"""One-shot: discover every PFS rule since *since* and pull every
|
||||||
|
comment on each of their dockets.
|
||||||
|
|
||||||
|
Heavy run. A single PFS proposed rule can hold 5K–25K comments;
|
||||||
|
times ~20 rules and with attachments this runs for many hours
|
||||||
|
under the reg.gov rate budget. Use ``--per-docket-limit`` to
|
||||||
|
smoke-test first.
|
||||||
|
"""
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from bib import connect
|
||||||
|
from bib.federalregister import pfs_rules, split_docket_ids
|
||||||
|
from bib.regulations_gov import Client, upsert_comment
|
||||||
|
from bib.tag import Tag
|
||||||
|
from bib.translate import federal_register
|
||||||
|
|
||||||
|
store = connect()
|
||||||
|
rules = pfs_rules(since=since, until=until or None)
|
||||||
|
typer.echo(f"==> {len(rules)} PFS rules since {since}")
|
||||||
|
|
||||||
|
# Only proposed rules have public comment periods — skip finals +
|
||||||
|
# corrections to keep the work focused.
|
||||||
|
proposed = [r for r in rules if r.type == "Proposed Rule"]
|
||||||
|
typer.echo(f" {len(proposed)} proposed rules (the ones with comments)")
|
||||||
|
|
||||||
|
with Client(sleep=sleep) as api:
|
||||||
|
for doc in proposed:
|
||||||
|
cms_ids = split_docket_ids(doc.dockets)
|
||||||
|
try:
|
||||||
|
rule = federal_register(
|
||||||
|
doc.html_url
|
||||||
|
or f"https://www.federalregister.gov/documents/{doc.document_number}"
|
||||||
|
)
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
typer.echo(f" skipped rule meta: {e}")
|
||||||
|
continue
|
||||||
|
rule.add_tag(Tag.source("federal-register").label)
|
||||||
|
rule.add_tag("module:pfs")
|
||||||
|
for cid in cms_ids:
|
||||||
|
rule.add_tag(f"cms-rule:{cid}")
|
||||||
|
store.upsert(rule)
|
||||||
|
|
||||||
|
# Resolve each CMS-XXXX-P to its reg.gov docket id.
|
||||||
|
for cms_id in cms_ids:
|
||||||
|
reg_docket = api.resolve_docket(cms_id)
|
||||||
|
if not reg_docket:
|
||||||
|
typer.echo(f" skip {cms_id}: no reg.gov docket found")
|
||||||
|
continue
|
||||||
|
typer.echo(f" {cms_id} → {reg_docket} ({doc.publication_date})")
|
||||||
|
rule.add_tag(f"reg-docket:{reg_docket}")
|
||||||
|
store.upsert(rule)
|
||||||
|
|
||||||
|
fr_docs = api.find_documents_in_docket(reg_docket)
|
||||||
|
per_docket_count = 0
|
||||||
|
scratch = Path(f".state/comments/{reg_docket}")
|
||||||
|
for fr_doc in fr_docs:
|
||||||
|
attrs = fr_doc.get("attributes") or {}
|
||||||
|
object_id = attrs.get("objectId")
|
||||||
|
# Skip docs with no objectId or no real comment
|
||||||
|
# window — final rules and internal display versions
|
||||||
|
# don't carry meaningful comment traffic.
|
||||||
|
if not object_id:
|
||||||
|
continue
|
||||||
|
if not attrs.get("commentEndDate"):
|
||||||
|
continue
|
||||||
|
for c in api.iter_comments(object_id):
|
||||||
|
if per_docket_limit and per_docket_count >= per_docket_limit:
|
||||||
|
break
|
||||||
|
key = upsert_comment(
|
||||||
|
store,
|
||||||
|
c,
|
||||||
|
cms_id=cms_id,
|
||||||
|
extra_tags=[f"reg-docket:{reg_docket}"],
|
||||||
|
)
|
||||||
|
if attachments and c.attachment_count:
|
||||||
|
for att in api.attachments_for(c.id):
|
||||||
|
path = api.download_attachment(
|
||||||
|
att.url,
|
||||||
|
scratch / c.id,
|
||||||
|
)
|
||||||
|
if path:
|
||||||
|
store.attach_file(key, path, title=att.filename)
|
||||||
|
per_docket_count += 1
|
||||||
|
if per_docket_count % 50 == 0:
|
||||||
|
typer.echo(f" {per_docket_count} comments")
|
||||||
|
store._con().commit() # noqa: SLF001
|
||||||
|
if per_docket_limit and per_docket_count >= per_docket_limit:
|
||||||
|
break
|
||||||
|
store._con().commit() # noqa: SLF001
|
||||||
|
typer.echo(f" docket total: {per_docket_count}")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="ingest-mail")
|
||||||
|
def ingest_mail(
|
||||||
|
user: str = typer.Option(
|
||||||
|
"cmsupdates@mail.fhirworx.io",
|
||||||
|
"--user",
|
||||||
|
help="Mailbox address to poll (must have a password in .state/mail/credentials.json).",
|
||||||
|
),
|
||||||
|
host: str = typer.Option("mail.fhirworx.io", "--host"),
|
||||||
|
port: int = typer.Option(993, "--port"),
|
||||||
|
folder: str = typer.Option("INBOX", "--folder"),
|
||||||
|
limit: int = typer.Option(0, "--limit", "-n"),
|
||||||
|
) -> None:
|
||||||
|
"""Pull UNSEEN mail from a mailbox via IMAPS, upsert as bib Sources.
|
||||||
|
|
||||||
|
Idempotent: marks each ingested message ``\\Seen`` on the server,
|
||||||
|
so re-runs only fetch new mail. Attachments ride along into bib
|
||||||
|
storage. After this, ``stack bib sync-zotero`` routes everything
|
||||||
|
tagged ``source:email`` into the appropriate Zotero collection.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
|
||||||
|
from bib import connect
|
||||||
|
from bib.email_ingest import Mailbox, ingest
|
||||||
|
|
||||||
|
creds_path = Path(".state/mail/credentials.json")
|
||||||
|
if not creds_path.exists():
|
||||||
|
raise typer.BadParameter(
|
||||||
|
"no .state/mail/credentials.json — run `stack mail provision` first."
|
||||||
|
)
|
||||||
|
creds = json.loads(creds_path.read_text())
|
||||||
|
pw = creds.get(user) or creds.get(user.split("@", 1)[0])
|
||||||
|
if not pw:
|
||||||
|
raise typer.BadParameter(
|
||||||
|
f"no cached password for {user}.\nRun: stack mail rotate-creds {user}"
|
||||||
|
)
|
||||||
|
|
||||||
|
store = connect()
|
||||||
|
stats = ingest(
|
||||||
|
store,
|
||||||
|
Mailbox(host=host, port=port, username=user, password=pw, folder=folder),
|
||||||
|
limit=limit or None,
|
||||||
|
)
|
||||||
|
for k, v in stats.items():
|
||||||
|
typer.echo(f" {k}: {v}")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="backfill-comments")
|
||||||
|
def backfill_comments(
|
||||||
|
limit: int = typer.Option(
|
||||||
|
0,
|
||||||
|
"--limit",
|
||||||
|
"-n",
|
||||||
|
help="Cap items processed this run. 0 = no cap (multi-day crawl).",
|
||||||
|
),
|
||||||
|
sleep: float = typer.Option(
|
||||||
|
3.7,
|
||||||
|
"--sleep",
|
||||||
|
help="Seconds between API calls. 3.7s ≈ 970/hr — just under the "
|
||||||
|
"1000/hr reg.gov cap.",
|
||||||
|
),
|
||||||
|
log_path: Path = typer.Option(
|
||||||
|
Path("/tmp/bib-backfill-comments.log"),
|
||||||
|
"--log",
|
||||||
|
help="Append-only progress log (survives restarts).",
|
||||||
|
),
|
||||||
|
) -> None:
|
||||||
|
"""Enrich every reg-gov comment stub with body + attachments.
|
||||||
|
|
||||||
|
Walks items tagged ``source:regulations-gov`` whose ``abstract`` is
|
||||||
|
still empty and hits ``/v4/comments/{id}?include=attachments`` for
|
||||||
|
each one. Resumable: stop any time, start again, it picks up where
|
||||||
|
it left off by skipping already-enriched items. At 1000/hr against
|
||||||
|
164K stubs this is a ~7-day crawl; longer if attachments are big.
|
||||||
|
"""
|
||||||
|
from bib import connect
|
||||||
|
from bib.regulations_gov import Client, backfill_details
|
||||||
|
|
||||||
|
store = connect()
|
||||||
|
with Client(sleep=sleep) as api:
|
||||||
|
stats = backfill_details(
|
||||||
|
store,
|
||||||
|
api,
|
||||||
|
limit=limit or None,
|
||||||
|
log_path=log_path,
|
||||||
|
)
|
||||||
|
for k, v in stats.items():
|
||||||
|
typer.echo(f" {k}: {v}")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="ingest-iom")
|
||||||
|
def ingest_iom(
|
||||||
|
pubs: list[str] = typer.Option(
|
||||||
|
None,
|
||||||
|
"--pub",
|
||||||
|
"-p",
|
||||||
|
help="Limit to specific pub numbers (e.g. 100-02). Repeat for multiple.",
|
||||||
|
),
|
||||||
|
) -> None:
|
||||||
|
"""Crawl the CMS IOM index and upsert every chapter as a Manual.
|
||||||
|
|
||||||
|
Idempotent: re-running only adds missing chapters (dedup is by URL
|
||||||
|
in the store). Without --pub, ingests every manual the CMS index
|
||||||
|
advertises.
|
||||||
|
"""
|
||||||
|
from bib import connect
|
||||||
|
from bib.iom import ingest_all
|
||||||
|
|
||||||
|
store = connect()
|
||||||
|
with _http_client() as client:
|
||||||
|
summary = ingest_all(store, client, pubs=pubs)
|
||||||
|
|
||||||
|
for pub, n in sorted(summary.items()):
|
||||||
|
typer.echo(f" {pub}: {n} chapters")
|
||||||
|
typer.echo(
|
||||||
|
f"Total: {sum(summary.values())} chapters across {len(summary)} manuals."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="attach-iom")
|
||||||
|
def attach_iom(
|
||||||
|
pubs: list[str] = typer.Option(
|
||||||
|
None,
|
||||||
|
"--pub",
|
||||||
|
"-p",
|
||||||
|
help="Limit to specific pub numbers. Repeat for multiple.",
|
||||||
|
),
|
||||||
|
force: bool = typer.Option(
|
||||||
|
False,
|
||||||
|
"--force",
|
||||||
|
help="Re-download even if local SHA-256 matches.",
|
||||||
|
),
|
||||||
|
) -> None:
|
||||||
|
"""Download the PDF for every Manual and attach it to the record.
|
||||||
|
|
||||||
|
Idempotent by SHA-256 — only downloads files that have changed on
|
||||||
|
CMS since the last run. Safe to schedule on a timer.
|
||||||
|
"""
|
||||||
|
from bib import connect
|
||||||
|
from bib.iom import download_attachments
|
||||||
|
|
||||||
|
store = connect()
|
||||||
|
with _http_client() as client:
|
||||||
|
results = download_attachments(store, client, pubs=pubs, force=force)
|
||||||
|
|
||||||
|
for pub, n in sorted(results.items()):
|
||||||
|
typer.echo(f" {pub}: {n} new/updated")
|
||||||
|
typer.echo(f"Total: {sum(results.values())} attachments updated.")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="watch-iom")
|
||||||
|
def watch_iom() -> None:
|
||||||
|
"""Poll the CMS futurepdf.pdf schedule; attach if changed.
|
||||||
|
|
||||||
|
CMS publishes upcoming IOM transmittals at
|
||||||
|
``/manuals/downloads/futurepdf.pdf``. When its checksum changes,
|
||||||
|
chapters in the main manuals are probably about to update — run
|
||||||
|
``ingest-iom`` + ``attach-iom`` afterward to pick them up.
|
||||||
|
"""
|
||||||
|
from bib import connect
|
||||||
|
from bib.iom import check_future_updates
|
||||||
|
|
||||||
|
store = connect()
|
||||||
|
with _http_client() as client:
|
||||||
|
changed, digest = check_future_updates(store, client)
|
||||||
|
|
||||||
|
status = "CHANGED" if changed else "unchanged"
|
||||||
|
typer.echo(f"futurepdf.pdf: {status} (sha256={digest[:12]})")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="sync-zotero")
|
||||||
|
def sync_zotero(
|
||||||
|
tag: str = typer.Option("", "--tag", help="Only sync items matching a tag."),
|
||||||
|
item_type: str = typer.Option("", "--type", help="Only sync one item_type."),
|
||||||
|
hold_zotero: bool = typer.Option(
|
||||||
|
True,
|
||||||
|
"--hold-zotero/--no-hold",
|
||||||
|
help="Stop the 'zotero' container during sync so Zotero releases "
|
||||||
|
"its exclusive SQLite lock; restart afterwards.",
|
||||||
|
),
|
||||||
|
) -> None:
|
||||||
|
"""Push bib items (and their attachments) to the Zotero database.
|
||||||
|
|
||||||
|
Zotero holds an exclusive write lock on its SQLite file while
|
||||||
|
running. By default we stop the 'zotero' compose service for the
|
||||||
|
duration of the sync and start it again afterward.
|
||||||
|
"""
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
from bib import connect
|
||||||
|
from bib.sync import push_to_zotero
|
||||||
|
|
||||||
|
store = connect()
|
||||||
|
items = store.list_items(tag=tag, item_type=item_type)
|
||||||
|
|
||||||
|
def _docker(*args: str) -> None:
|
||||||
|
subprocess.run(
|
||||||
|
["docker", *args],
|
||||||
|
capture_output=True,
|
||||||
|
check=False,
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
if hold_zotero:
|
||||||
|
_docker("stop", "zotero")
|
||||||
|
stats = push_to_zotero(items, store=store)
|
||||||
|
finally:
|
||||||
|
if hold_zotero:
|
||||||
|
_docker("start", "zotero")
|
||||||
|
|
||||||
|
for k, v in stats.items():
|
||||||
|
typer.echo(f" {k}: {v}")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="ingest-oig")
|
||||||
|
def ingest_oig(
|
||||||
|
kinds: list[str] = typer.Option(
|
||||||
|
None,
|
||||||
|
"--kind",
|
||||||
|
"-k",
|
||||||
|
help="Restrict to a subset: 'cpg' and/or 'alerts'. Repeat for multiple.",
|
||||||
|
),
|
||||||
|
) -> None:
|
||||||
|
"""Crawl OIG Compliance Program Guidance + fraud alerts indexes."""
|
||||||
|
from bib import connect
|
||||||
|
from bib.oig import ingest_all
|
||||||
|
|
||||||
|
store = connect()
|
||||||
|
with _http_client() as client:
|
||||||
|
summary = ingest_all(store, client, kinds=kinds)
|
||||||
|
for k, n in summary.items():
|
||||||
|
typer.echo(f" {k}: {n} docs")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="attach-oig")
|
||||||
|
def attach_oig(
|
||||||
|
force: bool = typer.Option(False, "--force", help="Re-download even if unchanged."),
|
||||||
|
) -> None:
|
||||||
|
"""Download and attach the PDF/HTML for every OIG guidance record."""
|
||||||
|
from bib import connect
|
||||||
|
from bib.oig import download_attachments
|
||||||
|
|
||||||
|
store = connect()
|
||||||
|
with _http_client() as client:
|
||||||
|
n = download_attachments(store, client, force=force)
|
||||||
|
typer.echo(f" {n} attachments updated")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="refresh-oig")
|
||||||
|
def refresh_oig() -> None:
|
||||||
|
"""One-shot: ingest-oig → attach-oig → sync-zotero (OIG tag)."""
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
from bib import connect
|
||||||
|
from bib.oig import download_attachments, ingest_all
|
||||||
|
from bib.sync import push_to_zotero
|
||||||
|
|
||||||
|
store = connect()
|
||||||
|
with _http_client() as client:
|
||||||
|
typer.echo("==> ingest-oig")
|
||||||
|
summary = ingest_all(store, client)
|
||||||
|
typer.echo(f" {summary['cpg']} CPGs, {summary['alerts']} alerts")
|
||||||
|
|
||||||
|
typer.echo("==> attach-oig")
|
||||||
|
n = download_attachments(store, client)
|
||||||
|
typer.echo(f" {n} attachments new/changed")
|
||||||
|
|
||||||
|
typer.echo("==> sync-zotero")
|
||||||
|
items = store.list_items(tag="agency:oig")
|
||||||
|
try:
|
||||||
|
subprocess.run(
|
||||||
|
["docker", "stop", "zotero"], capture_output=True, check=False, timeout=30
|
||||||
|
)
|
||||||
|
stats = push_to_zotero(items, store=store)
|
||||||
|
finally:
|
||||||
|
subprocess.run(
|
||||||
|
["docker", "start", "zotero"], capture_output=True, check=False, timeout=30
|
||||||
|
)
|
||||||
|
typer.echo(
|
||||||
|
f" created={stats['created']} skipped={stats['skipped']} "
|
||||||
|
f"attachments={stats['attachments']}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="refresh-iom")
|
||||||
|
def refresh_iom(
|
||||||
|
pubs: list[str] = typer.Option(None, "--pub", "-p"),
|
||||||
|
) -> None:
|
||||||
|
"""One-shot: ingest-iom → watch-iom → attach-iom → sync-zotero.
|
||||||
|
|
||||||
|
Safe to put behind a cron. Idempotent at every step:
|
||||||
|
- ingest: upserts by URL, prunes stale records
|
||||||
|
- watch: re-attaches futurepdf.pdf only when its SHA-256 changes
|
||||||
|
- attach: skips PDFs whose checksum matches the stored copy
|
||||||
|
- sync: skips Zotero items whose URL already exists
|
||||||
|
|
||||||
|
Zotero is held (stopped) during the sync phase only.
|
||||||
|
"""
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
from bib import connect
|
||||||
|
from bib.iom import check_future_updates, download_attachments, ingest_all
|
||||||
|
from bib.sync import push_to_zotero
|
||||||
|
|
||||||
|
store = connect()
|
||||||
|
with _http_client() as client:
|
||||||
|
typer.echo("==> ingest-iom")
|
||||||
|
ingest_summary = ingest_all(store, client, pubs=pubs)
|
||||||
|
typer.echo(
|
||||||
|
f" {sum(ingest_summary.values())} chapters across "
|
||||||
|
f"{len([p for p, n in ingest_summary.items() if n])} manuals"
|
||||||
|
)
|
||||||
|
|
||||||
|
typer.echo("==> watch-iom")
|
||||||
|
changed, digest = check_future_updates(store, client)
|
||||||
|
typer.echo(
|
||||||
|
f" futurepdf.pdf: {'CHANGED' if changed else 'unchanged'} "
|
||||||
|
f"(sha256={digest[:12]})"
|
||||||
|
)
|
||||||
|
|
||||||
|
typer.echo("==> attach-iom")
|
||||||
|
attach_summary = download_attachments(store, client, pubs=pubs)
|
||||||
|
typer.echo(f" {sum(attach_summary.values())} PDFs downloaded or refreshed")
|
||||||
|
|
||||||
|
typer.echo("==> sync-zotero")
|
||||||
|
items = store.list_items(item_type="manual")
|
||||||
|
try:
|
||||||
|
subprocess.run(
|
||||||
|
["docker", "stop", "zotero"], capture_output=True, check=False, timeout=30
|
||||||
|
)
|
||||||
|
stats = push_to_zotero(items, store=store)
|
||||||
|
finally:
|
||||||
|
subprocess.run(
|
||||||
|
["docker", "start", "zotero"], capture_output=True, check=False, timeout=30
|
||||||
|
)
|
||||||
|
typer.echo(
|
||||||
|
f" created={stats['created']} skipped={stats['skipped']} "
|
||||||
|
f"attachments={stats['attachments']}"
|
||||||
|
)
|
||||||
|
|||||||
123
src/cli/mail.py
Normal file
@@ -0,0 +1,123 @@
|
|||||||
|
"""`stack mail` — touchless Maddy mail server lifecycle."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import typer
|
||||||
|
|
||||||
|
app = typer.Typer(no_args_is_help=True)
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="provision")
|
||||||
|
def provision(
|
||||||
|
region: str = typer.Option("nyc3", help="DO region slug (one your ISP can reach)."),
|
||||||
|
) -> None:
|
||||||
|
"""End-to-end idempotent: up → wait → DNS → DKIM → smarthost → git env."""
|
||||||
|
from mail.droplet import provision as _provision
|
||||||
|
|
||||||
|
_provision(region=region)
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="up")
|
||||||
|
def up(
|
||||||
|
region: str = typer.Option("nyc3", help="DO region slug."),
|
||||||
|
) -> None:
|
||||||
|
"""Adopt-or-create the mail droplet. Idempotent."""
|
||||||
|
from mail.droplet import up as _up
|
||||||
|
|
||||||
|
_up(region=region)
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="down")
|
||||||
|
def down(
|
||||||
|
yes: bool = typer.Option(False, "--yes", help="Skip confirmation."),
|
||||||
|
) -> None:
|
||||||
|
"""Destroy the mail droplet. Idempotent no-op if none exists."""
|
||||||
|
from mail.droplet import down as _down
|
||||||
|
|
||||||
|
if not yes:
|
||||||
|
typer.confirm("Destroy the mail droplet?", abort=True)
|
||||||
|
_down(confirm=True)
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="status")
|
||||||
|
def status() -> None:
|
||||||
|
"""Show droplet state from DO + PTR check."""
|
||||||
|
from mail.droplet import status as _status
|
||||||
|
|
||||||
|
info = _status()
|
||||||
|
if not info:
|
||||||
|
typer.echo("no mail droplet exists — `stack mail provision`")
|
||||||
|
return
|
||||||
|
for k in ("name", "id", "region", "public_ip", "status", "ptr", "ptr_match"):
|
||||||
|
typer.echo(f" {k:11s} {info.get(k)}")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="dns")
|
||||||
|
def dns(
|
||||||
|
claim_apex: bool = typer.Option(
|
||||||
|
False,
|
||||||
|
"--claim-apex",
|
||||||
|
help="Also write apex MX/SPF/DMARC. Skip when another provider "
|
||||||
|
"(Proton, Fastmail, Workspace) already owns the apex.",
|
||||||
|
),
|
||||||
|
) -> None:
|
||||||
|
"""Upsert mail DNS records in Cloudflare. Idempotent."""
|
||||||
|
from mail.droplet import apply_dns
|
||||||
|
|
||||||
|
apply_dns(claim_apex=claim_apex)
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="dkim-export")
|
||||||
|
def dkim_export() -> None:
|
||||||
|
"""Publish Maddy's DKIM public key to Cloudflare. Idempotent."""
|
||||||
|
from mail.droplet import export_dkim
|
||||||
|
|
||||||
|
export_dkim()
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="attach-smarthost")
|
||||||
|
def attach_smarthost() -> None:
|
||||||
|
"""Register at Resend, push DNS, install smarthost on droplet."""
|
||||||
|
from mail.droplet import attach_smarthost as _attach
|
||||||
|
|
||||||
|
_attach()
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="rotate-creds")
|
||||||
|
def rotate_creds(
|
||||||
|
user: str = typer.Argument(
|
||||||
|
..., help="Mailbox to rotate (e.g. git, postmaster, cmsupdates)."
|
||||||
|
),
|
||||||
|
) -> None:
|
||||||
|
"""Mint a new password for *user*@fhirworx.io and push to droplet."""
|
||||||
|
from mail.droplet import rotate_creds as _rotate
|
||||||
|
|
||||||
|
_rotate(user)
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="seed-mailboxes")
|
||||||
|
def seed_mailboxes() -> None:
|
||||||
|
"""Ensure the standard set of mailboxes exists (postmaster, git, cmsupdates).
|
||||||
|
|
||||||
|
Idempotent: only creates missing ones; existing mailboxes keep
|
||||||
|
their cached passwords. Useful after adding a new entry to
|
||||||
|
``DEFAULT_MAILBOXES`` in mail/droplet.py.
|
||||||
|
"""
|
||||||
|
from mail.droplet import seed_mailboxes as _seed
|
||||||
|
|
||||||
|
_seed()
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="wire-git")
|
||||||
|
def wire_git() -> None:
|
||||||
|
"""Write .state/git/mailer.env from the mail droplet's credentials.
|
||||||
|
|
||||||
|
Then run ``docker compose up -d git`` to apply (a plain ``docker
|
||||||
|
restart`` won't re-read the new env_file directive in compose.yml).
|
||||||
|
"""
|
||||||
|
from mail.droplet import write_git_mailer_env
|
||||||
|
|
||||||
|
if write_git_mailer_env():
|
||||||
|
typer.echo(" git mailer env written → run: docker compose up -d git")
|
||||||
|
else:
|
||||||
|
typer.echo(" git mailer env unchanged or skipped")
|
||||||
527
src/cli/prisma.py
Normal file
@@ -0,0 +1,527 @@
|
|||||||
|
"""stack prisma — LLM-driven PRISMA screening workflow.
|
||||||
|
|
||||||
|
Stages:
|
||||||
|
|
||||||
|
1 identified (implicit — everything in the project tag)
|
||||||
|
2 screen title/abstract LLM pass
|
||||||
|
3 eligibility full-text LLM pass
|
||||||
|
3+ extract structured data extraction on included items
|
||||||
|
4 included derived from tag state
|
||||||
|
|
||||||
|
Commands are idempotent: tags record the transition a row has already
|
||||||
|
made, and the screen / eligibility loops skip items already past that
|
||||||
|
stage.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import typer
|
||||||
|
|
||||||
|
app = typer.Typer(no_args_is_help=True)
|
||||||
|
|
||||||
|
|
||||||
|
_DEFAULT_ZOT = Path("data/zotero/data/zotero.sqlite")
|
||||||
|
_DEFAULT_STORAGE = Path("data/zotero/data/storage")
|
||||||
|
|
||||||
|
|
||||||
|
def _hold(fn, *, hold: bool):
|
||||||
|
if not hold:
|
||||||
|
return fn()
|
||||||
|
subprocess.run(
|
||||||
|
["docker", "stop", "zotero"], capture_output=True, check=False, timeout=30
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
return fn()
|
||||||
|
finally:
|
||||||
|
subprocess.run(
|
||||||
|
["docker", "start", "zotero"], capture_output=True, check=False, timeout=30
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="init")
|
||||||
|
def init_project(
|
||||||
|
name: str = typer.Argument(..., help="Project slug, e.g. skin-subs."),
|
||||||
|
db: Path = typer.Option(_DEFAULT_ZOT, "--db", help="Zotero SQLite path."),
|
||||||
|
hold: bool = typer.Option(True, "--hold/--no-hold"),
|
||||||
|
) -> None:
|
||||||
|
"""Create (or re-load) the project's three Zotero anchor items:
|
||||||
|
criteria, extraction-template, and the reasons codebook."""
|
||||||
|
from prisma.project import init as _init
|
||||||
|
from zot.db import Db
|
||||||
|
|
||||||
|
def _go():
|
||||||
|
with Db(str(db)) as zdb:
|
||||||
|
project = _init(zdb, name)
|
||||||
|
typer.echo(f"project: {project.name}")
|
||||||
|
typer.echo(f" criteria: {len(project.criteria):>6} chars")
|
||||||
|
typer.echo(
|
||||||
|
f" extraction_template: {len(project.extraction_template):>6} chars"
|
||||||
|
)
|
||||||
|
typer.echo(f" reasons: {len(project.reasons):>6} chars")
|
||||||
|
|
||||||
|
_hold(_go, hold=hold)
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="export")
|
||||||
|
def export_item(
|
||||||
|
zot_id: int = typer.Argument(..., help="Zotero itemID to export."),
|
||||||
|
fulltext: bool = typer.Option(False, "--fulltext"),
|
||||||
|
db: Path = typer.Option(_DEFAULT_ZOT, "--db"),
|
||||||
|
storage: Path = typer.Option(_DEFAULT_STORAGE, "--storage"),
|
||||||
|
hold: bool = typer.Option(True, "--hold/--no-hold"),
|
||||||
|
) -> None:
|
||||||
|
"""Dump one Zotero item as the LLM-ready markdown snapshot."""
|
||||||
|
from prisma.export import load_item, to_markdown
|
||||||
|
from zot.db import Db
|
||||||
|
|
||||||
|
def _go() -> str:
|
||||||
|
with Db(str(db)) as zdb:
|
||||||
|
snap = load_item(zdb, zot_id, storage)
|
||||||
|
return to_markdown(snap, include_fulltext=fulltext)
|
||||||
|
|
||||||
|
typer.echo(_hold(_go, hold=hold))
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="ping-llm")
|
||||||
|
def ping_llm() -> None:
|
||||||
|
"""Send a cheap ping through the configured provider.
|
||||||
|
|
||||||
|
Useful for verifying ``PRISMA_LLM_PROVIDER`` + auth before kicking
|
||||||
|
off a long screening run.
|
||||||
|
"""
|
||||||
|
from prisma.llm import LLMCall, LLMMessage, make_provider
|
||||||
|
|
||||||
|
provider = make_provider()
|
||||||
|
call = LLMCall(
|
||||||
|
messages=[
|
||||||
|
LLMMessage(role="user", content="Respond with exactly: pong."),
|
||||||
|
],
|
||||||
|
max_tokens=8,
|
||||||
|
)
|
||||||
|
result = provider.complete(call)
|
||||||
|
typer.echo(f"text: {result.text.strip()}")
|
||||||
|
typer.echo(f"usage: {result.usage}")
|
||||||
|
|
||||||
|
|
||||||
|
# ── Stage loops ─────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="screen")
|
||||||
|
def screen(
|
||||||
|
name: str = typer.Argument(..., help="Project slug."),
|
||||||
|
limit: int = typer.Option(
|
||||||
|
0, "--limit", "-n", help="0 = screen every unscreened item."
|
||||||
|
),
|
||||||
|
db: Path = typer.Option(_DEFAULT_ZOT, "--db"),
|
||||||
|
storage: Path = typer.Option(_DEFAULT_STORAGE, "--storage"),
|
||||||
|
hold: bool = typer.Option(True, "--hold/--no-hold"),
|
||||||
|
) -> None:
|
||||||
|
"""Stage 2 — LLM-driven title/abstract screening pass."""
|
||||||
|
from prisma import screen as _screen
|
||||||
|
from prisma.llm import make_provider
|
||||||
|
from prisma.project import load
|
||||||
|
from zot.db import Db
|
||||||
|
|
||||||
|
def _go():
|
||||||
|
provider = make_provider()
|
||||||
|
with Db(str(db)) as zdb:
|
||||||
|
project = load(zdb, name)
|
||||||
|
return _screen.run(
|
||||||
|
zdb,
|
||||||
|
provider,
|
||||||
|
project,
|
||||||
|
storage_dir=storage,
|
||||||
|
limit=limit or None,
|
||||||
|
)
|
||||||
|
|
||||||
|
stats = _hold(_go, hold=hold)
|
||||||
|
for k, v in stats.items():
|
||||||
|
typer.echo(f" {k}: {v}")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="eligible")
|
||||||
|
def eligible(
|
||||||
|
name: str = typer.Argument(..., help="Project slug."),
|
||||||
|
limit: int = typer.Option(0, "--limit", "-n"),
|
||||||
|
db: Path = typer.Option(_DEFAULT_ZOT, "--db"),
|
||||||
|
storage: Path = typer.Option(_DEFAULT_STORAGE, "--storage"),
|
||||||
|
hold: bool = typer.Option(True, "--hold/--no-hold"),
|
||||||
|
) -> None:
|
||||||
|
"""Stage 3 — LLM-driven full-text eligibility pass."""
|
||||||
|
from prisma import eligibility
|
||||||
|
from prisma.llm import make_provider
|
||||||
|
from prisma.project import load
|
||||||
|
from zot.db import Db
|
||||||
|
|
||||||
|
def _go():
|
||||||
|
provider = make_provider()
|
||||||
|
with Db(str(db)) as zdb:
|
||||||
|
project = load(zdb, name)
|
||||||
|
return eligibility.run(
|
||||||
|
zdb,
|
||||||
|
provider,
|
||||||
|
project,
|
||||||
|
storage_dir=storage,
|
||||||
|
limit=limit or None,
|
||||||
|
)
|
||||||
|
|
||||||
|
stats = _hold(_go, hold=hold)
|
||||||
|
for k, v in stats.items():
|
||||||
|
typer.echo(f" {k}: {v}")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="extract")
|
||||||
|
def extract(
|
||||||
|
name: str = typer.Argument(..., help="Project slug."),
|
||||||
|
limit: int = typer.Option(0, "--limit", "-n"),
|
||||||
|
db: Path = typer.Option(_DEFAULT_ZOT, "--db"),
|
||||||
|
storage: Path = typer.Option(_DEFAULT_STORAGE, "--storage"),
|
||||||
|
hold: bool = typer.Option(True, "--hold/--no-hold"),
|
||||||
|
) -> None:
|
||||||
|
"""Stage 3+ — structured data extraction on included studies."""
|
||||||
|
from prisma import extract as _extract
|
||||||
|
from prisma.llm import make_provider
|
||||||
|
from prisma.project import load
|
||||||
|
from zot.db import Db
|
||||||
|
|
||||||
|
def _go():
|
||||||
|
provider = make_provider()
|
||||||
|
with Db(str(db)) as zdb:
|
||||||
|
project = load(zdb, name)
|
||||||
|
return _extract.run(
|
||||||
|
zdb,
|
||||||
|
provider,
|
||||||
|
project,
|
||||||
|
storage_dir=storage,
|
||||||
|
limit=limit or None,
|
||||||
|
)
|
||||||
|
|
||||||
|
stats = _hold(_go, hold=hold)
|
||||||
|
for k, v in stats.items():
|
||||||
|
typer.echo(f" {k}: {v}")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="flow")
|
||||||
|
def flow(
|
||||||
|
name: str = typer.Argument(..., help="Project slug."),
|
||||||
|
mermaid: bool = typer.Option(True, "--mermaid/--text"),
|
||||||
|
db: Path = typer.Option(_DEFAULT_ZOT, "--db"),
|
||||||
|
hold: bool = typer.Option(True, "--hold/--no-hold"),
|
||||||
|
) -> None:
|
||||||
|
"""Emit the PRISMA 2020 flow diagram from current tag counts."""
|
||||||
|
from prisma import flow as _flow
|
||||||
|
from zot.db import Db
|
||||||
|
|
||||||
|
def _go() -> str:
|
||||||
|
with Db(str(db)) as zdb:
|
||||||
|
counts = _flow.count(zdb, name)
|
||||||
|
return (
|
||||||
|
_flow.mermaid(counts, project=name)
|
||||||
|
if mermaid
|
||||||
|
else _flow.text_summary(counts)
|
||||||
|
)
|
||||||
|
|
||||||
|
typer.echo(_hold(_go, hold=hold))
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="fetch")
|
||||||
|
def fetch(
|
||||||
|
name: str = typer.Argument(..., help="Project slug."),
|
||||||
|
limit: int = typer.Option(0, "--limit", "-n"),
|
||||||
|
db: Path = typer.Option(_DEFAULT_ZOT, "--db"),
|
||||||
|
storage: Path = typer.Option(_DEFAULT_STORAGE, "--storage"),
|
||||||
|
scratch: Path = typer.Option(Path(".state/prisma-fetch"), "--scratch"),
|
||||||
|
proxy: str = typer.Option(
|
||||||
|
"", "--proxy", help="Override PRISMA_FETCH_PROXY (else env)."
|
||||||
|
),
|
||||||
|
use_vpn: bool = typer.Option(
|
||||||
|
True,
|
||||||
|
"--vpn/--no-vpn",
|
||||||
|
help="Auto-open an SSH tunnel to the provisioned droplet when one "
|
||||||
|
"exists. --no-vpn skips the tunnel and uses only the "
|
||||||
|
"unproxied sources.",
|
||||||
|
),
|
||||||
|
verify: bool = typer.Option(
|
||||||
|
True,
|
||||||
|
"--verify/--no-verify",
|
||||||
|
help="Curl through the proxy once before fetching and confirm "
|
||||||
|
"egress is non-US.",
|
||||||
|
),
|
||||||
|
hold: bool = typer.Option(True, "--hold/--no-hold"),
|
||||||
|
) -> None:
|
||||||
|
"""Fetch PDFs for every non-excluded item lacking an attachment.
|
||||||
|
|
||||||
|
Source cascade: Unpaywall → PMC → fallback (via the VPN droplet).
|
||||||
|
Items excluded at stage 2 are untouched — the queue filter only
|
||||||
|
passes `screen:include` and `screen:uncertain` through.
|
||||||
|
|
||||||
|
Tunnel lifecycle is automatic: if a droplet is tracked in
|
||||||
|
``.state/prisma-vpn/`` this command opens an SSH port-forward for
|
||||||
|
the duration of the fetch and closes it on exit. No sudo required.
|
||||||
|
"""
|
||||||
|
import contextlib
|
||||||
|
import os
|
||||||
|
|
||||||
|
from prisma import fetch as _fetch
|
||||||
|
from prisma import vpn as _vpn
|
||||||
|
from zot.db import Db
|
||||||
|
|
||||||
|
email = os.environ.get("UNPAYWALL_EMAIL", "dev@fhirworx.io")
|
||||||
|
explicit_proxy = proxy or os.environ.get("PRISMA_FETCH_PROXY") or None
|
||||||
|
|
||||||
|
droplet_up = _vpn.status().get("status") == "up"
|
||||||
|
auto_tunnel = use_vpn and droplet_up and not explicit_proxy
|
||||||
|
|
||||||
|
@contextlib.contextmanager
|
||||||
|
def _proxy_cm():
|
||||||
|
if explicit_proxy:
|
||||||
|
yield explicit_proxy
|
||||||
|
elif auto_tunnel:
|
||||||
|
with _vpn.active() as url:
|
||||||
|
if verify:
|
||||||
|
info = _vpn.verify_egress(url)
|
||||||
|
typer.echo(
|
||||||
|
f" egress: {info.get('ip', '?')} ({info.get('country', '?')})"
|
||||||
|
)
|
||||||
|
yield url
|
||||||
|
else:
|
||||||
|
if not droplet_up:
|
||||||
|
typer.echo(
|
||||||
|
" note: no VPN droplet tracked — fallback tier disabled. "
|
||||||
|
"Run `stack prisma vpn up` to provision one."
|
||||||
|
)
|
||||||
|
yield None
|
||||||
|
|
||||||
|
def _go():
|
||||||
|
with _proxy_cm() as fetch_proxy, Db(str(db)) as zdb:
|
||||||
|
return _fetch.run(
|
||||||
|
zdb,
|
||||||
|
project=name,
|
||||||
|
storage_dir=storage,
|
||||||
|
scratch_dir=scratch,
|
||||||
|
email=email,
|
||||||
|
fetch_proxy=fetch_proxy,
|
||||||
|
limit=limit or None,
|
||||||
|
)
|
||||||
|
|
||||||
|
stats = _hold(_go, hold=hold)
|
||||||
|
for k, v in stats.items():
|
||||||
|
typer.echo(f" {k}: {v}")
|
||||||
|
|
||||||
|
|
||||||
|
# ── VPN droplet lifecycle ───────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
vpn_app = typer.Typer(no_args_is_help=True)
|
||||||
|
app.add_typer(
|
||||||
|
vpn_app,
|
||||||
|
name="vpn",
|
||||||
|
help="DigitalOcean SOCKS5 exit for fallback fetch (SSH tunnel, no sudo needed).",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@vpn_app.command(name="up")
|
||||||
|
def vpn_up(
|
||||||
|
region: str = typer.Option(
|
||||||
|
"", "--region", help="DO region slug. Defaults to PRISMA_VPN_REGION."
|
||||||
|
),
|
||||||
|
attach_zotero: bool = typer.Option(
|
||||||
|
True,
|
||||||
|
"--attach-zotero/--no-attach-zotero",
|
||||||
|
help="Also start a sidecar SSH tunnel on the stack gateway network "
|
||||||
|
"and rewrite Zotero's prefs.js to proxy through it, so the "
|
||||||
|
"fallback plugin downloads via the droplet.",
|
||||||
|
),
|
||||||
|
) -> None:
|
||||||
|
"""Provision the droplet, wait for dante, wire Zotero up.
|
||||||
|
|
||||||
|
End state: Zotero's fallback-PDF plugin downloads through
|
||||||
|
``socks5://fetch-proxy:1080``; stack's own `fetch` command pulls
|
||||||
|
through the same route via ``vpn.active()``. Zero client-side sudo.
|
||||||
|
"""
|
||||||
|
from prisma import vpn as _vpn
|
||||||
|
|
||||||
|
info = _vpn.up(region=region or None, attach_zotero=attach_zotero)
|
||||||
|
typer.echo(f" droplet: {info['name']} ({info['droplet_id']})")
|
||||||
|
typer.echo(f" region: {info['region']}")
|
||||||
|
typer.echo(f" public IP: {info['public_ip']}")
|
||||||
|
typer.echo(f" ssh key: {info['ssh_key']}")
|
||||||
|
typer.echo(f" proxy URL: {info['proxy_url']}")
|
||||||
|
if info.get("sidecar"):
|
||||||
|
typer.echo(
|
||||||
|
f" zotero proxy: {info['zotero_proxy']} (sidecar: {info['sidecar']})"
|
||||||
|
)
|
||||||
|
typer.echo("")
|
||||||
|
typer.echo("Next: `stack prisma fetch <project>` — tunnel opens automatically.")
|
||||||
|
|
||||||
|
|
||||||
|
@vpn_app.command(name="down")
|
||||||
|
def vpn_down() -> None:
|
||||||
|
"""Destroy the droplet and wipe local state."""
|
||||||
|
from prisma import vpn as _vpn
|
||||||
|
|
||||||
|
info = _vpn.down()
|
||||||
|
for k, v in info.items():
|
||||||
|
typer.echo(f" {k}: {v}")
|
||||||
|
|
||||||
|
|
||||||
|
@vpn_app.command(name="status")
|
||||||
|
def vpn_status() -> None:
|
||||||
|
"""Report whether a PRISMA VPN droplet is currently tracked."""
|
||||||
|
from prisma import vpn as _vpn
|
||||||
|
|
||||||
|
for k, v in _vpn.status().items():
|
||||||
|
typer.echo(f" {k}: {v}")
|
||||||
|
|
||||||
|
|
||||||
|
@vpn_app.command(name="attach-zotero")
|
||||||
|
def vpn_attach_zotero() -> None:
|
||||||
|
"""Start the sidecar tunnel + rewire Zotero prefs.
|
||||||
|
|
||||||
|
Use when the droplet is already up but Zotero was never attached,
|
||||||
|
or to re-attach after a zotero container rebuild wiped prefs.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
|
||||||
|
from prisma import vpn as _vpn
|
||||||
|
|
||||||
|
if not _vpn._DROPLET_JSON.is_file(): # noqa: SLF001
|
||||||
|
raise typer.BadParameter(
|
||||||
|
"no droplet tracked — run `stack prisma vpn up` first."
|
||||||
|
)
|
||||||
|
meta = json.loads(_vpn._DROPLET_JSON.read_text()) # noqa: SLF001
|
||||||
|
_vpn.attach_zotero_proxy(meta["public_ip"])
|
||||||
|
typer.echo(" sidecar started on gateway network as `fetch-proxy`")
|
||||||
|
typer.echo(" zotero prefs patched → socks5://fetch-proxy:1080")
|
||||||
|
typer.echo(" zotero restarted")
|
||||||
|
|
||||||
|
|
||||||
|
@vpn_app.command(name="detach-zotero")
|
||||||
|
def vpn_detach_zotero() -> None:
|
||||||
|
"""Stop the sidecar and restore Zotero's pre-proxy prefs."""
|
||||||
|
from prisma import vpn as _vpn
|
||||||
|
|
||||||
|
_vpn.detach_zotero_proxy()
|
||||||
|
typer.echo(" sidecar removed; prefs restored; zotero restarted.")
|
||||||
|
|
||||||
|
|
||||||
|
@vpn_app.command(name="verify")
|
||||||
|
def vpn_verify() -> None:
|
||||||
|
"""Open the tunnel, curl ifconfig.co, print egress, close.
|
||||||
|
|
||||||
|
Confirms the droplet is routable and reports its public IP /
|
||||||
|
country — the touchless way to sanity-check before a long fetch
|
||||||
|
run.
|
||||||
|
"""
|
||||||
|
from prisma import vpn as _vpn
|
||||||
|
|
||||||
|
with _vpn.active() as url:
|
||||||
|
info = _vpn.verify_egress(url)
|
||||||
|
typer.echo(f" proxy: {url}")
|
||||||
|
typer.echo(f" egress: {info.get('ip', '?')}")
|
||||||
|
typer.echo(
|
||||||
|
f" country: {info.get('country', '?')} ({info.get('country_iso', '?')})"
|
||||||
|
)
|
||||||
|
typer.echo(f" asn: {info.get('asn_org', '?')}")
|
||||||
|
|
||||||
|
|
||||||
|
# ── Orchestrated run ────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="run")
|
||||||
|
def run_all(
|
||||||
|
name: str = typer.Argument(..., help="Project slug."),
|
||||||
|
limit: int = typer.Option(
|
||||||
|
0, "--limit", "-n", help="Same limit applied to every stage."
|
||||||
|
),
|
||||||
|
skip_fetch: bool = typer.Option(False, "--skip-fetch"),
|
||||||
|
skip_extract: bool = typer.Option(False, "--skip-extract"),
|
||||||
|
db: Path = typer.Option(_DEFAULT_ZOT, "--db"),
|
||||||
|
storage: Path = typer.Option(_DEFAULT_STORAGE, "--storage"),
|
||||||
|
hold: bool = typer.Option(True, "--hold/--no-hold"),
|
||||||
|
) -> None:
|
||||||
|
"""Orchestrate all PRISMA stages in order.
|
||||||
|
|
||||||
|
screen → fetch → eligible → extract → flow, all inside one zotero
|
||||||
|
hold. Fetch only pulls PDFs for non-excluded items; stage 2
|
||||||
|
exclusions are terminal by design.
|
||||||
|
"""
|
||||||
|
import contextlib
|
||||||
|
import os
|
||||||
|
|
||||||
|
from prisma import (
|
||||||
|
eligibility,
|
||||||
|
)
|
||||||
|
from prisma import (
|
||||||
|
extract as _extract,
|
||||||
|
)
|
||||||
|
from prisma import (
|
||||||
|
fetch as _fetch,
|
||||||
|
)
|
||||||
|
from prisma import (
|
||||||
|
flow as _flow,
|
||||||
|
)
|
||||||
|
from prisma import (
|
||||||
|
screen as _screen,
|
||||||
|
)
|
||||||
|
from prisma import (
|
||||||
|
vpn as _vpn,
|
||||||
|
)
|
||||||
|
from prisma.llm import make_provider
|
||||||
|
from prisma.project import load
|
||||||
|
from zot.db import Db
|
||||||
|
|
||||||
|
def _proxy_cm():
|
||||||
|
explicit = os.environ.get("PRISMA_FETCH_PROXY")
|
||||||
|
if explicit:
|
||||||
|
return contextlib.nullcontext(explicit)
|
||||||
|
if _vpn.status().get("status") == "up":
|
||||||
|
return _vpn.active()
|
||||||
|
return contextlib.nullcontext(None)
|
||||||
|
|
||||||
|
def _go() -> None:
|
||||||
|
provider = make_provider()
|
||||||
|
with Db(str(db)) as zdb:
|
||||||
|
project = load(zdb, name)
|
||||||
|
|
||||||
|
typer.echo("==> stage 2 (title/abstract screen)")
|
||||||
|
s2 = _screen.run(
|
||||||
|
zdb, provider, project, storage_dir=storage, limit=limit or None
|
||||||
|
)
|
||||||
|
typer.echo(f" {s2}")
|
||||||
|
|
||||||
|
if not skip_fetch:
|
||||||
|
typer.echo("==> fetch (PDFs for non-excluded items)")
|
||||||
|
with _proxy_cm() as fetch_proxy:
|
||||||
|
fetch_stats = _fetch.run(
|
||||||
|
zdb,
|
||||||
|
project=name,
|
||||||
|
storage_dir=storage,
|
||||||
|
scratch_dir=Path(".state/prisma-fetch"),
|
||||||
|
email=os.environ.get("UNPAYWALL_EMAIL", "dev@fhirworx.io"),
|
||||||
|
fetch_proxy=fetch_proxy,
|
||||||
|
limit=limit or None,
|
||||||
|
)
|
||||||
|
typer.echo(f" {fetch_stats}")
|
||||||
|
|
||||||
|
typer.echo("==> stage 3 (full-text eligibility)")
|
||||||
|
s3 = eligibility.run(
|
||||||
|
zdb, provider, project, storage_dir=storage, limit=limit or None
|
||||||
|
)
|
||||||
|
typer.echo(f" {s3}")
|
||||||
|
|
||||||
|
if not skip_extract:
|
||||||
|
typer.echo("==> stage 3+ (data extraction)")
|
||||||
|
se = _extract.run(
|
||||||
|
zdb, provider, project, storage_dir=storage, limit=limit or None
|
||||||
|
)
|
||||||
|
typer.echo(f" {se}")
|
||||||
|
|
||||||
|
typer.echo("==> flow diagram")
|
||||||
|
counts = _flow.count(zdb, name)
|
||||||
|
typer.echo(_flow.text_summary(counts))
|
||||||
|
|
||||||
|
_hold(_go, hold=hold)
|
||||||
128
src/cli/zot.py
Normal file
@@ -0,0 +1,128 @@
|
|||||||
|
"""stack zot — Zotero database maintenance operations.
|
||||||
|
|
||||||
|
All commands expect the ``zotero`` compose service to be stopped while
|
||||||
|
they run (SQLite file is WAL-locked by the desktop app). ``--hold``
|
||||||
|
(default on) handles that around each invocation.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import typer
|
||||||
|
|
||||||
|
app = typer.Typer(no_args_is_help=True)
|
||||||
|
|
||||||
|
_DEFAULT_DB = Path("data/zotero/data/zotero.sqlite")
|
||||||
|
|
||||||
|
|
||||||
|
def _hold_zotero(fn, hold: bool):
|
||||||
|
"""Stop the zotero container, run *fn*, restart, return fn's result."""
|
||||||
|
if not hold:
|
||||||
|
return fn()
|
||||||
|
subprocess.run(
|
||||||
|
["docker", "stop", "zotero"], capture_output=True, check=False, timeout=30
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
return fn()
|
||||||
|
finally:
|
||||||
|
subprocess.run(
|
||||||
|
["docker", "start", "zotero"], capture_output=True, check=False, timeout=30
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve(db_path: Path | None) -> Path:
|
||||||
|
path = db_path or _DEFAULT_DB
|
||||||
|
if not path.is_file():
|
||||||
|
raise typer.BadParameter(f"Zotero SQLite not found: {path}")
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="dump-schema")
|
||||||
|
def dump_schema(
|
||||||
|
db_path: Path = typer.Option(None, "--db", help=f"Default: {_DEFAULT_DB}"),
|
||||||
|
as_python: bool = typer.Option(
|
||||||
|
True,
|
||||||
|
"--python/--json",
|
||||||
|
help="Emit paste-ready Python literals for zot/db.py (default) or JSON.",
|
||||||
|
),
|
||||||
|
) -> None:
|
||||||
|
"""Dump itemType / field / creatorType IDs.
|
||||||
|
|
||||||
|
Regenerate after a Zotero upgrade: the output replaces TYPE_MAP,
|
||||||
|
FIELD_MAP, and CREATOR_TYPES in ``src/zot/db.py`` verbatim.
|
||||||
|
"""
|
||||||
|
import json as _json
|
||||||
|
|
||||||
|
from zot.ops import dump_schema as _dump
|
||||||
|
|
||||||
|
maps = _dump(_resolve(db_path))
|
||||||
|
if as_python:
|
||||||
|
for name, m in maps.items():
|
||||||
|
typer.echo(f"{name}: dict[str, int] = {{")
|
||||||
|
for k, v in m.items():
|
||||||
|
typer.echo(f' "{k}": {v},')
|
||||||
|
typer.echo("}")
|
||||||
|
typer.echo("")
|
||||||
|
else:
|
||||||
|
typer.echo(_json.dumps(maps, indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="fix-dates")
|
||||||
|
def fix_dates(
|
||||||
|
db_path: Path = typer.Option(None, "--db"),
|
||||||
|
hold: bool = typer.Option(True, "--hold/--no-hold"),
|
||||||
|
) -> None:
|
||||||
|
"""Normalize item timestamps to ISO 8601 + install auto-fix triggers."""
|
||||||
|
from zot.ops import fix_dates as _op
|
||||||
|
|
||||||
|
out = _hold_zotero(lambda: _op(_resolve(db_path)), hold)
|
||||||
|
for k, v in out.items():
|
||||||
|
typer.echo(f" {k}: {v}")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="fix-keys")
|
||||||
|
def fix_keys(
|
||||||
|
db_path: Path = typer.Option(None, "--db"),
|
||||||
|
hold: bool = typer.Option(True, "--hold/--no-hold"),
|
||||||
|
backup: bool = typer.Option(True, "--backup/--no-backup"),
|
||||||
|
) -> None:
|
||||||
|
"""Replace invalid Zotero object keys (0/1/O/lowercase → valid)."""
|
||||||
|
from zot.ops import fix_keys as _op
|
||||||
|
|
||||||
|
out = _hold_zotero(
|
||||||
|
lambda: _op(_resolve(db_path), backup=backup),
|
||||||
|
hold,
|
||||||
|
)
|
||||||
|
for k, v in out.items():
|
||||||
|
typer.echo(f" {k}: {v}")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="fix-fields")
|
||||||
|
def fix_fields(
|
||||||
|
db_path: Path = typer.Option(None, "--db"),
|
||||||
|
hold: bool = typer.Option(True, "--hold/--no-hold"),
|
||||||
|
) -> None:
|
||||||
|
"""Remap base-field rows to type-specific fieldIDs; drop orphans."""
|
||||||
|
from zot.ops import fix_fields as _op
|
||||||
|
|
||||||
|
out = _hold_zotero(lambda: _op(_resolve(db_path)), hold)
|
||||||
|
for k, v in out.items():
|
||||||
|
typer.echo(f" {k}: {v}")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command(name="verify-parity")
|
||||||
|
def verify_parity(
|
||||||
|
db_path: Path = typer.Option(None, "--db"),
|
||||||
|
) -> None:
|
||||||
|
"""Check that ``zot.db`` constants match the live Zotero schema.
|
||||||
|
|
||||||
|
The check runs implicitly on every ``Db(...)`` construction; this
|
||||||
|
command just exposes it explicitly so CI can fail loudly if someone
|
||||||
|
bumps Zotero without regenerating the maps.
|
||||||
|
"""
|
||||||
|
from zot.db import Db
|
||||||
|
|
||||||
|
with Db(str(_resolve(db_path))):
|
||||||
|
typer.echo("parity OK")
|
||||||
12
src/mail/__init__.py
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
"""Touchless mail server for fhirworx — Maddy on DO + Resend smarthost.
|
||||||
|
|
||||||
|
Mirrors the corwins.media pattern (port mappings, gotchas, state model
|
||||||
|
all documented in MEMORY.md → "corwins.media Maddy mail stack"):
|
||||||
|
provision a single DO droplet named ``mail.fhirworx.io`` (the FQDN-as-
|
||||||
|
name trick auto-sets PTR), let Maddy handle SMTP/IMAP/DKIM, route
|
||||||
|
outbound through ``smtp.resend.com:2587`` (DO blocks 25/465/587).
|
||||||
|
|
||||||
|
State lives in ``.state/mail/`` as a cache only — every command
|
||||||
|
re-discovers truth from the DO API by tag ``stack-mail``, so wiping
|
||||||
|
``.state`` never breaks future runs.
|
||||||
|
"""
|
||||||
32
src/mail/_helpers.py
Normal file
@@ -0,0 +1,32 @@
|
|||||||
|
"""Tiny shared bits: paths, env, output. No third-party deps."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[2] # /home/kert/stack
|
||||||
|
STATE_DIR = ROOT / ".state" / "mail"
|
||||||
|
|
||||||
|
DOMAIN = "fhirworx.io"
|
||||||
|
HOSTNAME = f"mail.{DOMAIN}"
|
||||||
|
TAG = "stack-mail"
|
||||||
|
DROPLET_SIZE = "s-1vcpu-1gb"
|
||||||
|
IMAGE = "ubuntu-24-04-x64"
|
||||||
|
|
||||||
|
|
||||||
|
def env(name: str, default: str = "") -> str:
|
||||||
|
return os.environ.get(name) or default
|
||||||
|
|
||||||
|
|
||||||
|
def step(msg: str) -> None:
|
||||||
|
print(f"==> {msg}", flush=True, file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
def ok(msg: str) -> None:
|
||||||
|
print(f" {msg}", flush=True, file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
def warn(msg: str) -> None:
|
||||||
|
print(f" WARN: {msg}", flush=True, file=sys.stderr)
|
||||||