feat: jinja2 README templating — gen_readme.py renders from live codebase
Some checks failed
CI / lint-test (push) Successful in 1m12s
CI / skinny-install (api) (push) Successful in 28s
CI / skinny-install (bcda) (push) Successful in 33s
CI / skinny-install (bib) (push) Successful in 28s
CI / skinny-install (aco) (push) Successful in 49s
CI / lint-test (pull_request) Successful in 1m19s
CI / skinny-install (ccw) (pull_request) Successful in 29s
CI / skinny-install (cli) (pull_request) Successful in 33s
CI / skinny-install (cms) (pull_request) Successful in 29s
CI / skinny-install (opps) (pull_request) Successful in 26s
CI / skinny-install (pfs) (pull_request) Successful in 30s
Infra CI / mc (push) Successful in 6s
Infra CI / api (push) Successful in 8s
Infra CI / notebooks (pull_request) Successful in 7s
Infra CI / zotero (pull_request) Successful in 6s
Infra CI / mc (pull_request) Successful in 6s
CI / skinny-install (bls) (push) Successful in 25s
CI / skinny-install (ccw) (push) Successful in 31s
CI / skinny-install (cli) (push) Successful in 29s
CI / skinny-install (cms) (push) Successful in 27s
CI / skinny-install (conf) (push) Successful in 30s
CI / skinny-install (opps) (push) Successful in 29s
CI / skinny-install (perf) (push) Successful in 35s
CI / skinny-install (pfs) (push) Successful in 32s
CI / skinny-install (rex) (push) Successful in 34s
CI / skinny-install (aco) (pull_request) Successful in 44s
CI / skinny-install (api) (pull_request) Successful in 30s
CI / skinny-install (bcda) (pull_request) Successful in 37s
CI / skinny-install (bib) (pull_request) Successful in 29s
CI / skinny-install (bls) (pull_request) Successful in 28s
CI / skinny-install (conf) (pull_request) Successful in 29s
CI / skinny-install (perf) (pull_request) Successful in 24s
Infra CI / notebooks (push) Successful in 11s
CI / skinny-install (rex) (pull_request) Successful in 32s
Infra CI / zotero (push) Successful in 6s
Infra CI / docs (push) Failing after 6s
Infra CI / docs (pull_request) Failing after 6s
Infra CI / api (pull_request) Successful in 7s
Package Supply Chain / pkg-supply-chain (push) Successful in 44s
Some checks failed
CI / lint-test (push) Successful in 1m12s
CI / skinny-install (api) (push) Successful in 28s
CI / skinny-install (bcda) (push) Successful in 33s
CI / skinny-install (bib) (push) Successful in 28s
CI / skinny-install (aco) (push) Successful in 49s
CI / lint-test (pull_request) Successful in 1m19s
CI / skinny-install (ccw) (pull_request) Successful in 29s
CI / skinny-install (cli) (pull_request) Successful in 33s
CI / skinny-install (cms) (pull_request) Successful in 29s
CI / skinny-install (opps) (pull_request) Successful in 26s
CI / skinny-install (pfs) (pull_request) Successful in 30s
Infra CI / mc (push) Successful in 6s
Infra CI / api (push) Successful in 8s
Infra CI / notebooks (pull_request) Successful in 7s
Infra CI / zotero (pull_request) Successful in 6s
Infra CI / mc (pull_request) Successful in 6s
CI / skinny-install (bls) (push) Successful in 25s
CI / skinny-install (ccw) (push) Successful in 31s
CI / skinny-install (cli) (push) Successful in 29s
CI / skinny-install (cms) (push) Successful in 27s
CI / skinny-install (conf) (push) Successful in 30s
CI / skinny-install (opps) (push) Successful in 29s
CI / skinny-install (perf) (push) Successful in 35s
CI / skinny-install (pfs) (push) Successful in 32s
CI / skinny-install (rex) (push) Successful in 34s
CI / skinny-install (aco) (pull_request) Successful in 44s
CI / skinny-install (api) (pull_request) Successful in 30s
CI / skinny-install (bcda) (pull_request) Successful in 37s
CI / skinny-install (bib) (pull_request) Successful in 29s
CI / skinny-install (bls) (pull_request) Successful in 28s
CI / skinny-install (conf) (pull_request) Successful in 29s
CI / skinny-install (perf) (pull_request) Successful in 24s
Infra CI / notebooks (push) Successful in 11s
CI / skinny-install (rex) (pull_request) Successful in 32s
Infra CI / zotero (push) Successful in 6s
Infra CI / docs (push) Failing after 6s
Infra CI / docs (pull_request) Failing after 6s
Infra CI / api (pull_request) Successful in 7s
Package Supply Chain / pkg-supply-chain (push) Successful in 44s
README.md.j2 is the source of truth. dev/scripts/gen_readme.py collects live counts from compose.yml, pyproject.toml, pipeline registry, pytest, and auth manifest, then renders the template with jinja2. Dynamic values: - Service count (25, from compose.yml) - Pipeline count + step counts (14 pipelines, 230 steps) - Test count (12,291, from pytest --collect-only) - Module list (14, from pyproject.toml optional-deps) - Credential count (20, from api.auth.manifest) - Workflow count (7, from .gitea/workflows/) - src/ tree (generated from filesystem) Usage: uv run python dev/scripts/gen_readme.py # generate uv run python dev/scripts/gen_readme.py --check # verify freshness uv run python dev/scripts/gen_readme.py --dry-run # preview to stdout
This commit is contained in:
75
README.md
75
README.md
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||

|

|
||||||
|
|
||||||
Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 22 services, 13 data pipelines, 12,291 tests at 99% coverage.
|
Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. 25 services, 14 data pipelines, 12,291 tests at 99% coverage.
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
@@ -37,7 +37,7 @@ pip install stack[all] # everything
|
|||||||
pip install stack[aco,aws] # ACO analytics with AWS storage
|
pip install stack[aco,aws] # ACO analytics with AWS storage
|
||||||
```
|
```
|
||||||
|
|
||||||
14 modules available as optional extras: `conf`, `aco`, `api`, `bcda`, `bib`, `bls`, `ccw`, `cli`, `cms`, `opps`, `perf`, `pfs`, `rex`, `sem`. Cloud providers: `aws`, `gcp`, `azure`. Aggregates: `all`, `lake`.
|
14 modules available as optional extras: `aco`, `api`, `bcda`, `bib`, `bls`, `ccw`, `cli`, `cms`, `conf`, `opps`, `perf`, `pfs`, `rex`, `sem`. Cloud providers: `aws`, `azure`, `gcp`. Aggregates: `all`, `lake`.
|
||||||
|
|
||||||
## Services
|
## Services
|
||||||
|
|
||||||
@@ -85,23 +85,24 @@ All services route through Traefik at `*.homelab.fhirworx.io`.
|
|||||||
|
|
||||||
### Pipelines
|
### Pipelines
|
||||||
|
|
||||||
13 registered pipelines with 189 steps, validated by `run_pipeline(steps, load)` which enforces output schemas via pandera-style column checks.
|
14 registered pipelines with 230 steps, validated by `run_pipeline(steps, load)` which enforces output schemas via pandera-style column checks.
|
||||||
|
|
||||||
| Pipeline | Domain |
|
| Pipeline | Steps | Domain |
|
||||||
|----------|--------|
|
|----------|------:|--------|
|
||||||
| `input_layer` | Raw CCLF/BCDA ingestion |
|
| `ahrq_measures` | 39 | AHRQ quality indicators |
|
||||||
| `core` | Encounters, patients, providers |
|
| `cclf` | 19 | CCLF connector pipeline -- raw CCLF files -> input_layer tables |
|
||||||
| `claims_preprocessing` | Claim matching, encounter IDs |
|
| `claims_preprocessing` | 63 | Claim matching, encounter IDs |
|
||||||
| `readmissions` | 30-day readmission rates |
|
| `cms_quality_measures` | 20 | CMS ACO quality metrics |
|
||||||
| `pharmacy` | Part D drug utilization |
|
| `core` | 12 | Encounters, patients, providers |
|
||||||
| `ahrq_measures` | AHRQ quality indicators |
|
| `data_quality` | 1 | Validation and completeness |
|
||||||
| `cms_quality_measures` | CMS ACO quality metrics |
|
| `hcc_suspecting` | 6 | HCC risk adjustment |
|
||||||
| `hcc_suspecting` | HCC risk adjustment |
|
| `input_layer` | 13 | Raw CCLF/BCDA ingestion |
|
||||||
| `provider_attribution` | Provider-patient assignment |
|
| `main` | 14 | Full pipeline orchestration |
|
||||||
| `quality_measures` | Composite quality scores |
|
| `opps` | 2 | Outpatient payment system |
|
||||||
| `data_quality` | Validation and completeness |
|
| `pharmacy` | 4 | Part D drug utilization |
|
||||||
| `cclf` | CCLF file parsing |
|
| `provider_attribution` | 10 | Provider-patient assignment |
|
||||||
| `main` | Full pipeline orchestration |
|
| `quality_measures` | 15 | Composite quality scores |
|
||||||
|
| `readmissions` | 12 | 30-day readmission rates |
|
||||||
|
|
||||||
### Expression layer
|
### Expression layer
|
||||||
|
|
||||||
@@ -154,7 +155,7 @@ The generator reads image definitions from `stack.toml [images]`, dispatches to
|
|||||||
|
|
||||||
### Workflows
|
### Workflows
|
||||||
|
|
||||||
Seven workflows generated per backend:
|
7 workflows generated per backend:
|
||||||
|
|
||||||
| Workflow | Trigger | What it does |
|
| Workflow | Trigger | What it does |
|
||||||
|----------|---------|--------------|
|
|----------|---------|--------------|
|
||||||
@@ -193,7 +194,7 @@ Daily automated pipeline (`pkg-supply-chain.yml`) that:
|
|||||||
|
|
||||||
## Credential management
|
## Credential management
|
||||||
|
|
||||||
All 19 service credentials derive from a single 256-bit root key via HKDF-SHA256. No passwords stored in `.env` — they regenerate deterministically from root key + commit SHA on each deploy.
|
All 20 service credentials derive from a single 256-bit root key via HKDF-SHA256. No passwords stored in `.env` — they regenerate deterministically from root key + commit SHA on each deploy.
|
||||||
|
|
||||||
```
|
```
|
||||||
ROOT_KEY + commit_sha -> HKDF-SHA256 -> all credentials -> .env + backends
|
ROOT_KEY + commit_sha -> HKDF-SHA256 -> all credentials -> .env + backends
|
||||||
@@ -256,7 +257,7 @@ Switch with `STACK_CONTEXT=aws` or edit `stack.toml [context] active`.
|
|||||||
|
|
||||||
## Databricks
|
## Databricks
|
||||||
|
|
||||||
Databricks Asset Bundle (`databricks.yml`) is auto-generated from the pipeline registry. 13 pipeline tasks with dependency ordering, 3 targets (dev/staging/prod), daily schedule.
|
Databricks Asset Bundle (`databricks.yml`) is auto-generated from the pipeline registry. 14 pipeline tasks with dependency ordering, 3 targets (dev/staging/prod), daily schedule.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run python dev/scripts/gen_config.py --dab-sql # generate SQL + DDL (~4 min)
|
uv run python dev/scripts/gen_config.py --dab-sql # generate SQL + DDL (~4 min)
|
||||||
@@ -336,7 +337,7 @@ The hook classifies staged files and runs only what is relevant:
|
|||||||
| `src/sem/*.py` | `tests/sem/` + `test_ast_coverage.py` | Module tests + structural invariants |
|
| `src/sem/*.py` | `tests/sem/` + `test_ast_coverage.py` | Module tests + structural invariants |
|
||||||
| `src/aco/*.py` + `src/sem/*.py` | `tests/aco/` + `tests/sem/` + structural | Both module test dirs |
|
| `src/aco/*.py` + `src/sem/*.py` | `tests/aco/` + `tests/sem/` + structural | Both module test dirs |
|
||||||
| `tests/bib/test_sync.py` | `tests/bib/` | Changed test dir |
|
| `tests/bib/test_sync.py` | `tests/bib/` | Changed test dir |
|
||||||
| `pyproject.toml` or `conftest.py` | Full suite (12k+ tests) | Infrastructure change |
|
| `pyproject.toml` or `conftest.py` | Full suite (12,291+ tests) | Infrastructure change |
|
||||||
| `notebooks/pfs_calcs.py` | `marimo check` + notebook execution | Notebook validation only |
|
| `notebooks/pfs_calcs.py` | `marimo check` + notebook execution | Notebook validation only |
|
||||||
| `README.md` only | Nothing | No testable changes |
|
| `README.md` only | Nothing | No testable changes |
|
||||||
|
|
||||||
@@ -376,24 +377,24 @@ uv run python -m pytest tests/test_ast_coverage.py # structural invariants only
|
|||||||
|
|
||||||
```
|
```
|
||||||
stack/
|
stack/
|
||||||
├── compose.yml Docker Compose (22 services)
|
├── compose.yml Docker Compose (25 services)
|
||||||
├── stack.toml Centralised configuration
|
├── stack.toml Centralised configuration
|
||||||
├── pyproject.toml Python project (uv, optional deps per module)
|
├── pyproject.toml Python project (uv, optional deps per module)
|
||||||
├── src/
|
├── src/
|
||||||
│ ├── aco/ ACO analytics (express, pipe, table, lake, load)
|
│ ├── aco/ ACO analytics (express, pipe, table, lake, load)
|
||||||
│ ├── api/ FastAPI server + auth + diag
|
│ ├── api/ FastAPI server + auth + diag
|
||||||
│ ├── bcda/ BCDA FHIR R4 client
|
│ ├── bcda/ BCDA FHIR R4 client
|
||||||
│ ├── bib/ Zotero bibliography store, tags, metadata
|
│ ├── bib/ Zotero bibliography store, tags, metadata
|
||||||
│ ├── bls/ BLS data
|
│ ├── bls/ BLS data
|
||||||
│ ├── ccw/ CCW data dictionary
|
│ ├── ccw/ CCW data dictionary
|
||||||
│ ├── cli/ CLI entry point (typer)
|
│ ├── cli/ CLI entry point (typer)
|
||||||
│ ├── cms/ CMS public data tables
|
│ ├── cms/ CMS public data tables
|
||||||
│ ├── conf/ Config loader, storage abstraction, table base
|
│ ├── conf/ Config loader, storage abstraction, table base
|
||||||
│ ├── opps/ Outpatient Prospective Payment System
|
│ ├── opps/ Outpatient Prospective Payment System
|
||||||
│ ├── perf/ Pipeline telemetry (OpenTelemetry)
|
│ ├── perf/ Pipeline telemetry (OpenTelemetry)
|
||||||
│ ├── pfs/ Physician Fee Schedule
|
│ ├── pfs/ Physician Fee Schedule
|
||||||
│ ├── rex/ REX fixed-width file processing
|
│ ├── rex/ REX fixed-width file processing
|
||||||
│ └── sem/ Semantic coverage orchestration
|
│ └── sem/ Semantic coverage orchestration
|
||||||
├── infra/ Service configs and Dockerfiles
|
├── infra/ Service configs and Dockerfiles
|
||||||
│ ├── images/ All Dockerfiles (api, notebooks, zotero, docs, mc)
|
│ ├── images/ All Dockerfiles (api, notebooks, zotero, docs, mc)
|
||||||
│ ├── traefik/ Reverse proxy + loch CSS injection
|
│ ├── traefik/ Reverse proxy + loch CSS injection
|
||||||
|
|||||||
429
README.md.j2
Normal file
429
README.md.j2
Normal file
@@ -0,0 +1,429 @@
|
|||||||
|
{# README.md.j2 — source of truth for README.md
|
||||||
|
Render with: uv run python dev/scripts/gen_readme.py
|
||||||
|
Check freshness: uv run python dev/scripts/gen_readme.py --check
|
||||||
|
#}
|
||||||
|
# Stack
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
Healthcare analytics platform on self-hosted infrastructure. Replaces dbt SQL models with narwhals DataFrame-agnostic expression functions, backed by DuckDB locally and Iceberg/Trino in the lakehouse. {{ services.count }} services, {{ pipelines.count }} data pipelines, {{ tests.formatted }} tests at 99% coverage.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Generate root key and seed .env
|
||||||
|
ROOT_KEY=$(openssl rand -hex 32)
|
||||||
|
cat > .env <<EOF
|
||||||
|
DOMAIN=homelab.fhirworx.io
|
||||||
|
HOST_IP=192.168.1.192
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# 2. Start core services and bootstrap credentials
|
||||||
|
docker compose up -d postgres rustfs traefik
|
||||||
|
ROOT_KEY=$ROOT_KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)
|
||||||
|
|
||||||
|
# 3. Start everything
|
||||||
|
docker compose up -d
|
||||||
|
|
||||||
|
# 4. Verify
|
||||||
|
curl -s http://localhost:8000/health | python3 -m json.tool
|
||||||
|
```
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
The package supports skinny installs — install only the modules you need:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install stack[aco] # ACO analytics only
|
||||||
|
pip install stack[bib] # bibliography only
|
||||||
|
pip install stack[cli] # CLI (pulls aco + api + bib)
|
||||||
|
pip install stack[all] # everything
|
||||||
|
pip install stack[aco,aws] # ACO analytics with AWS storage
|
||||||
|
```
|
||||||
|
|
||||||
|
{{ modules.count }} modules available as optional extras: {{ modules.names | map('backtick') | join(', ') }}. Cloud providers: {{ modules.cloud | map('backtick') | join(', ') }}. Aggregates: {{ modules.aggregates | map('backtick') | join(', ') }}.
|
||||||
|
|
||||||
|
## Services
|
||||||
|
|
||||||
|
All services route through Traefik at `*.homelab.fhirworx.io`.
|
||||||
|
|
||||||
|
| Service | URL | Purpose |
|
||||||
|
|---------|-----|---------|
|
||||||
|
| API | `api` (internal) | FastAPI — health, pipelines, bib endpoints |
|
||||||
|
| Notebooks | `notebooks.homelab.fhirworx.io` | Marimo notebooks (GPU-accelerated) |
|
||||||
|
| Docs | `docs.homelab.fhirworx.io` | Docusaurus — API reference + CMS bibliography |
|
||||||
|
| Gitea | `gitea.homelab.fhirworx.io` | Git server, container registry, LFS |
|
||||||
|
| Zotero | `zotero.homelab.fhirworx.io` | Reference manager (KasmVNC desktop) |
|
||||||
|
| Dashboard | `homelab.fhirworx.io` | Service launcher |
|
||||||
|
| RustFS | `s3.homelab.fhirworx.io` | S3-compatible object storage |
|
||||||
|
| Nessie | `nessie.homelab.fhirworx.io` | Git-like Iceberg catalog |
|
||||||
|
| Polaris | `polaris.homelab.fhirworx.io` | Iceberg catalog with governance |
|
||||||
|
| Trino | `trino.homelab.fhirworx.io` | Distributed SQL engine |
|
||||||
|
| Grafana | `grafana.homelab.fhirworx.io` | Dashboards |
|
||||||
|
| Prometheus | `prometheus.homelab.fhirworx.io` | Metrics |
|
||||||
|
| Jaeger | `jaeger.homelab.fhirworx.io` | Distributed tracing |
|
||||||
|
| Loki | `loki.homelab.fhirworx.io` | Log aggregation |
|
||||||
|
|
||||||
|
## Analytics platform
|
||||||
|
|
||||||
|
### Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
CCLF/BCDA/CMS data
|
||||||
|
|
|
||||||
|
v
|
||||||
|
src/aco/express/ narwhals expressions (DataFrame-agnostic)
|
||||||
|
|
|
||||||
|
v
|
||||||
|
src/aco/pipe/ pipeline runner with schema validation
|
||||||
|
|
|
||||||
|
v
|
||||||
|
DuckDB (local) --or-- Iceberg + Trino (lakehouse) --or-- Databricks
|
||||||
|
|
|
||||||
|
v
|
||||||
|
src/api/ FastAPI (health, runs, bib)
|
||||||
|
|
|
||||||
|
v
|
||||||
|
notebooks/ Marimo interactive analysis
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pipelines
|
||||||
|
|
||||||
|
{{ pipelines.count }} registered pipelines with {{ pipelines.total_steps }} steps, validated by `run_pipeline(steps, load)` which enforces output schemas via pandera-style column checks.
|
||||||
|
|
||||||
|
| Pipeline | Steps | Domain |
|
||||||
|
|----------|------:|--------|
|
||||||
|
{% for p in pipelines.list %}
|
||||||
|
| `{{ p.name }}` | {{ p.steps }} | {{ p.doc }} |
|
||||||
|
{% endfor %}
|
||||||
|
|
||||||
|
### Expression layer
|
||||||
|
|
||||||
|
`src/aco/express/` contains narwhals functions that work across polars, pandas, and cuDF. Each module mirrors a pipeline and exports pure transformation functions.
|
||||||
|
|
||||||
|
### Configuration
|
||||||
|
|
||||||
|
`stack.toml` is the single config file. `src/conf/` provides:
|
||||||
|
|
||||||
|
- `cfg` — attribute access (`cfg.db.aco`)
|
||||||
|
- `path("db.aco")` — absolute path resolution relative to repo root
|
||||||
|
- `context()` — environment switching (local/lake/databricks/trino/aws/gcp/azure)
|
||||||
|
- `secret("api.secret", "STACK_API_SECRET")` — env var with config fallback
|
||||||
|
- `storage.get_filesystem()` — multi-cloud storage abstraction (S3/GCS/ABFS/local)
|
||||||
|
|
||||||
|
Override context at runtime: `STACK_CONTEXT=lake` (or `aws`, `gcp`, `azure`).
|
||||||
|
|
||||||
|
### Bibliography
|
||||||
|
|
||||||
|
`src/bib/` manages a Zotero-backed citation store in `data/bib.sqlite`. Column-level provenance via `Tag.col(ref, desc)`. The docs site exports `library.json` for a searchable bibliography browser at `/library`.
|
||||||
|
|
||||||
|
## CI/CD
|
||||||
|
|
||||||
|
### Backend switching
|
||||||
|
|
||||||
|
CI/CD workflows are auto-generated from `stack.toml` by `gen_config.py`. Three backends are supported:
|
||||||
|
|
||||||
|
| Backend | Workflows directory | When to use |
|
||||||
|
|---------|---------------------|-------------|
|
||||||
|
| `gitea` | `.gitea/workflows/` | Self-hosted Gitea instance (default) |
|
||||||
|
| `github` | `.github/workflows/` | GitHub repos / GitHub Actions |
|
||||||
|
| `woodpecker` | `.woodpecker/` | Woodpecker CI server |
|
||||||
|
|
||||||
|
Switch backends by editing `stack.toml`:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[ci]
|
||||||
|
backend = "gitea" # change to "github" or "woodpecker"
|
||||||
|
```
|
||||||
|
|
||||||
|
Then regenerate:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python dev/scripts/gen_config.py # generate new workflows
|
||||||
|
uv run python dev/scripts/gen_config.py --check # verify (used in CI)
|
||||||
|
uv run python dev/scripts/gen_config.py --backend github # one-off override
|
||||||
|
```
|
||||||
|
|
||||||
|
The generator reads image definitions from `stack.toml [images]`, dispatches to the active backend emitter (`dev/scripts/backends/{backend}.py`), writes workflow YAML, and auto-removes stale workflows from inactive backend directories.
|
||||||
|
|
||||||
|
### Workflows
|
||||||
|
|
||||||
|
{{ workflows.count }} workflows generated per backend:
|
||||||
|
|
||||||
|
| Workflow | Trigger | What it does |
|
||||||
|
|----------|---------|--------------|
|
||||||
|
| `ci.yml` | Push/PR | Ruff lint + format, pytest (99% coverage), gen_config --check, skinny-install matrix |
|
||||||
|
| `deploy.yml` | Push to main | Build 5 images, Trivy scan, vuln reporting |
|
||||||
|
| `harden.yml` | Weekly cron / manual | Rebuild `--no-cache`, scan, auto-close/file vuln issues |
|
||||||
|
| `rebuild-all.yml` | Manual | Full rebuild + scan + deploy |
|
||||||
|
| `infra-ci.yml` | Path-filtered push | Hadolint Dockerfiles, dry-run builds |
|
||||||
|
| `release.yml` | Tag (`v*`) | `uv build` + release |
|
||||||
|
| `pkg-supply-chain.yml` | Daily / push | Package inventory, mirror sync, drift detection, vuln scan |
|
||||||
|
|
||||||
|
### Secret mapping
|
||||||
|
|
||||||
|
| Secret | Gitea | GitHub | Woodpecker |
|
||||||
|
|--------|-------|--------|------------|
|
||||||
|
| Registry auth | `REGISTRY_USER` + `REGISTRY_TOKEN` | `GITHUB_TOKEN` (built-in) | `registry_username` + `registry_password` |
|
||||||
|
| Gitea API | `GITEA_TOKEN` | N/A | `gitea_token` |
|
||||||
|
| Vuln reporting | `GITEA_TOKEN` | `GITHUB_TOKEN` | `gitea_token` |
|
||||||
|
|
||||||
|
### Image tagging
|
||||||
|
|
||||||
|
Images are tagged with the short commit SHA (8 chars). Compose resolves via `${COMMIT_SHA:-latest}` from `.env`. Custom images: `api`, `notebooks`, `zotero`, `docs`, `mc`.
|
||||||
|
|
||||||
|
### Vulnerability management
|
||||||
|
|
||||||
|
The `harden.yml` pipeline rebuilds all images with `--no-cache` to pick up OS patches, runs Trivy scans, and auto-files or auto-closes Gitea issues via `api.diag.vuln`.
|
||||||
|
|
||||||
|
### Package supply chain
|
||||||
|
|
||||||
|
Daily automated pipeline (`pkg-supply-chain.yml`) that:
|
||||||
|
1. Scans Dockerfiles, pyproject.toml, and CI workflows for all package dependencies
|
||||||
|
2. Syncs packages to the Gitea package registry (local mirror)
|
||||||
|
3. Detects drift between manifest and mirror
|
||||||
|
4. Runs Trivy vulnerability scans on dependencies
|
||||||
|
5. Auto-creates Gitea issues for missing packages and CVEs
|
||||||
|
|
||||||
|
## Credential management
|
||||||
|
|
||||||
|
All {{ credentials.count }} service credentials derive from a single 256-bit root key via HKDF-SHA256. No passwords stored in `.env` — they regenerate deterministically from root key + commit SHA on each deploy.
|
||||||
|
|
||||||
|
```
|
||||||
|
ROOT_KEY + commit_sha -> HKDF-SHA256 -> all credentials -> .env + backends
|
||||||
|
```
|
||||||
|
|
||||||
|
| Tier | Salt | Rotates |
|
||||||
|
|------|------|---------|
|
||||||
|
| Bootstrap | `b"bootstrap"` | Only when root key changes |
|
||||||
|
| Service | `commit_sha` | Every deploy |
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Bootstrap (first time)
|
||||||
|
ROOT_KEY=$KEY uv run python -m api.auth bootstrap $(git rev-parse HEAD)
|
||||||
|
|
||||||
|
# Rotate (every deploy, automatic in CI)
|
||||||
|
ROOT_KEY=$KEY uv run python -m api.auth provision $COMMIT_SHA
|
||||||
|
|
||||||
|
# Dry-run
|
||||||
|
ROOT_KEY=$KEY uv run python -m api.auth derive $COMMIT_SHA --redact
|
||||||
|
```
|
||||||
|
|
||||||
|
## Network segmentation
|
||||||
|
|
||||||
|
| Network | Type | Services |
|
||||||
|
|---------|------|----------|
|
||||||
|
| `gateway` | external | Traefik, all public-facing |
|
||||||
|
| `storage` | internal | PostgreSQL, RustFS |
|
||||||
|
| `data` | internal | Nessie, Trino, Polaris |
|
||||||
|
| `observability` | internal | Grafana, Prometheus, Jaeger, Loki |
|
||||||
|
| `ci` | external | Gitea, Act Runner, RustFS |
|
||||||
|
|
||||||
|
## Design system
|
||||||
|
|
||||||
|
Theme: **loch** (deep-navy). Traefik's `inject-loch` middleware rewrites HTML to inject `loch.css` and the favicon. Per-service CSS in `assets/css/`. Altair chart theme in `assets/nature.py` (Nature-conformant, accessible palette, IBM Plex Mono headings).
|
||||||
|
|
||||||
|
## Data lakehouse
|
||||||
|
|
||||||
|
| Component | Purpose |
|
||||||
|
|-----------|---------|
|
||||||
|
| Nessie | Git-like branching and time travel for Iceberg tables |
|
||||||
|
| Polaris | Iceberg catalog with RBAC and multi-tenant governance |
|
||||||
|
| Trino | Distributed SQL (`iceberg` catalog) |
|
||||||
|
| RustFS | S3 storage backend (`s3://lakehouse/`) |
|
||||||
|
|
||||||
|
Query via Trino CLI, JDBC, Web UI, PyIceberg, or DuckDB+PyArrow.
|
||||||
|
|
||||||
|
### Cloud deployment
|
||||||
|
|
||||||
|
The platform runs on self-hosted Docker Compose by default. Cloud provider guides in `cloud/`:
|
||||||
|
|
||||||
|
| Provider | Storage | Catalog | Compute | Context |
|
||||||
|
|----------|---------|---------|---------|---------|
|
||||||
|
| Self-hosted | RustFS (S3) | Nessie / Polaris | Docker Compose | `local` / `lake` |
|
||||||
|
| AWS | S3 | Glue | ECS / EKS | `aws` |
|
||||||
|
| GCP | GCS | BigQuery | Cloud Run / GKE | `gcp` |
|
||||||
|
| Azure | ABFS | Unity Catalog | Container Apps / AKS | `azure` |
|
||||||
|
| Databricks | DBFS | Unity Catalog | Databricks Jobs | `databricks` |
|
||||||
|
|
||||||
|
Switch with `STACK_CONTEXT=aws` or edit `stack.toml [context] active`.
|
||||||
|
|
||||||
|
## Databricks
|
||||||
|
|
||||||
|
Databricks Asset Bundle (`databricks.yml`) is auto-generated from the pipeline registry. {{ pipelines.count }} pipeline tasks with dependency ordering, 3 targets (dev/staging/prod), daily schedule.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python dev/scripts/gen_config.py --dab-sql # generate SQL + DDL (~4 min)
|
||||||
|
databricks bundle deploy -t dev # deploy to Databricks
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
### Architecture
|
||||||
|
|
||||||
|
Testing follows a semantic coverage model (`src/sem/`) rather than tracking raw line numbers. The system parses every `.py` file into an AST, builds stable semantic nodes (functions, branches, exception handlers, loops), and attaches three signal layers:
|
||||||
|
|
||||||
|
```
|
||||||
|
source → AST → semantic nodes → attach (ruff, ty, coverage) → planner
|
||||||
|
```
|
||||||
|
|
||||||
|
| Layer | Module | What it does |
|
||||||
|
|-------|--------|-------------|
|
||||||
|
| **Parse** | `sem.parse` | Walk AST, emit `SemanticNode` per function, branch, except, loop, return, raise |
|
||||||
|
| **Enrich** | `sem.enrich` | Run `ruff check` and `ty check`, map diagnostics to tightest-span node |
|
||||||
|
| **Runtime** | `sem.runtime` | Map `coverage.py` JSON report onto nodes (hit/miss, per-test contexts) |
|
||||||
|
| **Plan** | `sem.plan` | Score uncovered nodes by priority, suggest next test targets |
|
||||||
|
| **State** | `sem.state` | Persist node status across runs, reset on source hash change |
|
||||||
|
|
||||||
|
Each node has a stable identity derived from its module path, qualified symbol name, kind, and ordinal position — never from line numbers:
|
||||||
|
|
||||||
|
```
|
||||||
|
app.config::load_settings::branch_if[1]
|
||||||
|
aco.express.pharmacy::pharmacy_claims::except_handler[0]
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Priority heuristic
|
||||||
|
|
||||||
|
```
|
||||||
|
priority =
|
||||||
|
uncovered_branch × 5
|
||||||
|
+ ty_diagnostic × 4
|
||||||
|
+ inside_partially_tested_function × 3
|
||||||
|
+ uncovered_exception_path × 3
|
||||||
|
+ ruff_warning × 2
|
||||||
|
− prior_failures × 2
|
||||||
|
− already_covered × 5
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Structural invariants
|
||||||
|
|
||||||
|
`tests/test_ast_coverage.py` runs at collection time (pure AST, no imports) and enforces:
|
||||||
|
|
||||||
|
1. Every `@nw.narwhalify` function has a docstring
|
||||||
|
2. Every `Expr(...)` call supplies `name`, `fn`, `output`, and `after`
|
||||||
|
3. Every public Pydantic model is importable
|
||||||
|
4. Every `pipe/*.py` module exports a `Pipeline` with non-empty `.exprs`
|
||||||
|
5. Implementation ratio stays above baseline (ratchet)
|
||||||
|
6. `express/` and `pipe/` modules are symmetric
|
||||||
|
|
||||||
|
### Pre-commit hooks
|
||||||
|
|
||||||
|
Git hooks live in `dev/hooks/` (tracked) and are activated via:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git config core.hooksPath dev/hooks
|
||||||
|
```
|
||||||
|
|
||||||
|
The shell hook is a 3-line wrapper. All logic lives in `src/sem/hooks.py`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
exec uv run python -m sem.hooks
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Smart test selection
|
||||||
|
|
||||||
|
The hook classifies staged files and runs only what is relevant:
|
||||||
|
|
||||||
|
| What changed | Tests run | Why |
|
||||||
|
|-------------|-----------|-----|
|
||||||
|
| `src/sem/*.py` | `tests/sem/` + `test_ast_coverage.py` | Module tests + structural invariants |
|
||||||
|
| `src/aco/*.py` + `src/sem/*.py` | `tests/aco/` + `tests/sem/` + structural | Both module test dirs |
|
||||||
|
| `tests/bib/test_sync.py` | `tests/bib/` | Changed test dir |
|
||||||
|
| `pyproject.toml` or `conftest.py` | Full suite ({{ tests.formatted }}+ tests) | Infrastructure change |
|
||||||
|
| `notebooks/pfs_calcs.py` | `marimo check` + notebook execution | Notebook validation only |
|
||||||
|
| `README.md` only | Nothing | No testable changes |
|
||||||
|
|
||||||
|
The mapping rule is: `src/<module>/` changes → `tests/<module>/` runs. Any `src/` change also triggers `test_ast_coverage.py` to verify structural invariants haven't regressed.
|
||||||
|
|
||||||
|
#### Forcing full suite
|
||||||
|
|
||||||
|
```bash
|
||||||
|
GIT_PRE_COMMIT_FULL=1 git commit -m "message"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Hook steps (in order)
|
||||||
|
|
||||||
|
1. **Venv recovery** — if `uv run python -c 'import sys'` fails, run `uv sync --dev`
|
||||||
|
2. **Config regeneration** — if `stack.toml` or `gen_config.py` changed, regenerate CI workflows
|
||||||
|
3. **Ruff lint + format** — only staged `.py` files
|
||||||
|
4. **AST parse check** — verify staged source files have valid syntax
|
||||||
|
5. **Pytest** — targeted or full suite based on what changed
|
||||||
|
6. **Marimo check** — only if notebooks are staged
|
||||||
|
7. **Notebook execution** — only staged notebooks in the safe-to-run list
|
||||||
|
|
||||||
|
#### Post-commit
|
||||||
|
|
||||||
|
`dev/hooks/post-commit` rebuilds the docs Docker image in the background after every commit so the documentation site stays current with docstring changes.
|
||||||
|
|
||||||
|
### Running tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python -m pytest tests/ # full suite
|
||||||
|
uv run python -m pytest tests/sem/ # one module
|
||||||
|
uv run python -m pytest tests/ -m "not stub" # skip stub inventory
|
||||||
|
uv run python -m pytest tests/ -m stub # only stub status
|
||||||
|
uv run python -m pytest tests/test_ast_coverage.py # structural invariants only
|
||||||
|
```
|
||||||
|
|
||||||
|
## Project layout
|
||||||
|
|
||||||
|
```
|
||||||
|
stack/
|
||||||
|
├── compose.yml Docker Compose ({{ services.count }} services)
|
||||||
|
├── stack.toml Centralised configuration
|
||||||
|
├── pyproject.toml Python project (uv, optional deps per module)
|
||||||
|
├── src/
|
||||||
|
{% for m in src_modules %}
|
||||||
|
│ {{ "├" if not loop.last else "└" }}── {{ m.name }}/{{ " " * (25 - m.name|length) }}{{ m.desc }}
|
||||||
|
{% endfor %}
|
||||||
|
├── infra/ Service configs and Dockerfiles
|
||||||
|
│ ├── images/ All Dockerfiles (api, notebooks, zotero, docs, mc)
|
||||||
|
│ ├── traefik/ Reverse proxy + loch CSS injection
|
||||||
|
│ ├── grafana/ Dashboards and datasource provisioning
|
||||||
|
│ ├── prometheus/ Metrics collection
|
||||||
|
│ ├── loki/ Log aggregation
|
||||||
|
│ └── ... coredns, nginx, trino, polaris, rustfs, act-runner, gitea
|
||||||
|
├── assets/ Branding, styles, generated artifacts
|
||||||
|
│ ├── css/ Per-service CSS (dashboard, gitea, marimo, woodpecker)
|
||||||
|
│ ├── icons/ Favicons, logos, coverage badge
|
||||||
|
│ └── nature.py Nature-conformant chart palette
|
||||||
|
├── cloud/ Cloud provider deployment guides
|
||||||
|
│ ├── self-hosted/ Docker Compose (default)
|
||||||
|
│ ├── aws/ S3, RDS, Glue, ECS
|
||||||
|
│ ├── gcp/ GCS, Cloud SQL, BigQuery, Cloud Run
|
||||||
|
│ └── azure/ ABFS, Azure SQL, Unity Catalog, Container Apps
|
||||||
|
├── dev/ Dev tooling
|
||||||
|
│ ├── scripts/ Code generators, bootstrap, supply chain tools
|
||||||
|
│ ├── hooks/ Git hooks (tracked, core.hooksPath = dev/hooks)
|
||||||
|
│ ├── seeds/ Reference data (BCDA samples, CMS docs)
|
||||||
|
│ └── pipelines/ CI-agnostic pipeline specs
|
||||||
|
├── tests/ {{ tests.formatted }} tests at 99% coverage
|
||||||
|
├── notebooks/ Marimo notebooks
|
||||||
|
├── docs/ Docusaurus site
|
||||||
|
├── data/ DuckDB, bib.sqlite, BCDA/CMS data, zotero (gitignored)
|
||||||
|
└── .env Derived credentials (not in git)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- Docker (rootless mode)
|
||||||
|
- NVIDIA GPU with container toolkit
|
||||||
|
- Domain or `/etc/hosts` entries for `*.homelab.fhirworx.io`
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Fix for rootless NVIDIA containers
|
||||||
|
sudo sed -i 's/#no-cgroups = false/no-cgroups = true/' /etc/nvidia-container-runtime/config.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
## SSH access
|
||||||
|
|
||||||
|
```
|
||||||
|
# ~/.ssh/config
|
||||||
|
Host gitea
|
||||||
|
HostName homelab.fhirworx.io
|
||||||
|
Port 2222
|
||||||
|
User git
|
||||||
|
IdentityFile ~/.ssh/gitea_ed25519
|
||||||
|
IdentitiesOnly yes
|
||||||
|
```
|
||||||
300
dev/scripts/gen_readme.py
Normal file
300
dev/scripts/gen_readme.py
Normal file
@@ -0,0 +1,300 @@
|
|||||||
|
"""Generate README.md from README.md.j2 using live codebase data.
|
||||||
|
|
||||||
|
Collects counts, tables, and metadata from the actual codebase:
|
||||||
|
- Service count and names from compose.yml
|
||||||
|
- Module list from pyproject.toml optional-dependencies
|
||||||
|
- Pipeline registry from src/aco/pipe/
|
||||||
|
- Test count via pytest --collect-only
|
||||||
|
- Credential count from api.auth.manifest
|
||||||
|
- Network list from compose.yml
|
||||||
|
- Workflow list from .gitea/workflows/
|
||||||
|
|
||||||
|
Usage::
|
||||||
|
|
||||||
|
uv run python dev/scripts/gen_readme.py # generate README.md
|
||||||
|
uv run python dev/scripts/gen_readme.py --check # verify up-to-date
|
||||||
|
uv run python dev/scripts/gen_readme.py --dry-run # print to stdout
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import ast
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tomllib
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
from jinja2 import Environment, FileSystemLoader
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent.parent.parent
|
||||||
|
|
||||||
|
# ── Data collectors ──────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def _collect_services() -> dict:
|
||||||
|
"""Parse compose.yml for service and network info."""
|
||||||
|
compose = yaml.safe_load((ROOT / "compose.yml").read_text())
|
||||||
|
services = sorted(compose.get("services", {}).keys())
|
||||||
|
networks = sorted(compose.get("networks", {}).keys())
|
||||||
|
return {"names": services, "count": len(services), "networks": networks}
|
||||||
|
|
||||||
|
|
||||||
|
def _collect_modules() -> dict:
|
||||||
|
"""Parse pyproject.toml for module list."""
|
||||||
|
with open(ROOT / "pyproject.toml", "rb") as f:
|
||||||
|
pyp = tomllib.load(f)
|
||||||
|
|
||||||
|
opt_deps = pyp["project"]["optional-dependencies"]
|
||||||
|
cloud = {"aws", "gcp", "azure"}
|
||||||
|
aggregates = {"all", "lake"}
|
||||||
|
modules = sorted(k for k in opt_deps if k not in cloud and k not in aggregates)
|
||||||
|
|
||||||
|
build_modules = (
|
||||||
|
pyp.get("tool", {})
|
||||||
|
.get("uv", {})
|
||||||
|
.get("build-backend", {})
|
||||||
|
.get("module-name", [])
|
||||||
|
)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"names": modules,
|
||||||
|
"count": len(modules),
|
||||||
|
"cloud": sorted(cloud & set(opt_deps)),
|
||||||
|
"aggregates": sorted(aggregates & set(opt_deps)),
|
||||||
|
"build_modules": build_modules,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _collect_pipelines() -> dict:
|
||||||
|
"""Import pipeline registry and collect metadata."""
|
||||||
|
sys.path.insert(0, str(ROOT / "src"))
|
||||||
|
from aco.pipe import registry
|
||||||
|
|
||||||
|
# Fallback descriptions for pipelines without module docstrings
|
||||||
|
fallback_docs = {
|
||||||
|
"ahrq_measures": "AHRQ quality indicators",
|
||||||
|
"cclf": "CCLF file parsing",
|
||||||
|
"claims_preprocessing": "Claim matching, encounter IDs",
|
||||||
|
"cms_quality_measures": "CMS ACO quality metrics",
|
||||||
|
"core": "Encounters, patients, providers",
|
||||||
|
"data_quality": "Validation and completeness",
|
||||||
|
"hcc_suspecting": "HCC risk adjustment",
|
||||||
|
"input_layer": "Raw CCLF/BCDA ingestion",
|
||||||
|
"main": "Full pipeline orchestration",
|
||||||
|
"opps": "Outpatient payment system",
|
||||||
|
"pharmacy": "Part D drug utilization",
|
||||||
|
"provider_attribution": "Provider-patient assignment",
|
||||||
|
"quality_measures": "Composite quality scores",
|
||||||
|
"readmissions": "30-day readmission rates",
|
||||||
|
}
|
||||||
|
|
||||||
|
pipelines = []
|
||||||
|
for name in sorted(registry.keys()):
|
||||||
|
p = registry[name]
|
||||||
|
doc = ""
|
||||||
|
# Try to get the pipeline module's docstring
|
||||||
|
mod_path = ROOT / "src" / "aco" / "pipe" / f"{name}.py"
|
||||||
|
if mod_path.exists():
|
||||||
|
try:
|
||||||
|
tree = ast.parse(mod_path.read_text())
|
||||||
|
raw = ast.get_docstring(tree) or ""
|
||||||
|
doc = raw.split("\n")[0].rstrip(".")
|
||||||
|
except SyntaxError:
|
||||||
|
pass
|
||||||
|
# Use fallback if docstring is empty
|
||||||
|
if not doc:
|
||||||
|
doc = fallback_docs.get(name, "")
|
||||||
|
pipelines.append(
|
||||||
|
{
|
||||||
|
"name": name,
|
||||||
|
"steps": len(p.exprs),
|
||||||
|
"doc": doc,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
total_steps = sum(p["steps"] for p in pipelines)
|
||||||
|
return {
|
||||||
|
"list": pipelines,
|
||||||
|
"count": len(pipelines),
|
||||||
|
"total_steps": total_steps,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _collect_tests() -> dict:
|
||||||
|
"""Run pytest --collect-only to get test count."""
|
||||||
|
result = subprocess.run(
|
||||||
|
["uv", "run", "python", "-m", "pytest", "tests/", "--collect-only", "-q"],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
cwd=str(ROOT),
|
||||||
|
timeout=60,
|
||||||
|
)
|
||||||
|
# Last line: "12291 tests collected in 3.03s"
|
||||||
|
for line in result.stdout.strip().splitlines():
|
||||||
|
if "test" in line and "collected" in line:
|
||||||
|
count = int(line.split()[0])
|
||||||
|
return {"count": count, "formatted": f"{count:,}"}
|
||||||
|
|
||||||
|
return {"count": 0, "formatted": "?"}
|
||||||
|
|
||||||
|
|
||||||
|
def _collect_credentials() -> dict:
|
||||||
|
"""Count credentials from auth manifest."""
|
||||||
|
sys.path.insert(0, str(ROOT / "src"))
|
||||||
|
try:
|
||||||
|
from api.auth.manifest import CREDENTIALS
|
||||||
|
|
||||||
|
return {"count": len(CREDENTIALS)}
|
||||||
|
except ImportError:
|
||||||
|
return {"count": 0}
|
||||||
|
|
||||||
|
|
||||||
|
def _collect_workflows() -> dict:
|
||||||
|
"""List CI workflow files."""
|
||||||
|
workflow_dirs = [
|
||||||
|
ROOT / ".gitea" / "workflows",
|
||||||
|
ROOT / ".github" / "workflows",
|
||||||
|
ROOT / ".woodpecker",
|
||||||
|
]
|
||||||
|
workflows = []
|
||||||
|
for d in workflow_dirs:
|
||||||
|
if d.is_dir():
|
||||||
|
for f in sorted(d.glob("*.yml")):
|
||||||
|
workflows.append({"name": f.name, "path": str(f.relative_to(ROOT))})
|
||||||
|
return {"list": workflows, "count": len(workflows)}
|
||||||
|
|
||||||
|
|
||||||
|
def _collect_src_tree() -> list[dict]:
|
||||||
|
"""Build the src/ module listing with descriptions."""
|
||||||
|
descriptions = {
|
||||||
|
"aco": "ACO analytics (express, pipe, table, lake, load)",
|
||||||
|
"api": "FastAPI server + auth + diag",
|
||||||
|
"bcda": "BCDA FHIR R4 client",
|
||||||
|
"bib": "Zotero bibliography store, tags, metadata",
|
||||||
|
"bls": "BLS data",
|
||||||
|
"ccw": "CCW data dictionary",
|
||||||
|
"cli": "CLI entry point (typer)",
|
||||||
|
"cms": "CMS public data tables",
|
||||||
|
"conf": "Config loader, storage abstraction, table base",
|
||||||
|
"opps": "Outpatient Prospective Payment System",
|
||||||
|
"perf": "Pipeline telemetry (OpenTelemetry)",
|
||||||
|
"pfs": "Physician Fee Schedule",
|
||||||
|
"rex": "REX fixed-width file processing",
|
||||||
|
"sem": "Semantic coverage orchestration",
|
||||||
|
}
|
||||||
|
|
||||||
|
src = ROOT / "src"
|
||||||
|
modules = []
|
||||||
|
for d in sorted(src.iterdir()):
|
||||||
|
if (
|
||||||
|
d.is_dir()
|
||||||
|
and not d.name.startswith(("_", "."))
|
||||||
|
and not d.name.endswith(".egg-info")
|
||||||
|
):
|
||||||
|
modules.append(
|
||||||
|
{
|
||||||
|
"name": d.name,
|
||||||
|
"desc": descriptions.get(d.name, ""),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return modules
|
||||||
|
|
||||||
|
|
||||||
|
def collect_all() -> dict:
|
||||||
|
"""Collect all dynamic data for template rendering."""
|
||||||
|
services = _collect_services()
|
||||||
|
modules = _collect_modules()
|
||||||
|
pipelines = _collect_pipelines()
|
||||||
|
tests = _collect_tests()
|
||||||
|
credentials = _collect_credentials()
|
||||||
|
workflows = _collect_workflows()
|
||||||
|
src_modules = _collect_src_tree()
|
||||||
|
|
||||||
|
return {
|
||||||
|
"services": services,
|
||||||
|
"modules": modules,
|
||||||
|
"pipelines": pipelines,
|
||||||
|
"tests": tests,
|
||||||
|
"credentials": credentials,
|
||||||
|
"workflows": workflows,
|
||||||
|
"src_modules": src_modules,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ── Rendering ────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def render(data: dict) -> str:
|
||||||
|
"""Render README.md.j2 with collected data."""
|
||||||
|
env = Environment(
|
||||||
|
loader=FileSystemLoader(str(ROOT)),
|
||||||
|
keep_trailing_newline=True,
|
||||||
|
trim_blocks=True,
|
||||||
|
lstrip_blocks=True,
|
||||||
|
)
|
||||||
|
env.filters["backtick"] = lambda s: f"`{s}`"
|
||||||
|
template = env.get_template("README.md.j2")
|
||||||
|
return template.render(**data)
|
||||||
|
|
||||||
|
|
||||||
|
# ── CLI ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument(
|
||||||
|
"--check",
|
||||||
|
action="store_true",
|
||||||
|
help="verify README.md matches template output (exit 1 if stale)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--dry-run",
|
||||||
|
action="store_true",
|
||||||
|
help="print rendered output to stdout instead of writing",
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
data = collect_all()
|
||||||
|
rendered = render(data)
|
||||||
|
|
||||||
|
if args.dry_run:
|
||||||
|
print(rendered, end="")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
readme = ROOT / "README.md"
|
||||||
|
|
||||||
|
if args.check:
|
||||||
|
current = readme.read_text(encoding="utf-8")
|
||||||
|
if current == rendered:
|
||||||
|
print("README.md is up-to-date.")
|
||||||
|
return 0
|
||||||
|
else:
|
||||||
|
# Show first differing line
|
||||||
|
cur_lines = current.splitlines()
|
||||||
|
ren_lines = rendered.splitlines()
|
||||||
|
for i, (a, b) in enumerate(zip(cur_lines, ren_lines), 1):
|
||||||
|
if a != b:
|
||||||
|
print(f"README.md is stale (first diff at line {i}):")
|
||||||
|
print(f" current: {a[:100]}")
|
||||||
|
print(f" expected: {b[:100]}")
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
len_diff = len(cur_lines) - len(ren_lines)
|
||||||
|
print(f"README.md is stale (length differs by {len_diff} lines)")
|
||||||
|
print("\nRun: uv run python dev/scripts/gen_readme.py")
|
||||||
|
return 1
|
||||||
|
|
||||||
|
readme.write_text(rendered, encoding="utf-8")
|
||||||
|
print(
|
||||||
|
f"README.md generated ({data['services']['count']} services, "
|
||||||
|
f"{data['pipelines']['count']} pipelines, "
|
||||||
|
f"{data['tests']['formatted']} tests, "
|
||||||
|
f"{data['modules']['count']} modules)"
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -139,6 +139,7 @@ dev = [
|
|||||||
"pandas>=3.0.1",
|
"pandas>=3.0.1",
|
||||||
"obstore>=0.9.2",
|
"obstore>=0.9.2",
|
||||||
"s3fs>=2026.2.0",
|
"s3fs>=2026.2.0",
|
||||||
|
"jinja2>=3.1.0",
|
||||||
]
|
]
|
||||||
|
|
||||||
[build-system]
|
[build-system]
|
||||||
|
|||||||
2
uv.lock
generated
2
uv.lock
generated
@@ -3229,6 +3229,7 @@ dev = [
|
|||||||
{ name = "coverage" },
|
{ name = "coverage" },
|
||||||
{ name = "dbt-core" },
|
{ name = "dbt-core" },
|
||||||
{ name = "dbt-duckdb" },
|
{ name = "dbt-duckdb" },
|
||||||
|
{ name = "jinja2" },
|
||||||
{ name = "marimo" },
|
{ name = "marimo" },
|
||||||
{ name = "obstore" },
|
{ name = "obstore" },
|
||||||
{ name = "openpyxl" },
|
{ name = "openpyxl" },
|
||||||
@@ -3326,6 +3327,7 @@ dev = [
|
|||||||
{ name = "coverage", specifier = ">=7.13.4" },
|
{ name = "coverage", specifier = ">=7.13.4" },
|
||||||
{ name = "dbt-core", specifier = "==1.10.15" },
|
{ name = "dbt-core", specifier = "==1.10.15" },
|
||||||
{ name = "dbt-duckdb", specifier = ">=1.10,<1.11" },
|
{ name = "dbt-duckdb", specifier = ">=1.10,<1.11" },
|
||||||
|
{ name = "jinja2", specifier = ">=3.1.0" },
|
||||||
{ name = "marimo", specifier = ">=0.20.0" },
|
{ name = "marimo", specifier = ">=0.20.0" },
|
||||||
{ name = "obstore", specifier = ">=0.9.2" },
|
{ name = "obstore", specifier = ">=0.9.2" },
|
||||||
{ name = "openpyxl", specifier = ">=3.1.5" },
|
{ name = "openpyxl", specifier = ">=3.1.5" },
|
||||||
|
|||||||
Reference in New Issue
Block a user