feat: package supply chain — inventory, mirrors, drift, vuln scanning #169

Closed

kert wants to merge 4 commits from feat/pkg-supply-chain into main

pull from: feat/pkg-supply-chain

merge into: homelab:main

homelab:main

homelab:feat/zot-fetch-z1

Conversation 0 Commits 4 Files Changed 18 +1613 -23

kert commented 2026-03-24 03:59:01 +00:00

Owner

Copy Link

Summary

Complete package supply chain using Gitea's built-in package registry as the single mirror for all package types.

Scripts

• pkg_inventory.py — scans Dockerfiles, pyproject.toml, package.json, CI workflows → data/pkg-manifest.json (7 apt, 48 pypi, 11 npm, 3 CI tools, 8 base images)
• pkg_mirror_sync.py — downloads from upstream PyPI/Debian, uploads to Gitea package registry (39 PyPI wheels + 146 Debian .debs uploaded)
• pkg_drift.py — compares manifest against fresh scan, detects phantom/untracked packages
• pkg_issues.py — auto-creates Gitea issues for missing packages and CVEs, deduplicates, auto-closes resolved
• test_network_isolation.sh — builds Dockerfiles with --network=none to verify no internet access

Infrastructure

• Gitea package registry — PyPI + Debian registries enabled, S3-backed storage
• packages.homelab.fhirworx.io — new subdomain via traefik + coredns
• Dockerfiles — PYPI_INDEX_URL build-arg for local registry
• CI workflow — pkg-supply-chain.yml runs daily + on Dockerfile/dep changes: inventory → sync → drift → vuln-scan → auto-issues
• Removed apt-cacher-ng and devpi (replaced by Gitea)

Pipeline spec

• dev/pipelines/pkg-supply-chain.yml — CI-agnostic abstract definition
• Generated as .gitea/workflows/pkg-supply-chain.yml via gen_config.py

Test plan

• pkg_inventory.py produces accurate manifest (verified against manual audit)
• pkg_inventory.py --check exits 0 when manifest is current
• pkg_drift.py exits 0 with no drift on clean state
• pkg_mirror_sync.py --type pypi uploads 39 wheels to Gitea (201 for each)
• pkg_mirror_sync.py --type apt uploads 146 .debs to Gitea (201 for each)
• Gitea serves PyPI simple index (/api/packages/homelab/pypi/simple/pyasn1/)
• Gitea serves Debian Release file (/api/packages/homelab/debian/dists/bookworm/Release)
• pkg_issues.py --dry-run reports no issues when no drift/vulns
• All 11,931 tests pass
• CI workflow generated correctly via gen_config.py

Closes #159, #160, #161, #162, #163, #164, #165, #166, #167, #168

## Summary Complete package supply chain using Gitea's built-in package registry as the single mirror for all package types. ### Scripts - **pkg_inventory.py** — scans Dockerfiles, pyproject.toml, package.json, CI workflows → `data/pkg-manifest.json` (7 apt, 48 pypi, 11 npm, 3 CI tools, 8 base images) - **pkg_mirror_sync.py** — downloads from upstream PyPI/Debian, uploads to Gitea package registry (39 PyPI wheels + 146 Debian .debs uploaded) - **pkg_drift.py** — compares manifest against fresh scan, detects phantom/untracked packages - **pkg_issues.py** — auto-creates Gitea issues for missing packages and CVEs, deduplicates, auto-closes resolved - **test_network_isolation.sh** — builds Dockerfiles with `--network=none` to verify no internet access ### Infrastructure - **Gitea package registry** — PyPI + Debian registries enabled, S3-backed storage - **packages.homelab.fhirworx.io** — new subdomain via traefik + coredns - **Dockerfiles** — `PYPI_INDEX_URL` build-arg for local registry - **CI workflow** — `pkg-supply-chain.yml` runs daily + on Dockerfile/dep changes: inventory → sync → drift → vuln-scan → auto-issues - Removed apt-cacher-ng and devpi (replaced by Gitea) ### Pipeline spec - `dev/pipelines/pkg-supply-chain.yml` — CI-agnostic abstract definition - Generated as `.gitea/workflows/pkg-supply-chain.yml` via gen_config.py ## Test plan - [x] `pkg_inventory.py` produces accurate manifest (verified against manual audit) - [x] `pkg_inventory.py --check` exits 0 when manifest is current - [x] `pkg_drift.py` exits 0 with no drift on clean state - [x] `pkg_mirror_sync.py --type pypi` uploads 39 wheels to Gitea (201 for each) - [x] `pkg_mirror_sync.py --type apt` uploads 146 .debs to Gitea (201 for each) - [x] Gitea serves PyPI simple index (`/api/packages/homelab/pypi/simple/pyasn1/`) - [x] Gitea serves Debian Release file (`/api/packages/homelab/debian/dists/bookworm/Release`) - [x] `pkg_issues.py --dry-run` reports no issues when no drift/vulns - [x] All 11,931 tests pass - [x] CI workflow generated correctly via gen_config.py Closes #159, #160, #161, #162, #163, #164, #165, #166, #167, #168

kert added this to the P21: Package Supply Chain — inventory, mirrors, drift, vuln scanning milestone 2026-03-24 03:59:01 +00:00

kert added 1 commit 2026-03-24 03:59:01 +00:00

add package supply chain scaffold: inventory scanner, drift detector, pipeline spec ... fddb3cce55

refs #159 #165 #168

- pkg_inventory.py: scans all Dockerfiles, pyproject.toml, CI workflows
  for apt/apk/pypi/npm packages and emits data/pkg-manifest.json
- pkg_drift.py: compares manifest against fresh scan to detect phantom
  or untracked packages
- pkg-supply-chain.yml: CI-agnostic pipeline definition covering
  inventory → mirror-sync → drift-check → vuln-scan → auto-issues

kert added 1 commit 2026-03-24 04:03:55 +00:00

add package mirrors, auto-issue scripts, network isolation test, CI workflow ... b184e72f7b

refs #160 #161 #162 #163 #164 #166 #167 #168

- compose.yml: add apt-cacher-ng and devpi services on ci network
- pkg_mirror_sync.py: generates seed files for apt/apk/pypi mirrors
- pkg_issues.py: auto-creates Gitea issues for drift and CVEs,
  deduplicates, auto-closes resolved issues
- test_network_isolation.sh: builds all Dockerfiles with --network=none
- gitea.py: add pkg-supply-chain workflow (daily + on Dockerfile/dep changes)
- mirrors/: seed files for apt (5 pkgs), apk (index), pypi (39 pkgs)

kert added 1 commit 2026-03-24 04:32:26 +00:00

replace devpi/apt-cacher-ng with Gitea package registry ... 66932516a9

closes #160, closes #161, closes #162, closes #163, refs #164

- Remove apt-cacher-ng and devpi containers from compose.yml
- pkg_mirror_sync.py: downloads from upstream, uploads to Gitea's
  built-in PyPI and Debian registries via API
- Dockerfiles: add PYPI_INDEX_URL build-arg for local registry
- CI build steps pass --build-arg PYPI_INDEX_URL for mirror builds
- stack.toml: add [mirrors] section pointing at Gitea endpoints
- Add packages.homelab.fhirworx.io subdomain (traefik + coredns)
- Verified: 39 PyPI wheels + 146 Debian .debs uploaded to Gitea

kert added 1 commit 2026-03-24 04:45:44 +00:00

fix apk mirroring: enumerate installed packages from Alpine images, upload to Gitea ... a463d278dc

closes #161

- Scan nginx:alpine for all 72 installed packages via `apk info`
- Fetch each package individually (handles custom repos gracefully)
- 67 .apk files uploaded to Gitea Alpine registry (v3.23/main)
- 5 nginx modules from custom repos skipped (baked into base image)

kert closed this pull request 2026-03-24 21:07:00 +00:00

kert deleted branch feat/pkg-supply-chain 2026-03-24 21:07:00 +00:00

Some checks are pending

Hide all checks

CI / lint-test (push) Successful in 1m19s

Details

CI / lint-test (pull_request) Successful in 1m30s

Details

Infra CI / zotero (pull_request) Successful in 6s

Details

Infra CI / docs (pull_request) Successful in 6s

Details

Infra CI / api (pull_request) Successful in 7s

Details

Infra CI / mc (pull_request) Successful in 6s

Details

Infra CI / notebooks (pull_request) Successful in 2m48s

Details

ci/woodpecker/push/ci

Required

ci/woodpecker/pr/ci

Required

ci/woodpecker/push/infra-ci

Required

ci/woodpecker/pr/infra-ci

Required

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

api

REST API (FastAPI)

auth

Authentication and authorization

bib

Bibliography, Zotero, citations

ci

CI/CD pipelines and Woodpecker plugins

cli

CLI commands and UX

cloud

Cloud provider abstraction (AWS, GCP, Azure, self-hosted)

config

Configuration and wiring

data

Data loading, CCLF, BCDA, seeds

databricks

Databricks jobs and deployment

docs

Documentation and generated references

image:api

image:docs

image:mc

image:notebooks

image:zotero

infra

lakehouse

Nessie, Polaris, Trino, Iceberg

llm

llm module — local RAG over comments + Zotero corpus

notebooks

marimo notebook errors (auto-filed by nb-watcher/nb-integration)

packaging

Python packaging, optional deps, build system

pfs

pipeline

Pipeline execution and validation

pipeline:ci

pipeline:deploy

pipeline:harden

pipeline:infra-ci

pipeline:rebuild-all

pipeline:release

qpp

quality

Testing, coverage, error handling

step:build

step:deploy

step:hadolint

step:lint

step:publish

step:scan

step:test

step:validate

styles

Charts, figures, visual theming

tree

Project directory structure and file organization

No Label

Milestone

No items

No Milestone P21: Package Supply Chain — inventory, mirrors, drift, vuln scanning

Projects

Clear projects

No project

Assignees

Clear assignees

kert renovate-bot

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#169