observability: remove cadvisor #681

Closed

kert wants to merge 0 commits from chore/remove-cadvisor into main

pull from: chore/remove-cadvisor

merge into: homelab:main

homelab:main

homelab:feat/bcda-v3

homelab:feat/zot-fetch-z1

homelab:feat/zotero-fetch-connector

homelab:feat/zotero-ama-coding-collection

Conversation 0 Commits - Files Changed -

kert commented 2026-09-09 01:19:15 +00:00

Owner

Copy Link

Rootless Docker hides container names from cgroup-based exporters, so this cadvisor only ever emitted anonymous series (152 live, zero with a name label). Every panel that queried it has been blank since it was added.

Removed

• compose.yml: the cadvisor service (privileged, rootfs/sys/docker.sock mounts).
• infra/prometheus/targets/services.yml: the cadvisor:8080 file_sd target.
• tests/test_observability.py: TestCadvisor; the targets test now asserts the job is absent; dropped from PROMTAIL_LABELED. tests/test_compose_mounts.py: dropped from the privileged-exempt set.
• infra/grafana/dashboards/homelab-overview.json: Containers up, Container CPU %, Container memory (MiB) (stat row re-flowed, logs panel moved up). infra/grafana/dashboards/gpu-notebooks.json: Notebooks/Zotero CPU % and memory.

Applied live (2026-09-08): container and gcr.io/cadvisor/cadvisor:latest image removed. Prometheus has already dropped the target (file_sd refresh); Grafana re-provisions the trimmed dashboards on its next scan.

Verification: pytest tests/test_observability.py tests/test_compose_mounts.py → 34 passed; docker compose config renders. The May 2026 plan/spec under docs/superpowers/ still mention cadvisor as history.

Rootless Docker hides container names from cgroup-based exporters, so this `cadvisor` only ever emitted anonymous series (152 live, zero with a `name` label). Every panel that queried it has been blank since it was added. **Removed** - `compose.yml`: the `cadvisor` service (privileged, rootfs/sys/docker.sock mounts). - `infra/prometheus/targets/services.yml`: the `cadvisor:8080` file_sd target. - `tests/test_observability.py`: `TestCadvisor`; the targets test now asserts the job is absent; dropped from `PROMTAIL_LABELED`. `tests/test_compose_mounts.py`: dropped from the privileged-exempt set. - `infra/grafana/dashboards/homelab-overview.json`: *Containers up*, *Container CPU %*, *Container memory (MiB)* (stat row re-flowed, logs panel moved up). `infra/grafana/dashboards/gpu-notebooks.json`: *Notebooks/Zotero CPU %* and *memory*. **Applied live (2026-09-08):** container and `gcr.io/cadvisor/cadvisor:latest` image removed. Prometheus has already dropped the target (file_sd refresh); Grafana re-provisions the trimmed dashboards on its next scan. **Verification:** `pytest tests/test_observability.py tests/test_compose_mounts.py` → 34 passed; `docker compose config` renders. The May 2026 plan/spec under `docs/superpowers/` still mention cadvisor as history.

kert added 40 commits 2026-09-09 01:19:15 +00:00

fix(bib): fetch-pfs-comments --docket; scope the CMS-2026-2377 re-farm to its own docket (refs #615) ... 86b44d9ae8

The re-farm script's first step walked all 12 PFS dockets (~93k comments,
~5 h of API budget) before reaching CMS-2026-2377. fetch-docket-comments
is not a substitute: it omits the reg-docket:<id> tag that
backfill-comments --docket filters on. Add --docket to fetch-pfs-comments
so tagging stays identical and the other dockets are skipped.

chore(comments): re-farm CMS-2026-2377 mirror-first, API steps as gap-fill (refs #615 #662) ... 832bfd511f

The API backfill enriches ~970 comments/hr, so 15k new comments meant a
next-day index. The Mirrulations mirror is a superset of the live docket
(18,651 vs 18,647 comments on 2026-09-08) and runs ~600/min, creating,
tagging and enriching items identically. Run mirror → extract → index
first, then the API list walk + backfill + incremental extract/index to
catch anything the mirror lags on.

docs(spec): comment pipeline docket seals + fingerprint-first skips — never redo known-complete work (refs #615) 5d3e65a009

docs(plan): comment pipeline seal-and-skip implementation plan (refs #615) 2c72f0c4df

chore: ignore .claude/worktrees e829159891

chore: ignore .claude/worktrees dea26fd638

feat(bib): Docket model, should_seal, fingerprint_files, seal_quiet_days knob (refs #615) 5c45dddc09

feat(bib): dockets table + Store.docket_* CRUD; index items.url (refs #615) e1439b7381

feat(bib): Store.upsert_status — identical re-upsert writes nothing (refs #615) 45c6fb2932

fix(bib): attach_file is idempotent on (item, filename) (refs #615) 1cacd0cce8

style(tests): ruff-format the attach idempotency test (refs #615) a6891a541c

feat(bib): dedupe_attachments migration + guarded unique index on (item, filename) (refs #615) 7be6ca8157

feat(bib): iter_comments since= watermark + on_error hook (refs #615) 272a5f3bfd

docs(plan): Task 3b data-preserving upsert + abstract recovery step after the 2026-09-08 clobber (refs #615) 23370e604c

feat(bib): discover_docket + walk_docket — watermark, counts, auto-seal (refs #615) 961e294794

fix(bib): upsert never blanks a stored value — empty incoming columns keep the stored ones (refs #615) 5936cb8d42

feat(cli): fetch commands walk from watermark, skip sealed dockets before any API call (refs #615) fb335605d5

fix(cli): make test_skip_bad_rule use a real Store so it exercises the federal_register exception path (refs #615) 0ade9bee41

feat(bib): backfill-comments skips sealed dockets; --force override (refs #615) 5ff3a5e5bf

feat(comments): mtime-based extract currency; skipped dirs are never read; skip_dockets + reattach (refs #615) e7287c827f

feat(cli): stack comments dockets/seal/unseal; extract skips sealed dockets, lazy bib lookup, --reattach (refs #615) 8b81beda14

fix(cli): close bib read connections in comments extract; test store fixture (refs #615) ... 8453b8a1f9

Fix round 1: the sealed-docket test file's _store() helper never closed
its in-memory Store, and extract's file-backed lookup connection was
never closed either — both leaked sqlite3 connections that surfaced as
nondeterministic ResourceWarnings when GC'd during unrelated tests.
Replace _store() with a store fixture that closes on teardown, and have
extract() close the private read connection _build_bib_helpers opens
for file-backed stores (the :memory: case still reuses store's own
connection, left untouched). Also reworded the _build_bib_helpers
docstring: attach_callback is None for --no-bib or --no-attach; the
store is None only for --no-bib.

feat(llm): index_state.fingerprint + index_docket_state migrations (refs #615) 3bc7c28077

feat(llm): lazy DocRefs with fingerprints, single-query listings, lazy Zotero snapshot (refs #615) fc18ffc1b9

fix(bib): tag/collection changes stamp updated_at (refs #615) 118032d974

fix(llm): rule fingerprint includes updated_at; Zotero snapshot honours the WAL (refs #615) b9c353c41a

feat(llm): fingerprint-first index_refs; PDFs opened only when embedding; docket completion (refs #615) 414b416bd6

fix(llm): fold fingerprint/hash skip counts into index_docs skipped (refs #615) 74d72f7d93

feat(cli): stack llm index lists only unsealed/incomplete dockets, lazy Zotero snapshot (refs #615) bac63363c6

chore(comments): re-farm script is the plain chain now that every stage is incremental (refs #615) d7fd23f83a

test(bib): move fetch-loop coverage to walk_docket; drop obsolete cli coverage class (refs #615) 3df7a3ff0e

fix(llm): migrate before docket_complete; completion requires zero unindexable items (refs #615) ... 066b304111

`stack llm index` read index_docket_state before anything had created
it: migrate() only ran inside index_refs, so the first sealed run on an
un-migrated database died with UndefinedTable. Build one engine in the
CLI, migrate it, and hand it down (index_refs takes an `engine` kwarg);
--force still builds nothing.

index_refs also marked a sealed docket complete when some of its refs
loaded blank or produced no chunks — a seal claiming coverage the run
never achieved. Count unindexable refs per docket and write
index_docket_state only for dockets with none.

fix(bib): walk_docket never seals an empty docket; --force re-fetches attachments (refs #615) ... 32c06ad204

An empty walk over a docket bib never ingested (a farm that never
landed) satisfied should_seal and sealed it shut with zero comments.
Read docket_counts before sealing and require at least one stored
comment; the counts read is reused for the seal row.

--force also skipped attachments for every comment whose row came back
"unchanged", so the one flag meant to repair a docket could not repair
its missing files.

fix(dev): dedupe unlinks after commit; report removed keys (refs #615) ... 4e9dc3d3d8

apply() deleted storage copies inside the BEGIN/COMMIT, so a failure
partway through rolled the rows back and left the surviving rows
pointing at files that were already gone. Collect the doomed paths
during the transaction, COMMIT, then unlink — rows and files can now
only be lost together.

--apply also prints how many keys it removed (first 20), and main()
closes the connection it opens.

fix: is_current stat guard, lazy Zotero utime only on success, docs (refs #615) ... e4fb0108d6

is_current() stat'd every source file bare, so a file removed between
iterdir() and stat() aborted the whole extract run; a vanished source
now just doesn't affect currency.

The lazy Zotero snapshot stamped the snapshot with the source's mtime
even when shutil.copy2 had failed, which would have passed a stale
snapshot off as current on every later run. Copying moved into
_copy_snapshot() so both callers know whether it worked.

Docs: upsert_status spells out the merge rules (extra_json exempt from
empty-keeps-stored, tags/collections union-merged) and
`backfill-comments --force` says it bypasses the docket seal only.

test: drop dead TestCliBibFetchPfsComments coverage class (refs #615) ... 6ca54a6da9

The class drove `fetch-pfs-comments` against a MagicMock store to cover
line numbers that no longer exist — the same defect as the class removed
in 3df7a3f. A mock store asserts nothing about the fetch loop, which now
lives in walk_docket and is covered by tests/bib/test_walk_docket.py and
tests/cli/test_bib_fetch_sealed.py.

fix(llm): empty-text items are stamped as final and count toward docket completion (refs #615) ... ed55a8f883

Treating a blank load() as unindexable defeated the feature it was
meant to protect: four of the eleven historical dockets hold 1-2
comments with no text at all, so none of them could ever be marked
complete and ~130k items would be re-listed on every run.

An item with no text is a finished state, not a failure. Stamp it into
index_state with content_hash='' / chunk_count=0 and its fingerprint —
the empty hash cannot collide with a real sha256 — so the next run
fingerprint-skips it without a load, and it no longer blocks its
docket's completion. Text appearing later changes the fingerprint and
the item is embedded normally.

Only a ref that loaded *with* text and chunked to nothing stays
unindexable and blocks index_docket_state; the embed/write path already
aborts the whole run on error. The index_state upsert both callers need
is now _record_state().

merge: P47 comment pipeline docket seals + fingerprint-first skips — never redo known-complete work (refs #615 #680) db8193cc79

fix(bib): discover_docket prefers the Proposed Rule document over a later-closing final rule (refs #615 #680) e331fc7e4d

chore(observability): remove cadvisor ... caf43846c5

Rootless Docker hides container names from cgroup-based exporters, so
cadvisor only ever produced anonymous series here (152 of them, none
with a name label). The five dashboard panels that queried it have been
blank since it was added; drop the service, its scrape target, the
tests that pinned it, and those panels.

kert added 1 commit 2026-09-09 01:23:39 +00:00

docs: drop the cadvisor task and mentions from the observability plan and spec aadf037b5e

kert added 1 commit 2026-09-09 01:23:52 +00:00

docs: last mention in the observability plan architecture note d0d80c4f47

kert closed this pull request 2026-09-09 14:39:11 +00:00

kert deleted branch chore/remove-cadvisor 2026-09-09 14:39:11 +00:00

Some checks are pending

Hide all checks

CI / lint (pull_request) Successful in 36s

Details

CI / notebooks-smoke (pull_request) Successful in 1m23s

Details

CI / lint (push) Successful in 30s

Details

CI / test (pull_request) Successful in 17m32s

Details

CI / notebooks-smoke (push) Successful in 1m20s

Details

Infra CI / notebooks (pull_request) Successful in 53s

Details

Infra CI / zotero (pull_request) Successful in 14s

Details

Infra CI / docs (pull_request) Successful in 20s

Details

Infra CI / api (pull_request) Successful in 13s

Details

Infra CI / llm (pull_request) Successful in 13s

Details

Infra CI / mc (pull_request) Successful in 22s

Details

Infra CI / notebooks (push) Successful in 46s

Details

CI / test (push) Successful in 14m45s

Details

Infra CI / zotero (push) Successful in 18s

Details

Infra CI / docs (push) Successful in 12s

Details

Infra CI / api (push) Successful in 13s

Details

Infra CI / llm (push) Successful in 12s

Details

Infra CI / mc (push) Successful in 15s

Details

ci/woodpecker/push/ci

Required

ci/woodpecker/pr/ci

Required

ci/woodpecker/push/infra-ci

Required

ci/woodpecker/pr/infra-ci

Required

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

api

REST API (FastAPI)

auth

Authentication and authorization

bib

Bibliography, Zotero, citations

ci

CI/CD pipelines and Woodpecker plugins

cli

CLI commands and UX

cloud

Cloud provider abstraction (AWS, GCP, Azure, self-hosted)

config

Configuration and wiring

data

Data loading, CCLF, BCDA, seeds

databricks

Databricks jobs and deployment

docs

Documentation and generated references

image:api

image:docs

image:mc

image:notebooks

image:zotero

infra

lakehouse

Nessie, Polaris, Trino, Iceberg

llm

llm module — local RAG over comments + Zotero corpus

notebooks

marimo notebook errors (auto-filed by nb-watcher/nb-integration)

packaging

Python packaging, optional deps, build system

pfs

pipeline

Pipeline execution and validation

pipeline:ci

pipeline:deploy

pipeline:harden

pipeline:infra-ci

pipeline:rebuild-all

pipeline:release

qpp

quality

Testing, coverage, error handling

step:build

step:deploy

step:hadolint

step:lint

step:publish

step:scan

step:test

step:validate

styles

Charts, figures, visual theming

tree

Project directory structure and file organization

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

kert renovate-bot

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#681