feat(zot): stack zot fetch — Zotero full-text connector, slice Z1 #826

Open

kert wants to merge 9 commits from feat/zot-fetch-z1 into main

pull from: feat/zot-fetch-z1

merge into: homelab:main

homelab:main

homelab:feat/zotero-fetch-connector

homelab:feat/zotero-ama-coding-collection

Conversation 1 Commits 9 Files Changed 29 +5434 -1

kert commented 2026-09-24 23:48:56 +00:00

Owner

Copy Link

Slice Z1 of the Zotero full-text connector (spec docs/superpowers/specs/2026-09-09-zotero-annas-connector-design.md, plan docs/superpowers/plans/2026-09-22-zotero-fetch-z1.md, operator-approved 2026-09-22).

What it does

stack zot fetch attaches a PDF/EPUB to Zotero items that lack one:

• select — --collection NAME | --tag TAG | --type TYPE | --item KEY | --all (+ --type as a narrowing filter); always skips items with a PDF/EPUB child, deleted items and the document type.
• resolve — DOI → Unpaywall (direct) then the libgen JSON index (through VPN_HTTP_PROXY); ISBN → libgen; ISBN-less book → libgen title search. Candidates ordered Unpaywall first; books epub before pdf, papers pdf only.
• deliver — libgen ads.php→get.php (browser UA, Referer, one retry after the DB-3306 throttle page), then Anna's fast-download API by md5 under ANNAS_DAILY_RESERVE (default 5). Files validated by status, content-type, 10 kB floor and magic bytes.
• attach — rows first, then sudo copy into data/zotero/data/storage/<KEY>/ as 100999:100999; provenance is exactly a source:<x> tag and an md5: line in extra.
• ledger — data/zotero/fetch_ledger.jsonl, one line per attempt with a note explaining every miss; no_candidate/all_failed wait 30 days (or --retry), quota and error (transport/5xx/throttle) retry next run.
• --dry-run lists candidates and writes nothing; --hold (default) stops/starts the zotero container around the run.

Review

Fresh-context review found no Critical issues; the three Important ones (an Anna's quota 403 aborting the run, no per-item exception isolation, transient failures becoming 30-day misses with no note) are fixed in d821ccc with tests that failed first. Deferred, to file as follow-ups: lazy selection before the Z3 timer (--all materialises every eligible item while Zotero is stopped), and eight minors listed in the plan workspace ledger.

Test plan

• uv run pytest tests/zot/fetch tests/cli/test_zot_fetch.py — 122 passed, src/zot/fetch at 100% line coverage, no network in tests (httpx.MockTransport, fixtures captured through the proxy 2026-09-22)
• full suite: 15,649 passed, 56 skipped (coverage 98.95% vs the 99% gate — pre-existing: main HEAD reports 98.94% and CI still passes)
• live acceptance (plan Task 8, from /home/kert/stack after merge): stack zot fetch --collection "AMA Coding Publications" --dry-run lists candidates; --type journalArticle --limit 20 attaches ≥1 Unpaywall and ≥1 libgen/Anna's PDF with every miss explained in the ledger

🤖 Generated with Claude Code

Slice Z1 of the Zotero full-text connector (spec `docs/superpowers/specs/2026-09-09-zotero-annas-connector-design.md`, plan `docs/superpowers/plans/2026-09-22-zotero-fetch-z1.md`, operator-approved 2026-09-22). ## What it does `stack zot fetch` attaches a PDF/EPUB to Zotero items that lack one: - **select** — `--collection NAME | --tag TAG | --type TYPE | --item KEY | --all` (+ `--type` as a narrowing filter); always skips items with a PDF/EPUB child, deleted items and the `document` type. - **resolve** — DOI → Unpaywall (direct) then the libgen JSON index (through `VPN_HTTP_PROXY`); ISBN → libgen; ISBN-less book → libgen title search. Candidates ordered Unpaywall first; books epub before pdf, papers pdf only. - **deliver** — libgen `ads.php→get.php` (browser UA, Referer, one retry after the DB-3306 throttle page), then Anna's fast-download API by md5 under `ANNAS_DAILY_RESERVE` (default 5). Files validated by status, content-type, 10 kB floor and magic bytes. - **attach** — rows first, then `sudo` copy into `data/zotero/data/storage/<KEY>/` as 100999:100999; provenance is exactly a `source:<x>` tag and an `md5:` line in `extra`. - **ledger** — `data/zotero/fetch_ledger.jsonl`, one line per attempt with a `note` explaining every miss; `no_candidate`/`all_failed` wait 30 days (or `--retry`), `quota` and `error` (transport/5xx/throttle) retry next run. - `--dry-run` lists candidates and writes nothing; `--hold` (default) stops/starts the zotero container around the run. ## Review Fresh-context review found no Critical issues; the three Important ones (an Anna's quota 403 aborting the run, no per-item exception isolation, transient failures becoming 30-day misses with no note) are fixed in d821ccc with tests that failed first. Deferred, to file as follow-ups: lazy selection before the Z3 timer (`--all` materialises every eligible item while Zotero is stopped), and eight minors listed in the plan workspace ledger. ## Test plan - [x] `uv run pytest tests/zot/fetch tests/cli/test_zot_fetch.py` — 122 passed, `src/zot/fetch` at 100% line coverage, no network in tests (httpx.MockTransport, fixtures captured through the proxy 2026-09-22) - [x] full suite: 15,649 passed, 56 skipped (coverage 98.95% vs the 99% gate — pre-existing: main HEAD reports 98.94% and CI still passes) - [ ] live acceptance (plan Task 8, from `/home/kert/stack` after merge): `stack zot fetch --collection "AMA Coding Publications" --dry-run` lists candidates; `--type journalArticle --limit 20` attaches ≥1 Unpaywall and ≥1 libgen/Anna's PDF with every miss explained in the ledger 🤖 Generated with [Claude Code](https://claude.com/claude-code)

kert added 9 commits 2026-09-24 23:48:56 +00:00

docs(plan): Z1 of the Zotero full-text connector — stack zot fetch (spec 2026-09-09) ... 3615f6e781

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(zot): fetch ledger — JSONL attempts with the 30-day retry rule (refs spec 2026-09-09 Z1) ... f149c6e984

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(zot): fetch.attach — storage copy, attachment row, source tag and md5 provenance (refs Z1) ... 6846554c97

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(zot): fetch.select — item population by collection/tag/type/item/all with the standing exclusions (refs Z1) ... 68c9531d98

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(zot): fetch.resolve — Unpaywall + libgen JSON (DOI/ISBN/title) → ordered candidates (refs Z1) ... 8ca3af3a8a

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(zot): fetch.deliver — libgen ads→get with throttle retry, Anna's fast download under a daily reserve (refs Z1) ... f69a3c3430

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(zot): fetch.run — the per-item loop with dry-run, limit, retry and quota outcomes (refs Z1) ... 773bbc5d18

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(zot): stack zot fetch — full-text connector CLI under the --hold guard, with docs (refs Z1) ... 0b29cd7acc

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

fix(zot): fetch review pass — quota 403 degrades instead of aborting, per-item isolation with rollback and scratch cleanup, ledger notes with an 'error' outcome that retries next run (refs Z1) ... d821ccc964

- Anna's 403 whose body is not a key/membership error disables Anna's for
  the run (annas_left=0) instead of raising AnnasAuthError.
- run() isolates each item: any exception rolls the transaction back,
  unlinks scratch files, ledgers outcome=error with the exception text,
  and continues; AnnasAuthError still aborts.
- attach_file writes rows before copying so a failed copy rolls back
  cleanly instead of orphaning a storage dir.
- Notes (miss vs error) from resolve/deliver land in every ledger line;
  an item whose lookups/deliveries errored (transport, 5xx, throttle
  page) is 'error' and retried next run rather than a 30-day miss.
- resolve tolerates non-dict file entries and non-numeric filesize.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

kert referenced this pull request 2026-09-24 23:52:03 +00:00

zot fetch: Z1 review deferrals — lazy selection before the Z3 timer, plus minors #828

kert commented 2026-09-24 23:52:12 +00:00

Author

Owner

Copy Link

Live acceptance, criterion 1 (run 2026-09-24 from the worktree with the live Zotero DB/storage symlinked in; dry-run is read-only):

stack zot fetch --collection "AMA Coding Publications" --dry-run --limit 10
569 eligible item(s)
NWPWYVS7  CPT Professional 2025  ids={'isbn': '9781640163041'}
    (no candidates: miss: libgen: no edition for isbn 9781640163041)
… (10 coding cards / current-year books, none in libgen)

stack zot fetch --item KTKN8YEH --dry-run
KTKN8YEH  HCPCS 2023 Level II Professional Edition  ids={'isbn': '9781640162280'}
    libgen    pdf  120ddd118103ea2ea927aaa489310dc8

Selection, ISBN parsing, the proxied libgen JSON lookups and the miss notes all behave live. Criterion 2 (--type journalArticle --limit 20, which writes) runs from /home/kert/stack once this lands on main. Follow-ups: #828 (milestone P51).

Live acceptance, criterion 1 (run 2026-09-24 from the worktree with the live Zotero DB/storage symlinked in; dry-run is read-only): ``` stack zot fetch --collection "AMA Coding Publications" --dry-run --limit 10 569 eligible item(s) NWPWYVS7 CPT Professional 2025 ids={'isbn': '9781640163041'} (no candidates: miss: libgen: no edition for isbn 9781640163041) … (10 coding cards / current-year books, none in libgen) stack zot fetch --item KTKN8YEH --dry-run KTKN8YEH HCPCS 2023 Level II Professional Edition ids={'isbn': '9781640162280'} libgen pdf 120ddd118103ea2ea927aaa489310dc8 ``` Selection, ISBN parsing, the proxied libgen JSON lookups and the miss notes all behave live. Criterion 2 (`--type journalArticle --limit 20`, which writes) runs from `/home/kert/stack` once this lands on main. Follow-ups: #828 (milestone P51).

Some checks are pending

Hide all checks

CI / lint (pull_request) Successful in 45s

Details

CI / notebooks-smoke (pull_request) Successful in 1m58s

Details

CI / test (pull_request) Successful in 2m55s

Details

CI / lint (push) Successful in 45s

Details

CI / notebooks-smoke (push) Successful in 1m43s

Details

CI / test (push) Successful in 2m38s

Details

Infra CI / zotero (pull_request) Successful in 15s

Details

Infra CI / notebooks (pull_request) Successful in 1m5s

Details

Infra CI / api (pull_request) Successful in 1m12s

Details

Infra CI / docs (pull_request) Successful in 2m1s

Details

Infra CI / mc (pull_request) Failing after 44s

Details

Infra CI / llm (pull_request) Successful in 1m5s

Details

Infra CI / zotero (push) Successful in 22s

Details

Infra CI / notebooks (push) Successful in 1m14s

Details

Infra CI / llm (push) Successful in 19s

Details

Infra CI / docs (push) Successful in 15s

Details

Infra CI / api (push) Successful in 24s

Details

Infra CI / mc (push) Failing after 55s

Details

ci/woodpecker/push/ci

Required

ci/woodpecker/pr/ci

Required

ci/woodpecker/push/infra-ci

Required

ci/woodpecker/pr/infra-ci

Required

This pull request is blocked because it's outdated.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feat/zot-fetch-z1:feat/zot-fetch-z1

git checkout feat/zot-fetch-z1

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

api

REST API (FastAPI)

auth

Authentication and authorization

bib

Bibliography, Zotero, citations

ci

CI/CD pipelines and Woodpecker plugins

cli

CLI commands and UX

cloud

Cloud provider abstraction (AWS, GCP, Azure, self-hosted)

config

Configuration and wiring

data

Data loading, CCLF, BCDA, seeds

databricks

Databricks jobs and deployment

docs

Documentation and generated references

image:api

image:docs

image:mc

image:notebooks

image:zotero

infra

lakehouse

Nessie, Polaris, Trino, Iceberg

llm

llm module — local RAG over comments + Zotero corpus

notebooks

marimo notebook errors (auto-filed by nb-watcher/nb-integration)

packaging

Python packaging, optional deps, build system

pfs

pipeline

Pipeline execution and validation

pipeline:ci

pipeline:deploy

pipeline:harden

pipeline:infra-ci

pipeline:rebuild-all

pipeline:release

qpp

quality

Testing, coverage, error handling

step:build

step:deploy

step:hadolint

step:lint

step:publish

step:scan

step:test

step:validate

styles

Charts, figures, visual theming

tree

Project directory structure and file organization

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

kert renovate-bot

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#826