zot fetch: Z1 review deferrals — lazy selection before the Z3 timer, plus minors #828

New Issue

kert · 2026-09-24T23:52:03Z

kert commented

2026-09-24 23:52:03 +00:00

Deferred from the fresh-context review of PR #826 (stack zot fetch, slice Z1). None affects the manual, targeted runs Z1 is for; the first one blocks the Z3 timer.

Blocks Z3 (timer)

Lazy selection. zot.fetch.select.select_items materialises every eligible item (get_item → fields/creators/tags/collections, plus get_item_type and has_full_text per id) before the first attempt, while the Zotero desktop is stopped under --hold. --all on ~167k file-less items will hold the desktop down for minutes before --limit can bite. Yield lazily (generator, or ids first and item_ref only after the ledger gate) and fold the deleted/full-text exclusions into the id query.

Minor

Title search (resolve.title_candidates) is unverified against a real libgen index.php page — the fixture is synthetic and the spec's "index page parser the AMA script already exercises" does not exist. Capture a real results page (a search through the proxy returned no rows on 2026-09-22) and pin the anchor regex; ([^<]*) also yields an empty title on nested markup.
Anna's: a 200 with download_url: null is retried on every candidate of every item instead of disabling Anna's after the first non-md5-specific error; and Anna's is tried once per libgen md5 per item, so an item with several md5s can spend several daily units on one bad CDN day. Consider one Anna's attempt per item.
--no-hold on a locked DB surfaces a raw sqlite3.OperationalError rather than a BadParameter naming the desktop.
--dry-run still stops/starts the Zotero desktop; either skip the hold for dry runs (reads are safe under WAL) or say so in the help. The env vars (VPN_HTTP_PROXY, ANNAS_ARCHIVE_KEY, ANNAS_DAILY_RESERVE, UNPAYWALL_EMAIL) are not in --help.
attach.append_extra strips leading whitespace from an existing extra; use rstrip("\n") + "\n" + line.
Scratch dir .state/zot/fetch/ is not swept at run start (per-item cleanup covers exceptions; a KeyboardInterrupt mid-stream still leaves a partial file).
select.first_isbn flattens hyphens but not spaces (978 1 64016 228 0 → no ISBN).
Budget: annas_left is taken verbatim from the API's downloads_left; if live observation shows the API reports the pre-call count, decrement once (else the reserve is breached by one per run).

Also noted (pre-existing, not this branch)

The repo-wide coverage gate: --cov-fail-under=99 prints FAIL at 98.94–98.95% on main and the CI job still reports success, so the gate is not enforced.

Deferred from the fresh-context review of PR #826 (`stack zot fetch`, slice Z1). None affects the manual, targeted runs Z1 is for; the first one blocks the Z3 timer. ## Blocks Z3 (timer) - [ ] **Lazy selection.** `zot.fetch.select.select_items` materialises every eligible item (`get_item` → fields/creators/tags/collections, plus `get_item_type` and `has_full_text` per id) before the first attempt, while the Zotero desktop is stopped under `--hold`. `--all` on ~167k file-less items will hold the desktop down for minutes before `--limit` can bite. Yield lazily (generator, or ids first and `item_ref` only after the ledger gate) and fold the deleted/full-text exclusions into the id query. ## Minor - [ ] Title search (`resolve.title_candidates`) is unverified against a real libgen `index.php` page — the fixture is synthetic and the spec's "index page parser the AMA script already exercises" does not exist. Capture a real results page (a search through the proxy returned no rows on 2026-09-22) and pin the anchor regex; `([^<]*)` also yields an empty title on nested markup. - [ ] Anna's: a 200 with `download_url: null` is retried on every candidate of every item instead of disabling Anna's after the first non-md5-specific error; and Anna's is tried once per libgen md5 per item, so an item with several md5s can spend several daily units on one bad CDN day. Consider one Anna's attempt per item. - [ ] `--no-hold` on a locked DB surfaces a raw `sqlite3.OperationalError` rather than a `BadParameter` naming the desktop. - [ ] `--dry-run` still stops/starts the Zotero desktop; either skip the hold for dry runs (reads are safe under WAL) or say so in the help. The env vars (`VPN_HTTP_PROXY`, `ANNAS_ARCHIVE_KEY`, `ANNAS_DAILY_RESERVE`, `UNPAYWALL_EMAIL`) are not in `--help`. - [ ] `attach.append_extra` strips leading whitespace from an existing `extra`; use `rstrip("\n") + "\n" + line`. - [ ] Scratch dir `.state/zot/fetch/` is not swept at run start (per-item cleanup covers exceptions; a `KeyboardInterrupt` mid-stream still leaves a partial file). - [ ] `select.first_isbn` flattens hyphens but not spaces (`978 1 64016 228 0` → no ISBN). - [ ] Budget: `annas_left` is taken verbatim from the API's `downloads_left`; if live observation shows the API reports the pre-call count, decrement once (else the reserve is breached by one per run). ## Also noted (pre-existing, not this branch) - The repo-wide coverage gate: `--cov-fail-under=99` prints FAIL at 98.94–98.95% on main and the CI job still reports success, so the gate is not enforced.

kert added this to the P51: Zotero full-text connector — Unpaywall, libgen, Anna's (Z1–Z4) milestone 2026-09-24 23:52:03 +00:00

kert added the bib label 2026-09-24 23:52:03 +00:00

kert referenced this issue

2026-09-24 23:52:12 +00:00

feat(zot): stack zot fetch — Zotero full-text connector, slice Z1 #826

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#828