stack/docs/superpowers/specs/2026-09-09-zotero-annas-connector-design.md

Zotero full-text connector (Anna's Archive, libgen, Unpaywall) — design

Status: draft for operator approval, 2026-09-09. Related: dev/scripts/add_ama_coding_to_zotero.py (PR #697), corwins.media issue #707 (Anna's key), corwins spec 2026-09-09-direct-source-tier-design.md (egress proxy).

Problem

The Zotero library on fhirworx has 190,995 parent items and 167,377 of them carry no PDF or EPUB. Among those, 13,543 journal articles have a DOI and 576 books have an ISBN. Zotero's own "Find Available PDF" reaches only open-access copies through the browser. The operator holds an Anna's Archive membership and wants Zotero items filled from every freely reachable source, papers and books together, without hand work.

What was verified from the VPN namespace (2026-09-09)

Source	Scriptable?	Use
api.unpaywall.org/v2/{doi}?email=	yes	open-access PDF for a DOI (legal, no quota beyond politeness)
libgen.li/json.php?object=e&doi=… / &isbn=… (addkeys=*)	yes	edition → file md5s for a DOI (scimag corpus) or ISBN
libgen.li/ads.php?md5= → get.php	yes, per-file 503 throttling	file download
annas-archive.gl/dyn/api/fast_download.json?md5=&key=	yes (member)	file download by md5, 50/day on the current tier
Anna's HTML (/search, /md5/, /isbn/, /doi/, /scidb/)	no — DDoS-Guard JS challenge on every mirror	none
Sci-Hub mirrors	no — .se/.st unresolvable from the exit, .ru robot check, .box 403	none
Anna's clone annas-archive.cc	not official	never

Consequence: discovery is Unpaywall and the libgen JSON API; delivery is libgen or Anna's fast download by md5. Nothing depends on Anna's HTML or Sci-Hub.

Decisions

1. One resolver, two identifier paths, one attach path. A Zotero parent item resolves to a Candidate(source, url_or_md5, ext, size) list: DOI → Unpaywall best_oa_location.url_for_pdf first, then libgen JSON editions → md5s; ISBN → libgen JSON editions → md5s; a book with no ISBN → libgen title search (the index page parser the AMA script already exercises) with the year-aware containment match. Delivery tries libgen ads.php→get.php for each md5, then Anna's fast download for the same md5, in that order. Papers and books share every stage except the first lookup.
2. Lives in the stack repo as stack zot fetch. Package src/zot/fetch/ (resolve.py, deliver.py, attach.py, ledger.py), CLI verb in src/cli/zot.py. Direct zot.db writes as every other stack zot command does, under the existing --hold guard (Zotero desktop stopped for the write, restarted after). The AMA script's attach logic moves into attach.py in the same change so there is one implementation.
3. Selection is explicit. --collection NAME, --tag TAG, --type journalArticle|book|…, --item KEY, or --all; always excludes items that already have a PDF/EPUB attachment, deleted items, and the document type (146k regulatory PDFs that are not papers or books and would burn the quota on misses). --limit N caps attempts per run, --dry-run prints the plan.
4. Quota-aware. Anna's daily budget is read from the API's account_fast_download_info at run start and guarded by ANNAS_DAILY_RESERVE (default 5 kept back). libgen queries are paced 3 s apart; Unpaywall gets the operator contact email in the query and 1 req/s. When the Anna's budget is exhausted the run continues with libgen-only delivery and reports what it skipped.
5. A ledger prevents re-trying misses forever. data/zotero/fetch_ledger.jsonl (gitignored data), one line per attempt: item key, identifiers used, outcome (attached / no_candidate / all_failed / quota), source, md5, timestamp. An item with a terminal miss is retried only after 30 days or with --retry. The ledger is the report the operator reads.
6. Provenance on the item. Tags source:unpaywall / source:libgen / source:annas, plus md5:<md5> or doi: note in extra; the attachment title is the file name; nothing on the parent item is edited beyond tags and extra.
7. Egress through the gluetun HTTP proxy (VPN_HTTP_PROXY in stack .env, host-local 127.0.0.1:19035, published by the corwins compose change). Unpaywall may go direct; libgen and Anna's must not. A missing proxy is an error naming the compose step, not a fallback.
8. Secrets stay in .env. ANNAS_ARCHIVE_KEY (already present in corwins .env; copied to stack .env by the operator, never committed), UNPAYWALL_EMAIL (defaults to the existing contact used elsewhere in the stack).
9. No timer in this spec. The operator runs stack zot fetch; a daily systemd timer is a separate, operator-approved step once a few manual runs have been read.

Components

src/zot/fetch/
  resolve.py   identifiers_for(item) -> {doi, isbn, title, year}; candidates(ids, http) -> list[Candidate]
  deliver.py   fetch(candidate, budget) -> Path | None   (libgen ads→get, 503 → next; Anna's fast download)
  attach.py    attach_file(db, item_id, path, *, source, md5) ; has_full_text(db, item_id) -> bool
  ledger.py    Ledger(path).record(...) / .should_try(item_key) ; summary()
  select.py    select_items(db, collection=, tag=, type_=, item=, all_=) -> list[ItemRef]
src/cli/zot.py  @app.command("fetch")
tests/zot/test_fetch_*.py  fixtures: unpaywall_doi.json, libgen_json_doi.json, libgen_json_isbn.json, libgen_ads.html, annas_fast_download.json

Candidate = dataclass(source: str, md5: str | None, url: str | None, ext: str, size_bytes: int | None, title: str). Order: Unpaywall (source unpaywall, url) before libgen md5s; within md5s, epub before pdf for books, pdf only for papers.

fetch() streams to a temp file, rejects non-file content types and bodies under 10 kB, and returns the path. Budget is a small object with annas_left decremented on each fast-download use.

attach_file() is the AMA script's _attach generalised: storage dir data/zotero/data/storage/<KEY>/, sudo mkdir/cp/chown 100999:100999, db.add_attachment(parent, key=, content_type=, path="storage:<name>"), title = file name; has_full_text checks itemAttachments.contentType for application/pdf or application/epub+zip under the parent.

Error handling

• Per-item failures never abort the run; they are ledger lines.
• Unpaywall 404 or is_oa=false is not an error (next resolver).
• libgen ads.php 500 → one retry after 3 s; get.php 503 → next md5, then Anna's.
• Anna's API 401 → stop the run with a clear message (key wrong); 403 Not a member → same; any other non-200 → skip Anna's for the run.
• Zotero DB locked → the --hold guard already stops the desktop; if --no-hold is passed and the DB is locked, fail before any download.

Testing

Unit tests per module with httpx.MockTransport (adopt httpx for this package; the stack has no single HTTP client convention) and pinned fixtures captured through the proxy. test_resolve.py: DOI → Unpaywall first, libgen JSON second, order of candidates; ISBN → md5s; no-ISBN book → title search with containment. test_deliver.py: 503 → next candidate → Anna's; HTML body rejected; budget decrements. test_attach.py: has_full_text truth table on a scratch Zotero DB built from src/zot/schema.sql; attachment row shape. test_ledger.py: 30-day retry rule. CLI: typer.testing.CliRunner with --dry-run on a scratch DB. No network in tests.

Slices

• Z1 — resolver + delivery + attach, manual runs. Everything above except the timer. Done when: stack zot fetch --collection "AMA Coding Publications" --dry-run lists candidates for the AMA books; a real run on --type journalArticle --limit 20 attaches at least one Unpaywall PDF and one libgen/Anna's PDF and the ledger explains every miss.
• Z2 — fold the AMA script in. add_ama_coding_to_zotero.py keeps only the item-minting half and calls attach.py; the folder-to-title match moves with it.
• Z3 — timer (separate approval): daily stack zot fetch --all --limit 40.
• Z4 (only on a measured gap) — a headless-browser challenge pass in the VPN namespace for Anna's HTML/SciDB, if the libgen JSON index proves to miss many DOIs the ledger shows Anna's has.

Out of scope

Editing Zotero metadata beyond tags/extra; Zotero Web API or sync; Sci-Hub; Anna's HTML scraping; Z-Library (would need an account and its own login flow; can join later as another deliver source); the 146k document items.