From 1fea6748fa5704bd2879d51f1febef3df147292c Mon Sep 17 00:00:00 2001 From: kert Date: Wed, 9 Sep 2026 14:14:34 -0400 Subject: [PATCH] docs(spec): Zotero full-text connector via Unpaywall, libgen JSON API and Anna's fast download --- ...026-09-09-zotero-annas-connector-design.md | 76 +++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-09-zotero-annas-connector-design.md diff --git a/docs/superpowers/specs/2026-09-09-zotero-annas-connector-design.md b/docs/superpowers/specs/2026-09-09-zotero-annas-connector-design.md new file mode 100644 index 0000000..4390e65 --- /dev/null +++ b/docs/superpowers/specs/2026-09-09-zotero-annas-connector-design.md @@ -0,0 +1,76 @@ +# Zotero full-text connector (Anna's Archive, libgen, Unpaywall) — design + +**Status:** draft for operator approval, 2026-09-09. +**Related:** `dev/scripts/add_ama_coding_to_zotero.py` (PR #697), corwins.media issue #707 (Anna's key), corwins spec `2026-09-09-direct-source-tier-design.md` (egress proxy). + +## Problem + +The Zotero library on fhirworx has 190,995 parent items and 167,377 of them carry no PDF or EPUB. Among those, 13,543 journal articles have a DOI and 576 books have an ISBN. Zotero's own "Find Available PDF" reaches only open-access copies through the browser. The operator holds an Anna's Archive membership and wants Zotero items filled from every freely reachable source, papers and books together, without hand work. + +## What was verified from the VPN namespace (2026-09-09) + +| Source | Scriptable? | Use | +|---|---|---| +| `api.unpaywall.org/v2/{doi}?email=` | yes | open-access PDF for a DOI (legal, no quota beyond politeness) | +| `libgen.li/json.php?object=e&doi=…` / `&isbn=…` (`addkeys=*`) | yes | edition → file md5s for a DOI (scimag corpus) or ISBN | +| `libgen.li/ads.php?md5=` → `get.php` | yes, per-file 503 throttling | file download | +| `annas-archive.gl/dyn/api/fast_download.json?md5=&key=` | yes (member) | file download by md5, 50/day on the current tier | +| Anna's HTML (`/search`, `/md5/`, `/isbn/`, `/doi/`, `/scidb/`) | no — DDoS-Guard JS challenge on every mirror | none | +| Sci-Hub mirrors | no — `.se/.st` unresolvable from the exit, `.ru` robot check, `.box` 403 | none | +| Anna's clone `annas-archive.cc` | not official | never | + +Consequence: discovery is Unpaywall and the libgen JSON API; delivery is libgen or Anna's fast download by md5. Nothing depends on Anna's HTML or Sci-Hub. + +## Decisions + +1. **One resolver, two identifier paths, one attach path.** A Zotero parent item resolves to a `Candidate(source, url_or_md5, ext, size)` list: DOI → Unpaywall `best_oa_location.url_for_pdf` first, then libgen JSON editions → md5s; ISBN → libgen JSON editions → md5s; a book with no ISBN → libgen title search (the index page parser the AMA script already exercises) with the year-aware containment match. Delivery tries libgen `ads.php→get.php` for each md5, then Anna's fast download for the same md5, in that order. Papers and books share every stage except the first lookup. +2. **Lives in the stack repo as `stack zot fetch`.** Package `src/zot/fetch/` (`resolve.py`, `deliver.py`, `attach.py`, `ledger.py`), CLI verb in `src/cli/zot.py`. Direct `zot.db` writes as every other `stack zot` command does, under the existing `--hold` guard (Zotero desktop stopped for the write, restarted after). The AMA script's attach logic moves into `attach.py` in the same change so there is one implementation. +3. **Selection is explicit.** `--collection NAME`, `--tag TAG`, `--type journalArticle|book|…`, `--item KEY`, or `--all`; always excludes items that already have a PDF/EPUB attachment, deleted items, and the `document` type (146k regulatory PDFs that are not papers or books and would burn the quota on misses). `--limit N` caps attempts per run, `--dry-run` prints the plan. +4. **Quota-aware.** Anna's daily budget is read from the API's `account_fast_download_info` at run start and guarded by `ANNAS_DAILY_RESERVE` (default 5 kept back). libgen queries are paced 3 s apart; Unpaywall gets the operator contact email in the query and 1 req/s. When the Anna's budget is exhausted the run continues with libgen-only delivery and reports what it skipped. +5. **A ledger prevents re-trying misses forever.** `data/zotero/fetch_ledger.jsonl` (gitignored data), one line per attempt: item key, identifiers used, outcome (`attached` / `no_candidate` / `all_failed` / `quota`), source, md5, timestamp. An item with a terminal miss is retried only after 30 days or with `--retry`. The ledger is the report the operator reads. +6. **Provenance on the item.** Tags `source:unpaywall` / `source:libgen` / `source:annas`, plus `md5:` or `doi:` note in `extra`; the attachment title is the file name; nothing on the parent item is edited beyond tags and `extra`. +7. **Egress through the gluetun HTTP proxy** (`VPN_HTTP_PROXY` in stack `.env`, host-local `127.0.0.1:19035`, published by the corwins compose change). Unpaywall may go direct; libgen and Anna's must not. A missing proxy is an error naming the compose step, not a fallback. +8. **Secrets stay in `.env`.** `ANNAS_ARCHIVE_KEY` (already present in corwins `.env`; copied to stack `.env` by the operator, never committed), `UNPAYWALL_EMAIL` (defaults to the existing contact used elsewhere in the stack). +9. **No timer in this spec.** The operator runs `stack zot fetch`; a daily systemd timer is a separate, operator-approved step once a few manual runs have been read. + +## Components + +``` +src/zot/fetch/ + resolve.py identifiers_for(item) -> {doi, isbn, title, year}; candidates(ids, http) -> list[Candidate] + deliver.py fetch(candidate, budget) -> Path | None (libgen ads→get, 503 → next; Anna's fast download) + attach.py attach_file(db, item_id, path, *, source, md5) ; has_full_text(db, item_id) -> bool + ledger.py Ledger(path).record(...) / .should_try(item_key) ; summary() + select.py select_items(db, collection=, tag=, type_=, item=, all_=) -> list[ItemRef] +src/cli/zot.py @app.command("fetch") +tests/zot/test_fetch_*.py fixtures: unpaywall_doi.json, libgen_json_doi.json, libgen_json_isbn.json, libgen_ads.html, annas_fast_download.json +``` + +`Candidate = dataclass(source: str, md5: str | None, url: str | None, ext: str, size_bytes: int | None, title: str)`. Order: Unpaywall (source `unpaywall`, url) before libgen md5s; within md5s, epub before pdf for books, pdf only for papers. + +`fetch()` streams to a temp file, rejects non-file content types and bodies under 10 kB, and returns the path. Budget is a small object with `annas_left` decremented on each fast-download use. + +`attach_file()` is the AMA script's `_attach` generalised: storage dir `data/zotero/data/storage//`, `sudo mkdir/cp/chown 100999:100999`, `db.add_attachment(parent, key=, content_type=, path="storage:")`, title = file name; `has_full_text` checks `itemAttachments.contentType` for `application/pdf` or `application/epub+zip` under the parent. + +## Error handling + +- Per-item failures never abort the run; they are ledger lines. +- Unpaywall 404 or `is_oa=false` is not an error (next resolver). +- libgen `ads.php` 500 → one retry after 3 s; `get.php` 503 → next md5, then Anna's. +- Anna's API 401 → stop the run with a clear message (key wrong); 403 `Not a member` → same; any other non-200 → skip Anna's for the run. +- Zotero DB locked → the `--hold` guard already stops the desktop; if `--no-hold` is passed and the DB is locked, fail before any download. + +## Testing + +Unit tests per module with `httpx.MockTransport` (adopt httpx for this package; the stack has no single HTTP client convention) and pinned fixtures captured through the proxy. `test_resolve.py`: DOI → Unpaywall first, libgen JSON second, order of candidates; ISBN → md5s; no-ISBN book → title search with containment. `test_deliver.py`: 503 → next candidate → Anna's; HTML body rejected; budget decrements. `test_attach.py`: `has_full_text` truth table on a scratch Zotero DB built from `src/zot/schema.sql`; attachment row shape. `test_ledger.py`: 30-day retry rule. CLI: `typer.testing.CliRunner` with `--dry-run` on a scratch DB. No network in tests. + +## Slices + +- **Z1 — resolver + delivery + attach, manual runs.** Everything above except the timer. Done when: `stack zot fetch --collection "AMA Coding Publications" --dry-run` lists candidates for the AMA books; a real run on `--type journalArticle --limit 20` attaches at least one Unpaywall PDF and one libgen/Anna's PDF and the ledger explains every miss. +- **Z2 — fold the AMA script in.** `add_ama_coding_to_zotero.py` keeps only the item-minting half and calls `attach.py`; the folder-to-title match moves with it. +- **Z3 — timer** (separate approval): daily `stack zot fetch --all --limit 40`. +- **Z4 (only on a measured gap)** — a headless-browser challenge pass in the VPN namespace for Anna's HTML/SciDB, if the libgen JSON index proves to miss many DOIs the ledger shows Anna's has. + +## Out of scope + +Editing Zotero metadata beyond tags/extra; Zotero Web API or sync; Sci-Hub; Anna's HTML scraping; Z-Library (would need an account and its own login flow; can join later as another `deliver` source); the 146k `document` items.