77 lines
8.4 KiB
Markdown
77 lines
8.4 KiB
Markdown
# Zotero full-text connector (Anna's Archive, libgen, Unpaywall) — design
|
||
|
||
**Status:** Z1 approved and built 2026-09-22/23 (plan `docs/superpowers/plans/2026-09-22-zotero-fetch-z1.md`); Z2–Z4 pending.
|
||
**Related:** `dev/scripts/add_ama_coding_to_zotero.py` (PR #697), corwins.media issue #707 (Anna's key), corwins spec `2026-09-09-direct-source-tier-design.md` (egress proxy).
|
||
|
||
## Problem
|
||
|
||
The Zotero library on fhirworx has 190,995 parent items and 167,377 of them carry no PDF or EPUB. Among those, 13,543 journal articles have a DOI and 576 books have an ISBN. Zotero's own "Find Available PDF" reaches only open-access copies through the browser. The operator holds an Anna's Archive membership and wants Zotero items filled from every freely reachable source, papers and books together, without hand work.
|
||
|
||
## What was verified from the VPN namespace (2026-09-09)
|
||
|
||
| Source | Scriptable? | Use |
|
||
|---|---|---|
|
||
| `api.unpaywall.org/v2/{doi}?email=` | yes | open-access PDF for a DOI (legal, no quota beyond politeness) |
|
||
| `libgen.li/json.php?object=e&doi=…` / `&isbn=…` (`addkeys=*`) | yes | edition → file md5s for a DOI (scimag corpus) or ISBN |
|
||
| `libgen.li/ads.php?md5=` → `get.php` | yes, per-file 503 throttling | file download |
|
||
| `annas-archive.gl/dyn/api/fast_download.json?md5=&key=` | yes (member) | file download by md5, 50/day on the current tier |
|
||
| Anna's HTML (`/search`, `/md5/`, `/isbn/`, `/doi/`, `/scidb/`) | no — DDoS-Guard JS challenge on every mirror | none |
|
||
| Sci-Hub mirrors | no — `.se/.st` unresolvable from the exit, `.ru` robot check, `.box` 403 | none |
|
||
| Anna's clone `annas-archive.cc` | not official | never |
|
||
|
||
Consequence: discovery is Unpaywall and the libgen JSON API; delivery is libgen or Anna's fast download by md5. Nothing depends on Anna's HTML or Sci-Hub.
|
||
|
||
## Decisions
|
||
|
||
1. **One resolver, two identifier paths, one attach path.** A Zotero parent item resolves to a `Candidate(source, url_or_md5, ext, size)` list: DOI → Unpaywall `best_oa_location.url_for_pdf` first, then libgen JSON editions → md5s; ISBN → libgen JSON editions → md5s; a book with no ISBN → libgen title search (the index page parser the AMA script already exercises) with the year-aware containment match. Delivery tries libgen `ads.php→get.php` for each md5, then Anna's fast download for the same md5, in that order. Papers and books share every stage except the first lookup.
|
||
2. **Lives in the stack repo as `stack zot fetch`.** Package `src/zot/fetch/` (`resolve.py`, `deliver.py`, `attach.py`, `ledger.py`), CLI verb in `src/cli/zot.py`. Direct `zot.db` writes as every other `stack zot` command does, under the existing `--hold` guard (Zotero desktop stopped for the write, restarted after). The AMA script's attach logic moves into `attach.py` in the same change so there is one implementation.
|
||
3. **Selection is explicit.** `--collection NAME`, `--tag TAG`, `--type journalArticle|book|…`, `--item KEY`, or `--all`; always excludes items that already have a PDF/EPUB attachment, deleted items, and the `document` type (146k regulatory PDFs that are not papers or books and would burn the quota on misses). `--limit N` caps attempts per run, `--dry-run` prints the plan.
|
||
4. **Quota-aware.** Anna's daily budget is read from the API's `account_fast_download_info` at run start and guarded by `ANNAS_DAILY_RESERVE` (default 5 kept back). libgen queries are paced 3 s apart; Unpaywall gets the operator contact email in the query and 1 req/s. When the Anna's budget is exhausted the run continues with libgen-only delivery and reports what it skipped.
|
||
5. **A ledger prevents re-trying misses forever.** `data/zotero/fetch_ledger.jsonl` (gitignored data), one line per attempt: item key, identifiers used, outcome (`attached` / `no_candidate` / `all_failed` / `quota`), source, md5, timestamp. An item with a terminal miss is retried only after 30 days or with `--retry`. The ledger is the report the operator reads.
|
||
6. **Provenance on the item.** Tags `source:unpaywall` / `source:libgen` / `source:annas`, plus `md5:<md5>` or `doi:` note in `extra`; the attachment title is the file name; nothing on the parent item is edited beyond tags and `extra`.
|
||
7. **Egress through the gluetun HTTP proxy** (`VPN_HTTP_PROXY` in stack `.env`, host-local `127.0.0.1:19035`, published by the corwins compose change). Unpaywall may go direct; libgen and Anna's must not. A missing proxy is an error naming the compose step, not a fallback.
|
||
8. **Secrets stay in `.env`.** `ANNAS_ARCHIVE_KEY` (already present in corwins `.env`; copied to stack `.env` by the operator, never committed), `UNPAYWALL_EMAIL` (defaults to the existing contact used elsewhere in the stack).
|
||
9. **No timer in this spec.** The operator runs `stack zot fetch`; a daily systemd timer is a separate, operator-approved step once a few manual runs have been read.
|
||
|
||
## Components
|
||
|
||
```
|
||
src/zot/fetch/
|
||
resolve.py identifiers_for(item) -> {doi, isbn, title, year}; candidates(ids, http) -> list[Candidate]
|
||
deliver.py fetch(candidate, budget) -> Path | None (libgen ads→get, 503 → next; Anna's fast download)
|
||
attach.py attach_file(db, item_id, path, *, source, md5) ; has_full_text(db, item_id) -> bool
|
||
ledger.py Ledger(path).record(...) / .should_try(item_key) ; summary()
|
||
select.py select_items(db, collection=, tag=, type_=, item=, all_=) -> list[ItemRef]
|
||
src/cli/zot.py @app.command("fetch")
|
||
tests/zot/test_fetch_*.py fixtures: unpaywall_doi.json, libgen_json_doi.json, libgen_json_isbn.json, libgen_ads.html, annas_fast_download.json
|
||
```
|
||
|
||
`Candidate = dataclass(source: str, md5: str | None, url: str | None, ext: str, size_bytes: int | None, title: str)`. Order: Unpaywall (source `unpaywall`, url) before libgen md5s; within md5s, epub before pdf for books, pdf only for papers.
|
||
|
||
`fetch()` streams to a temp file, rejects non-file content types and bodies under 10 kB, and returns the path. Budget is a small object with `annas_left` decremented on each fast-download use.
|
||
|
||
`attach_file()` is the AMA script's `_attach` generalised: storage dir `data/zotero/data/storage/<KEY>/`, `sudo mkdir/cp/chown 100999:100999`, `db.add_attachment(parent, key=, content_type=, path="storage:<name>")`, title = file name; `has_full_text` checks `itemAttachments.contentType` for `application/pdf` or `application/epub+zip` under the parent.
|
||
|
||
## Error handling
|
||
|
||
- Per-item failures never abort the run; they are ledger lines.
|
||
- Unpaywall 404 or `is_oa=false` is not an error (next resolver).
|
||
- libgen `ads.php` 500 → one retry after 3 s; `get.php` 503 → next md5, then Anna's.
|
||
- Anna's API 401 → stop the run with a clear message (key wrong); 403 `Not a member` → same; any other non-200 → skip Anna's for the run.
|
||
- Zotero DB locked → the `--hold` guard already stops the desktop; if `--no-hold` is passed and the DB is locked, fail before any download.
|
||
|
||
## Testing
|
||
|
||
Unit tests per module with `httpx.MockTransport` (adopt httpx for this package; the stack has no single HTTP client convention) and pinned fixtures captured through the proxy. `test_resolve.py`: DOI → Unpaywall first, libgen JSON second, order of candidates; ISBN → md5s; no-ISBN book → title search with containment. `test_deliver.py`: 503 → next candidate → Anna's; HTML body rejected; budget decrements. `test_attach.py`: `has_full_text` truth table on a scratch Zotero DB built from `src/zot/schema.sql`; attachment row shape. `test_ledger.py`: 30-day retry rule. CLI: `typer.testing.CliRunner` with `--dry-run` on a scratch DB. No network in tests.
|
||
|
||
## Slices
|
||
|
||
- **Z1 — resolver + delivery + attach, manual runs.** Everything above except the timer. Done when: `stack zot fetch --collection "AMA Coding Publications" --dry-run` lists candidates for the AMA books; a real run on `--type journalArticle --limit 20` attaches at least one Unpaywall PDF and one libgen/Anna's PDF and the ledger explains every miss.
|
||
- **Z2 — fold the AMA script in.** `add_ama_coding_to_zotero.py` keeps only the item-minting half and calls `attach.py`; the folder-to-title match moves with it.
|
||
- **Z3 — timer** (separate approval): daily `stack zot fetch --all --limit 40`.
|
||
- **Z4 (only on a measured gap)** — a headless-browser challenge pass in the VPN namespace for Anna's HTML/SciDB, if the libgen JSON index proves to miss many DOIs the ledger shows Anna's has.
|
||
|
||
## Out of scope
|
||
|
||
Editing Zotero metadata beyond tags/extra; Zotero Web API or sync; Sci-Hub; Anna's HTML scraping; Z-Library (would need an account and its own login flow; can join later as another `deliver` source); the 146k `document` items.
|