prisma fetch: parallelize the source cascade — sequential 1,977-item queue is a 1-2 day run #659

New Issue

kert · 2026-08-19T14:48:56Z

kert commented

2026-08-19 14:48:56 +00:00

The palliative-rfi full-text fetch (stack prisma fetch palliative-rfi) walks its 1,977-item queue strictly sequentially: each item runs the Unpaywall → PMC → VPN-fallback cascade one at a time, and a miss burns 30–90 s in per-tier timeouts before the loop moves on.

Measured on 2026-08-19: 3 PDFs attached in the first ~12 minutes, process nearly idle (1 s CPU) — it is pure serial network waiting. At this pace the queue is a 1–2 day run, which does not fit the P43/P44 schedule (comment due 2026-09-14, draft target 2026-09-12).

fetch_one is already safe to parallelize: unique per-item scratch filenames ({zot_id}-{hash}.pdf), thread-safe httpx clients, no shared mutable state. The plan:

run() fans fetch_one out over a ThreadPoolExecutor (new workers param, CLI --workers/-w, default 8)
all DB work (queue build, attach_pdf, commit) stays on the main thread — SQLite connection is not shared across threads
the fallback tier is serialized with a lock: _altcha_bootstrap mutates the shared proxied client's cookie jar, and concurrent challenge solves through one droplet would race (and hammer the mirrors)
progress line every 25 completions, unchanged format

Related: #657 (queue-time PMID→PMCID resolution would raise the PMC tier hit rate — separate change).

The palliative-rfi full-text fetch (`stack prisma fetch palliative-rfi`) walks its 1,977-item queue **strictly sequentially**: each item runs the Unpaywall → PMC → VPN-fallback cascade one at a time, and a miss burns 30–90 s in per-tier timeouts before the loop moves on. Measured on 2026-08-19: 3 PDFs attached in the first ~12 minutes, process nearly idle (1 s CPU) — it is pure serial network waiting. At this pace the queue is a 1–2 day run, which does not fit the P43/P44 schedule (comment due 2026-09-14, draft target 2026-09-12). `fetch_one` is already safe to parallelize: unique per-item scratch filenames (`{zot_id}-{hash}.pdf`), thread-safe httpx clients, no shared mutable state. The plan: - `run()` fans `fetch_one` out over a `ThreadPoolExecutor` (new `workers` param, CLI `--workers/-w`, default 8) - all DB work (queue build, `attach_pdf`, `commit`) stays on the main thread — SQLite connection is not shared across threads - the fallback tier is serialized with a lock: `_altcha_bootstrap` mutates the shared proxied client's cookie jar, and concurrent challenge solves through one droplet would race (and hammer the mirrors) - progress line every 25 completions, unchanged format Related: #657 (queue-time PMID→PMCID resolution would raise the PMC tier hit rate — separate change).

kert added the bib label 2026-08-19 14:48:56 +00:00

kert commented

2026-08-19 15:44:26 +00:00

Implemented and rolled out (19e24cd, 912272d, f330d92):

run() fans fetch_one out over a ThreadPoolExecutor; all SQLite work stays on the main thread
fallback tier serialized behind a module lock (shared proxied-client cookie jar + one VPN exit)
stack prisma fetch --workers/-w (default 8)

Measured on the live palliative-rfi queue: 17 PDFs attached in the first ~10 min at 8 workers vs 3 in 12 min sequential — ~6-7x. The run over the remaining ~1,960 items is in flight.

Also repaired in passing (19e24cd, closes #658): TestFetchPmc and TestExt* were stale against 81cbe53's OA-service fetch_pmc and extraction-template ValueError guard.

Implemented and rolled out (19e24cd, 912272d, f330d92): - `run()` fans `fetch_one` out over a ThreadPoolExecutor; all SQLite work stays on the main thread - fallback tier serialized behind a module lock (shared proxied-client cookie jar + one VPN exit) - `stack prisma fetch --workers/-w` (default 8) Measured on the live palliative-rfi queue: **17 PDFs attached in the first ~10 min at 8 workers vs 3 in 12 min sequential — ~6-7x**. The run over the remaining ~1,960 items is in flight. Also repaired in passing (19e24cd, closes #658): TestFetchPmc and TestExt* were stale against 81cbe53's OA-service fetch_pmc and extraction-template ValueError guard.

kert closed this issue

2026-08-19 15:44:26 +00:00

kert referenced this issue from a commit

2026-08-26 19:56:39 +00:00

feat(prisma): parallel fetch — ThreadPoolExecutor fan-out, serialized fallback tier (refs #659)

kert referenced this issue from a commit

2026-08-26 19:56:39 +00:00

feat(cli): stack prisma fetch --workers (refs #659)

Sign in to join this conversation.