prisma fetch: add Semantic Scholar openAccessPdf and all-Unpaywall-locations tiers; resolve PMCIDs via idconv before the PMC tiers #723

New Issue

kert · 2026-09-12T00:05:04Z

kert commented

2026-09-12 00:05:04 +00:00

From the #650 full-text pass (2026-09-11): on 30 items the existing cascade (Unpaywall best_oa_location → PMC S3 → PMC OA service → Europe PMC → VPN fallback) found 0; a one-off sweep that (1) tried every entry in Unpaywall's oa_locations, not just best_oa_location, and (2) queried Semantic Scholar's graph API (paper/DOI:?fields=openAccessPdf) found 4 — all four via Semantic Scholar. Also: 7 of the 26 remaining items had PMCIDs that Zotero's extra field lacked; NCBI idconv (pmc.ncbi.nlm.nih.gov/tools/idconv/api/v1/articles/?ids=…&idtype=pmid) resolved them in one call, so the PMC tiers never even ran for those. Scope: add both tiers to prisma.fetch.fetch_one (S2 after Unpaywall, rate-limited ~1 req/s, no key), iterate all oa_locations, and resolve PMCIDs via idconv in pending_queue when extra lacks one. Note for the PMC S3 tier: the pmc-oa-opendata bucket holds only the OA subset (PMC{n}.{v}/ keys); its author_manuscript/ and oa_noncomm/ prefixes list zero keys, so author manuscripts must come from Europe PMC (EBI still unreachable from the rack on 09-11; ptpmcrender backend 520) or a browser session on pmc.ncbi.nlm.nih.gov (bot check answers 203 to headless). Sweep script: the oa_sweep.py pattern in this session's scratchpad, not committed.

From the #650 full-text pass (2026-09-11): on 30 items the existing cascade (Unpaywall best_oa_location → PMC S3 → PMC OA service → Europe PMC → VPN fallback) found 0; a one-off sweep that (1) tried every entry in Unpaywall's oa_locations, not just best_oa_location, and (2) queried Semantic Scholar's graph API (paper/DOI:<doi>?fields=openAccessPdf) found 4 — all four via Semantic Scholar. Also: 7 of the 26 remaining items had PMCIDs that Zotero's extra field lacked; NCBI idconv (pmc.ncbi.nlm.nih.gov/tools/idconv/api/v1/articles/?ids=…&idtype=pmid) resolved them in one call, so the PMC tiers never even ran for those. Scope: add both tiers to prisma.fetch.fetch_one (S2 after Unpaywall, rate-limited ~1 req/s, no key), iterate all oa_locations, and resolve PMCIDs via idconv in pending_queue when extra lacks one. Note for the PMC S3 tier: the pmc-oa-opendata bucket holds only the OA subset (PMC{n}.{v}/ keys); its author_manuscript/ and oa_noncomm/ prefixes list zero keys, so author manuscripts must come from Europe PMC (EBI still unreachable from the rack on 09-11; ptpmcrender backend 520) or a browser session on pmc.ncbi.nlm.nih.gov (bot check answers 203 to headless). Sweep script: the oa_sweep.py pattern in this session's scratchpad, not committed.

kert added the quality label 2026-09-12 00:05:04 +00:00

kert referenced this issue from a commit

2026-09-22 19:32:52 +00:00

feat(prisma): fetch tiers — every Unpaywall location, Semantic Scholar openAccessPdf, idconv PMCID resolution (refs #723)

kert referenced this issue from a commit

2026-09-22 19:32:52 +00:00

merge: #723 — prisma fetch tiers: every Unpaywall location, Semantic Scholar openAccessPdf, idconv PMCID resolution (refs #723)

kert commented

2026-09-22 22:03:41 +00:00

Done — merged to main as c82fc51 (commit bb092c9; ci.yml lint/test/notebooks-smoke green on the merge).

prisma.fetch.unpaywall_urls(): best_oa_location plus every oa_locations entry, direct PDF URLs before landing pages, de-duplicated; fetch_one tries each in turn (fetch_unpaywall keeps its old one-URL contract).
fetch_semantic_scholar(): graph API paper/DOI:<doi>?fields=openAccessPdf,externalIds, throttled to 1 req/s across workers, no key; runs after Unpaywall. Its externalIds.PubMedCentral fills a missing PMCID so the PMC S3 / OA-service / Europe PMC tiers run for items Unpaywall only knows by landing page (live: JAMA 2016.16840 → PMC5226373).
pending_queue(client=, email=): PMIDs lacking a PMCID resolved via NCBI idconv in 200-id batches and persisted to extra as a PMCID: line, so the next run asks nothing twice; without a client the queue stays network-free.
run() reports an s2 counter; stack prisma fetch help and the docs page updated. 120 fetch tests green (22 new).

**Done — merged to main as c82fc51** (commit bb092c9; ci.yml lint/test/notebooks-smoke green on the merge). - `prisma.fetch.unpaywall_urls()`: `best_oa_location` plus every `oa_locations` entry, direct PDF URLs before landing pages, de-duplicated; `fetch_one` tries each in turn (`fetch_unpaywall` keeps its old one-URL contract). - `fetch_semantic_scholar()`: graph API `paper/DOI:<doi>?fields=openAccessPdf,externalIds`, throttled to 1 req/s across workers, no key; runs after Unpaywall. Its `externalIds.PubMedCentral` fills a missing PMCID so the PMC S3 / OA-service / Europe PMC tiers run for items Unpaywall only knows by landing page (live: JAMA 2016.16840 → PMC5226373). - `pending_queue(client=, email=)`: PMIDs lacking a PMCID resolved via NCBI idconv in 200-id batches and persisted to `extra` as a `PMCID:` line, so the next run asks nothing twice; without a client the queue stays network-free. - `run()` reports an `s2` counter; `stack prisma fetch` help and the docs page updated. 120 fetch tests green (22 new).

kert closed this issue

2026-09-22 22:03:41 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#723