prisma fetch: resolve PMID→PMCID at queue time — PMC tier was blind (0 hits on 2,355 items) #657

New Issue

kert · 2026-08-19T07:14:57Z

kert commented

2026-08-19 07:14:57 +00:00

The 2026-08-19 palliative-rfi PDF fetch (2,355 queued) returned pmc=0 because pending_queue only reads a PMCID: line from the item extra field (or the URL), and the PubMed search pipeline writes only PMID:/DOI: — so the PMC tier never had an ID to try. First pass yielded just 354 PDFs (321 Unpaywall + 33 fallback), while a one-off backfill via NCBI's ID converter (/pmc/utils/idconv/v1.0/, redirects to pmc.ncbi.nlm.nih.gov/tools/idconv/api/v1/articles/, 200 ids/request, returns pmid as int) found PMCIDs for 831 of the 1,995 misses — more recoverable open-access full text than the entire first pass fetched.

Fix options: (a) pending_queue bulk-resolves missing PMCIDs via idconv before the fetch loop (one batched call, cache into extra); or (b) article_to_source in the search scripts captures PMCID at ingest (efetch XML carries ArticleIdList). Prefer both: ingest-time capture + queue-time backfill for legacy items.

The 2026-08-19 palliative-rfi PDF fetch (2,355 queued) returned **pmc=0** because `pending_queue` only reads a `PMCID:` line from the item extra field (or the URL), and the PubMed search pipeline writes only `PMID:`/`DOI:` — so the PMC tier never had an ID to try. First pass yielded just 354 PDFs (321 Unpaywall + 33 fallback), while a one-off backfill via NCBI's ID converter (`/pmc/utils/idconv/v1.0/`, redirects to `pmc.ncbi.nlm.nih.gov/tools/idconv/api/v1/articles/`, 200 ids/request, returns pmid as **int**) found **PMCIDs for 831 of the 1,995 misses** — more recoverable open-access full text than the entire first pass fetched. Fix options: (a) `pending_queue` bulk-resolves missing PMCIDs via idconv before the fetch loop (one batched call, cache into extra); or (b) `article_to_source` in the search scripts captures PMCID at ingest (efetch XML carries `ArticleIdList`). Prefer both: ingest-time capture + queue-time backfill for legacy items.

kert added the bib label 2026-08-19 07:14:57 +00:00

kert commented

2026-08-19 10:31:30 +00:00

Findings from the 2026-08-19 recovery attempt (state of the art for scripted PMC retrieval):

Backfill done: 831/1,995 missed items now carry PMCID: in extra (NCBI idconv; note the API returns pmid as int). Queue parses them (811 verified).
/pmc/articles/<id>/pdf/ is a bot wall: returns 200 text/html (~1.8KB challenge) to scripted clients — the tier could never work. fetch_pmc now uses the sanctioned OA service (oa.fcgi), which resolves ~52% of our PMC items (sampled 21/40; free-to-read author manuscripts return idIsNotOpenAccess).
But the OA service's own links are stale: hrefs point at ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/{oa_pdf,oa_package}/... — FTP denies (retired), and the HTTPS mirror now 404s everything under /pub/pmc/ except PMC-ids.csv.gz + a deprecated/ stub.
Remaining viable paths: (a) AWS Open Data pmc-oa-opendata bucket — alive, but XML/TXT only (no PDFs) and my key-layout guesses 404; needs the filelist metadata to map PMCID→key, plus prisma.export support for XML/TXT full text (arguably better than pdfminer output anyway); (b) Europe PMC REST fullTextXML — currently unreachable from the rack (EBI connect timeout), retry later.

Current coverage: 378/2,355 (321 Unpaywall + 33 fallback + 24 second-pass). The ~430 OA-subset texts are recoverable via (a)/(b); the ~1,100 non-PMC paywalled misses need institutional access and are out of scripted scope.

Findings from the 2026-08-19 recovery attempt (state of the art for scripted PMC retrieval): 1. **Backfill done**: 831/1,995 missed items now carry `PMCID:` in extra (NCBI idconv; note the API returns pmid as *int*). Queue parses them (811 verified). 2. **`/pmc/articles/<id>/pdf/` is a bot wall**: returns 200 `text/html` (~1.8KB challenge) to scripted clients — the tier could never work. `fetch_pmc` now uses the sanctioned OA service (`oa.fcgi`), which resolves ~52% of our PMC items (sampled 21/40; free-to-read author manuscripts return `idIsNotOpenAccess`). 3. **But the OA service's own links are stale**: hrefs point at `ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/{oa_pdf,oa_package}/...` — FTP denies (retired), and the HTTPS mirror now 404s everything under `/pub/pmc/` except `PMC-ids.csv.gz` + a `deprecated/` stub. 4. **Remaining viable paths**: (a) AWS Open Data `pmc-oa-opendata` bucket — alive, but XML/TXT only (no PDFs) and my key-layout guesses 404; needs the filelist metadata to map PMCID→key, plus `prisma.export` support for XML/TXT full text (arguably better than pdfminer output anyway); (b) Europe PMC REST `fullTextXML` — currently unreachable from the rack (EBI connect timeout), retry later. **Current coverage: 378/2,355 (321 Unpaywall + 33 fallback + 24 second-pass).** The ~430 OA-subset texts are recoverable via (a)/(b); the ~1,100 non-PMC paywalled misses need institutional access and are out of scripted scope.

kert referenced this issue from a commit

2026-08-19 10:31:31 +00:00

fix(prisma): fetch_pmc via the OA Web Service instead of the bot-challenged /pdf/ guess URL (refs #657)

kert referenced this issue from a commit

2026-08-19 10:32:54 +00:00

docs: annotate 81cbe53 — it bundles two change sets (refs #652, #657)

kert referenced this issue

2026-08-19 14:48:56 +00:00

prisma fetch: parallelize the source cascade — sequential 1,977-item queue is a 1-2 day run #659

kert commented

2026-08-19 18:12:11 +00:00

Investigation (2026-08-19) overturns this issue's premise and finds the real bug:

Queue-time PMID→PMCID resolution is worth zero. The queue already derives PMCIDs from item URLs (_from_url_pmcid): 795 of 1,945 pending items have PMCIDs. The NCBI ID converter, sampled on 100 of the 1,144 PMID-but-no-PMCID items, returned 0 new PMCIDs — records without a PMC URL genuinely have no PMC version.
The PMC tier scored 0/795 because the download leg is dead, not the resolution leg. The OA Web Service (81cbe53) resolves fine (~16% of the 795 are in the OA subset), but its ftp:// hrefs rewritten to https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/... now 404 — NCBI moved /pub/pmc/ content under /pub/pmc/deprecated/.
The canonical OA route is now the pmc-oa-opendata S3 bucket: https://pmc-oa-opendata.s3.amazonaws.com/PMC{id}.{ver}/PMC{id}.{ver}.pdf (list by prefix=PMC{id}. to find the version) — no rate limit, no bot challenge, verified live.
The ~84% of PMC-having items outside the OA subset are author manuscripts: efetch returns front matter only, the site PDF is behind Google reCAPTCHA, and the S3 bucket excludes them. Europe PMC's render endpoint is the standard lawful route but europepmc.org/ebi.ac.uk are unreachable from both the rack and the droplet today (likely EBI outage) — adding the tier with a circuit breaker so it activates when EBI returns.

Fix: S3 tier (primary, by PMCID) + OA-service tier with the deprecated/ path fix + Europe PMC tier behind a connection-failure circuit breaker. Then re-run the fetch.

Investigation (2026-08-19) overturns this issue's premise and finds the real bug: 1. **Queue-time PMID→PMCID resolution is worth zero.** The queue already derives PMCIDs from item URLs (`_from_url_pmcid`): 795 of 1,945 pending items have PMCIDs. The NCBI ID converter, sampled on 100 of the 1,144 PMID-but-no-PMCID items, returned **0 new PMCIDs** — records without a PMC URL genuinely have no PMC version. 2. **The PMC tier scored 0/795 because the download leg is dead, not the resolution leg.** The OA Web Service (81cbe53) resolves fine (~16% of the 795 are in the OA subset), but its `ftp://` hrefs rewritten to `https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/...` now 404 — NCBI moved `/pub/pmc/` content under `/pub/pmc/deprecated/`. 3. **The canonical OA route is now the `pmc-oa-opendata` S3 bucket**: `https://pmc-oa-opendata.s3.amazonaws.com/PMC{id}.{ver}/PMC{id}.{ver}.pdf` (list by `prefix=PMC{id}.` to find the version) — no rate limit, no bot challenge, verified live. 4. The ~84% of PMC-having items outside the OA subset are **author manuscripts**: efetch returns front matter only, the site PDF is behind Google reCAPTCHA, and the S3 bucket excludes them. Europe PMC's render endpoint is the standard lawful route but europepmc.org/ebi.ac.uk are unreachable from both the rack and the droplet today (likely EBI outage) — adding the tier with a circuit breaker so it activates when EBI returns. Fix: S3 tier (primary, by PMCID) + OA-service tier with the `deprecated/` path fix + Europe PMC tier behind a connection-failure circuit breaker. Then re-run the fetch.

kert commented

2026-08-19 19:10:35 +00:00

Post-fix re-run complete (2026-08-19, ~50 min, 8 workers): pmc: 399, fallback: 20, unpaywall: 0, missed: 1,526, errors: 0 — the PMC tier went 0 → 399 on the same queue. Stage-3 full-text coverage is now 829 of 2,355 (35%), up from 407 this morning. The S3 tier out-delivered the 16% OA-service sample estimate (~50% of PMC-having items) — the bucket covers articles the OA service labels not-OA. Remainder: ~400 PMC author manuscripts awaiting the Europe PMC tier (circuit breaker armed; EBI unreachable today — re-run stack prisma fetch palliative-rfi -w 8 when it recovers) plus ~1,100 items with no PMC version and no open copy.

Post-fix re-run complete (2026-08-19, ~50 min, 8 workers): **pmc: 399, fallback: 20, unpaywall: 0, missed: 1,526, errors: 0** — the PMC tier went 0 → 399 on the same queue. Stage-3 full-text coverage is now **829 of 2,355 (35%)**, up from 407 this morning. The S3 tier out-delivered the 16% OA-service sample estimate (~50% of PMC-having items) — the bucket covers articles the OA service labels not-OA. Remainder: ~400 PMC author manuscripts awaiting the Europe PMC tier (circuit breaker armed; EBI unreachable today — re-run `stack prisma fetch palliative-rfi -w 8` when it recovers) plus ~1,100 items with no PMC version and no open copy.

kert referenced this issue from a commit

2026-08-26 19:56:39 +00:00

fix(prisma): repair tests stale since 81cbe53 — OA-service fetch_pmc + extraction-template guard (closes #658, refs #657)

kert referenced this issue from a commit

2026-08-26 19:56:39 +00:00

fix(prisma): PMC tier via pmc-oa-opendata S3 + deprecated FTP path + Europe PMC breaker tier (closes #657)

kert closed this issue

2026-08-26 19:56:39 +00:00

Sign in to join this conversation.