infra: SOCKS proxy for SciHub PDF retrieval #273

New Issue

kert · 2026-03-25T22:04:11Z

kert commented

2026-03-25 22:04:11 +00:00

Blocker

SciHub mirrors are DNS-blocked and Cloudflare-protected from this host. All mirrors (sci-hub.se/st/ru/ren/ee/wf/shop) fail.

Impact

9,658 skin-subs articles need PDFs. OA coverage is ~4% (Unpaywall/PMC/S2). SciHub is the only source with meaningful coverage for paywalled wound care literature.

Solution Options

Tor — apt install tor, proxy at socks5://localhost:9050. Simplest.
SSH tunnel — ssh -D 1080 remote-host, proxy at socks5://localhost:1080
VPN — route traffic through a VPN where SciHub works
Docker container — run a tor/privoxy container in the stack

Usage (once proxy available)

uv run python dev/scripts/fetch_pdfs.py --proxy socks5://localhost:9050
# or
export SCIHUB_PROXY=socks5://localhost:9050
uv run python dev/scripts/fetch_pdfs.py

The fetch_pdfs.py script is complete and tested — just needs the proxy.

## Blocker SciHub mirrors are DNS-blocked and Cloudflare-protected from this host. All mirrors (sci-hub.se/st/ru/ren/ee/wf/shop) fail. ## Impact 9,658 skin-subs articles need PDFs. OA coverage is ~4% (Unpaywall/PMC/S2). SciHub is the only source with meaningful coverage for paywalled wound care literature. ## Solution Options 1. **Tor** — `apt install tor`, proxy at `socks5://localhost:9050`. Simplest. 2. **SSH tunnel** — `ssh -D 1080 remote-host`, proxy at `socks5://localhost:1080` 3. **VPN** — route traffic through a VPN where SciHub works 4. **Docker container** — run a tor/privoxy container in the stack ## Usage (once proxy available) ```bash uv run python dev/scripts/fetch_pdfs.py --proxy socks5://localhost:9050 # or export SCIHUB_PROXY=socks5://localhost:9050 uv run python dev/scripts/fetch_pdfs.py ``` The fetch_pdfs.py script is complete and tested — just needs the proxy.

kert added this to the P29: Skin Substitutes — predatory pricing, distribution, and utilization analysis milestone 2026-03-25 22:04:11 +00:00

kert referenced this issue from a commit

2026-03-25 22:21:26 +00:00

fix: CoreDNS — route sci-hub domains through unfiltered DNS (refs #273)

kert referenced this issue from a commit

2026-03-26 01:57:17 +00:00

feat: Cloudflare Worker PDF proxy + headless batch retrieval (fixes #273)

kert closed this issue

2026-03-26 01:57:17 +00:00

Sign in to join this conversation.