perf(bib): bump CDN download workers 4 → 16
Some checks failed
CI / lint (push) Successful in 32s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / report (push) Successful in 31s
CI / test (push) Has been cancelled
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Failing after 12s
Infra CI / zotero (push) Successful in 12s
Infra CI / docs (push) Successful in 1m11s
Infra CI / api (push) Successful in 18s
Infra CI / mc (push) Successful in 12s

At 4 workers the backfill runs at ~30% of the API rate ceiling —
attachment downloads are the bottleneck. CDN has no published rate
limit and httpx default pool is 100 connections, so 16 leaves plenty
of headroom and should push throughput closer to the 46 items/min API
cap.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
kert
2026-04-23 11:30:07 -04:00
parent 22cbc74aae
commit 2a6429dc11

View File

@@ -90,7 +90,7 @@ class Client:
api_key: str | None = None, api_key: str | None = None,
*, *,
sleep: float = 1.3, # ~46 req/min — under the 50/min burst cap sleep: float = 1.3, # ~46 req/min — under the 50/min burst cap
dl_workers: int = 4, # concurrent attachment download threads dl_workers: int = 16, # concurrent attachment download threads
client: httpx.Client | None = None, client: httpx.Client | None = None,
) -> None: ) -> None:
key = api_key or os.environ.get("REGULATIONS_GOV_API_KEY") key = api_key or os.environ.get("REGULATIONS_GOV_API_KEY")