perf(bib): bump CDN download workers 4 → 16
Some checks failed
CI / lint (push) Successful in 32s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / report (push) Successful in 31s
CI / test (push) Has been cancelled
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Failing after 12s
Infra CI / zotero (push) Successful in 12s
Infra CI / docs (push) Successful in 1m11s
Infra CI / api (push) Successful in 18s
Infra CI / mc (push) Successful in 12s
Some checks failed
CI / lint (push) Successful in 32s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / report (push) Successful in 31s
CI / test (push) Has been cancelled
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Failing after 12s
Infra CI / zotero (push) Successful in 12s
Infra CI / docs (push) Successful in 1m11s
Infra CI / api (push) Successful in 18s
Infra CI / mc (push) Successful in 12s
At 4 workers the backfill runs at ~30% of the API rate ceiling — attachment downloads are the bottleneck. CDN has no published rate limit and httpx default pool is 100 connections, so 16 leaves plenty of headroom and should push throughput closer to the 46 items/min API cap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -90,7 +90,7 @@ class Client:
|
|||||||
api_key: str | None = None,
|
api_key: str | None = None,
|
||||||
*,
|
*,
|
||||||
sleep: float = 1.3, # ~46 req/min — under the 50/min burst cap
|
sleep: float = 1.3, # ~46 req/min — under the 50/min burst cap
|
||||||
dl_workers: int = 4, # concurrent attachment download threads
|
dl_workers: int = 16, # concurrent attachment download threads
|
||||||
client: httpx.Client | None = None,
|
client: httpx.Client | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
key = api_key or os.environ.get("REGULATIONS_GOV_API_KEY")
|
key = api_key or os.environ.get("REGULATIONS_GOV_API_KEY")
|
||||||
|
|||||||
Reference in New Issue
Block a user