fix(bib): revert sleep to 3.7s — 1000/hr is the binding cap
Some checks failed
CI / lint (push) Failing after 6s
Deploy / notebooks (push) Has been skipped
CI / test (push) Failing after 8s
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Infra CI / notebooks (push) Failing after 29s
Infra CI / docs (push) Failing after 5s
Deploy / mc (push) Has been skipped
Infra CI / zotero (push) Successful in 37s
Infra CI / api (push) Successful in 31s
Infra CI / mc (push) Successful in 29s
Deploy / report (push) Successful in 37s
Some checks failed
CI / lint (push) Failing after 6s
Deploy / notebooks (push) Has been skipped
CI / test (push) Failing after 8s
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Infra CI / notebooks (push) Failing after 29s
Infra CI / docs (push) Failing after 5s
Deploy / mc (push) Has been skipped
Infra CI / zotero (push) Successful in 37s
Infra CI / api (push) Successful in 31s
Infra CI / mc (push) Successful in 29s
Deploy / report (push) Successful in 37s
Empirical: at 1.3s pacing the API hits 429 constantly and our 60s backoff makes throughput WORSE than the conservative 3.7s pacing (observed ~5/min with 16 workers + heavy 429s vs ~16/min steady at 3.7s). The 50/min burst cap is looser than the 1000/hr sustained cap — 3.7s ≈ 970/hr stays under both. Concurrent CDN attachment downloads + 16 workers stay (those don't count against the API limit). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -89,7 +89,7 @@ class Client:
|
|||||||
self,
|
self,
|
||||||
api_key: str | None = None,
|
api_key: str | None = None,
|
||||||
*,
|
*,
|
||||||
sleep: float = 1.3, # ~46 req/min — under the 50/min burst cap
|
sleep: float = 3.7, # ~970 req/hr — under the 1000/hr cap (binding)
|
||||||
dl_workers: int = 16, # concurrent attachment download threads
|
dl_workers: int = 16, # concurrent attachment download threads
|
||||||
client: httpx.Client | None = None,
|
client: httpx.Client | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|||||||
@@ -345,10 +345,11 @@ def backfill_comments(
|
|||||||
help="Cap items processed this run. 0 = no cap (multi-day crawl).",
|
help="Cap items processed this run. 0 = no cap (multi-day crawl).",
|
||||||
),
|
),
|
||||||
sleep: float = typer.Option(
|
sleep: float = typer.Option(
|
||||||
1.3,
|
3.7,
|
||||||
"--sleep",
|
"--sleep",
|
||||||
help="Seconds between API calls. 1.3s ≈ 46/min — under the "
|
help="Seconds between API calls. 3.7s ≈ 970/hr — just under the "
|
||||||
"50/min reg.gov burst cap. CDN downloads are concurrent and free.",
|
"1000/hr reg.gov hourly cap (the binding limit; the 50/min burst "
|
||||||
|
"cap is looser). CDN downloads are concurrent and don't count.",
|
||||||
),
|
),
|
||||||
log_path: Path = typer.Option(
|
log_path: Path = typer.Option(
|
||||||
Path("/tmp/bib-backfill-comments.log"),
|
Path("/tmp/bib-backfill-comments.log"),
|
||||||
|
|||||||
Reference in New Issue
Block a user