fix(bib): revert sleep to 3.7s — 1000/hr is the binding cap
Some checks failed
CI / lint (push) Failing after 6s
Deploy / notebooks (push) Has been skipped
CI / test (push) Failing after 8s
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Infra CI / notebooks (push) Failing after 29s
Infra CI / docs (push) Failing after 5s
Deploy / mc (push) Has been skipped
Infra CI / zotero (push) Successful in 37s
Infra CI / api (push) Successful in 31s
Infra CI / mc (push) Successful in 29s
Deploy / report (push) Successful in 37s

Empirical: at 1.3s pacing the API hits 429 constantly and our 60s
backoff makes throughput WORSE than the conservative 3.7s pacing
(observed ~5/min with 16 workers + heavy 429s vs ~16/min steady at
3.7s). The 50/min burst cap is looser than the 1000/hr sustained cap
— 3.7s ≈ 970/hr stays under both.

Concurrent CDN attachment downloads + 16 workers stay (those don't
count against the API limit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
kert
2026-04-23 11:53:37 -04:00
parent 2a6429dc11
commit 1753b16cd9
2 changed files with 5 additions and 4 deletions

View File

@@ -89,7 +89,7 @@ class Client:
self, self,
api_key: str | None = None, api_key: str | None = None,
*, *,
sleep: float = 1.3, # ~46 req/min — under the 50/min burst cap sleep: float = 3.7, # ~970 req/hr — under the 1000/hr cap (binding)
dl_workers: int = 16, # concurrent attachment download threads dl_workers: int = 16, # concurrent attachment download threads
client: httpx.Client | None = None, client: httpx.Client | None = None,
) -> None: ) -> None:

View File

@@ -345,10 +345,11 @@ def backfill_comments(
help="Cap items processed this run. 0 = no cap (multi-day crawl).", help="Cap items processed this run. 0 = no cap (multi-day crawl).",
), ),
sleep: float = typer.Option( sleep: float = typer.Option(
1.3, 3.7,
"--sleep", "--sleep",
help="Seconds between API calls. 1.3s ≈ 46/min — under the " help="Seconds between API calls. 3.7s ≈ 970/hr — just under the "
"50/min reg.gov burst cap. CDN downloads are concurrent and free.", "1000/hr reg.gov hourly cap (the binding limit; the 50/min burst "
"cap is looser). CDN downloads are concurrent and don't count.",
), ),
log_path: Path = typer.Option( log_path: Path = typer.Option(
Path("/tmp/bib-backfill-comments.log"), Path("/tmp/bib-backfill-comments.log"),