ingress: crawler hammering Gitea /projects with permuted label filters (~1 req/s sustained) #656

New Issue

kert · 2026-08-18T20:29:02Z

kert commented

2026-08-18 20:29:02 +00:00

Noticed while diagnosing the 2026-08-18 DNS outage: 216.73.216.253 requests /homelab/stack/projects/1?labels=<permutation> roughly once per second, endlessly permuting label-ID combinations — a combinatorial crawler trap. Each response is ~43KB/10ms so load is minor, but it is sustained junk traffic filling traefik access logs (it was also the accidental reason git survived the DNS outage — its warm connection pool never re-resolved).

Options: robots.txt disallow on /*/projects with filters; a traefik rate-limit or bot-block middleware on the git router (216.73.216.0/24 resolves to a known AI-crawler range); or Gitea-side setting to noindex filtered views.

Noticed while diagnosing the 2026-08-18 DNS outage: 216.73.216.253 requests `/homelab/stack/projects/1?labels=<permutation>` roughly once per second, endlessly permuting label-ID combinations — a combinatorial crawler trap. Each response is ~43KB/10ms so load is minor, but it is sustained junk traffic filling traefik access logs (it was also the accidental reason git survived the DNS outage — its warm connection pool never re-resolved). Options: robots.txt disallow on /*/projects with filters; a traefik rate-limit or bot-block middleware on the git router (216.73.216.0/24 resolves to a known AI-crawler range); or Gitea-side setting to noindex filtered views.

kert added the infra label 2026-08-18 20:29:02 +00:00

kert referenced this issue from a commit

2026-09-11 20:47:06 +00:00

infra(traefik,gitea): rate-limit filtered project-board views per source IP and serve a robots.txt disallowing filtered views — the label-permutation crawler trap (refs #656)

kert commented

2026-09-11 20:47:30 +00:00

Fixed in 3a225c6 (live — traefik watches the dynamic dir). A dedicated router git-projects-filter (priority 1000) matches Host(git) && PathRegexp(^///projects) && QueryRegexp(labels, .+) on both entrypoints and applies a per-source-IP rateLimit (average 10/min, burst 20). Verified from the host: 30 rapid requests to /homelab/stack/projects/1?labels=… → 20×200 then 10×429; ordinary git traffic is untouched. A robots.txt disallowing filtered views (projects, ?labels=, ?state=, compare/commits/blame/raw, /api/) is served from Gitea's custom/public dir — note Cloudflare's managed robots.txt (AI-crawler content signals) is what the public sees at /robots.txt, so the rate limit is the effective control for a crawler that ignores robots anyway.

Fixed in 3a225c6 (live — traefik watches the dynamic dir). A dedicated router git-projects-filter (priority 1000) matches Host(git) && PathRegexp(^/<owner>/<repo>/projects) && QueryRegexp(labels, .+) on both entrypoints and applies a per-source-IP rateLimit (average 10/min, burst 20). Verified from the host: 30 rapid requests to /homelab/stack/projects/1?labels=… → 20×200 then 10×429; ordinary git traffic is untouched. A robots.txt disallowing filtered views (projects, ?labels=, ?state=, compare/commits/blame/raw, /api/) is served from Gitea's custom/public dir — note Cloudflare's managed robots.txt (AI-crawler content signals) is what the public sees at /robots.txt, so the rate limit is the effective control for a crawler that ignores robots anyway.

kert closed this issue

2026-09-11 20:47:31 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: homelab/stack#656