3a225c620f
Some checks failed
CI / lint (push) Successful in 30s
CI / notebooks-smoke (push) Successful in 1m33s
Deploy / notebooks (push) Has been skipped
Deploy / zotero (push) Has been skipped
Deploy / docs (push) Has been skipped
Deploy / api (push) Has been skipped
Deploy / llm (push) Has been skipped
Deploy / mc (push) Has been skipped
Infra CI / notebooks (push) Successful in 52s
Infra CI / zotero (push) Successful in 14s
Infra CI / docs (push) Successful in 19s
Infra CI / api (push) Successful in 58s
Infra CI / llm (push) Successful in 50s
Infra CI / mc (push) Failing after 16s
Deploy / report (push) Successful in 12s
CI / test (push) Has been cancelled
23 lines
700 B
Plaintext
23 lines
700 B
Plaintext
# Served by Gitea at /robots.txt (GITEA_CUSTOM/public). #656: a crawler
|
|
# permuted every label-filter combination of the project board about once
|
|
# a second for weeks — filtered views are a combinatorial trap and carry
|
|
# nothing worth indexing. The traefik router `git-projects-filter` rate
|
|
# limits the same URLs for clients that ignore this file.
|
|
User-agent: *
|
|
Disallow: /*/*/projects
|
|
Disallow: /*/*/issues?
|
|
Disallow: /*/*/pulls?
|
|
Disallow: /*/*/milestones?
|
|
Disallow: /*?labels=
|
|
Disallow: /*?state=
|
|
Disallow: /*?sort=
|
|
Disallow: /*?q=
|
|
Disallow: /*/*/compare/
|
|
Disallow: /*/*/commits/
|
|
Disallow: /*/*/blame/
|
|
Disallow: /*/*/raw/
|
|
Disallow: /*/*/search
|
|
Disallow: /explore
|
|
Disallow: /api/
|
|
Crawl-delay: 10
|