# Content-Signal (contentsignals.org) in every group: our pages may be indexed, quoted in assistants' answers and
# used to train models (the owner, 26 Oct 2026: agents first). A bot reads only its own group, so each repeats it.
#
# Private pages (settings, sign-in, watchlist, editor) and Pro tools carry noindex in every language (web/src/lib/seo.ts):
# they are not blocked here, or crawlers could not read it.
#
# Crawlers are never blocked (the owner's rule, 17 Oct 2026; docs/load.md "Crawlers"), and the main ones are never
# slowed: Googlebot, Bingbot, AhrefsSiteAudit, Applebot and DuckDuckBot read the `*` group, which has no Crawl-delay
# (Bing and Ahrefs honour one, so it must never go under `*`). AhrefsBot alone is asked for two seconds: it built up to
# 80 pages a minute (the owner, 26 Oct 2026); the API still never refuses it. The bots below, AI and SEO crawlers that come
# in many parallel threads, are asked for a second between requests, the SEO tools (no readers come
# from them: Semrush, MJ12, DotBot, Barkrowler, DataForSEO, Seekport) for ten. A bot reads only its own group, so each group
# repeats everything `*` allows. Those that ignore Crawl-delay are paced by the API (config/load.php `crawler_pace`).
User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes

User-agent: GPTBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 1

User-agent: OAI-SearchBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 1

User-agent: ChatGPT-User
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 1

User-agent: ClaudeBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 1

User-agent: Claude-SearchBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 1

User-agent: PerplexityBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 1

User-agent: CCBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 1

User-agent: Bytespider
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 1

User-agent: Amazonbot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 1

User-agent: meta-externalagent
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 1

User-agent: SemrushBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 10

User-agent: MJ12bot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 10

User-agent: DotBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 10

User-agent: PetalBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 1

User-agent: YandexBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 1

User-agent: Barkrowler
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 10

User-agent: DataForSeoBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 10
User-agent: ExaSearchBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 1

User-agent: AhrefsBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 2

User-agent: SeekportBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Crawl-delay: 10

Sitemap: https://api.keelstats.com/sitemap.xml