# Search engines and AI assistant crawlers: welcome (GEO strategy).
# SEO-tool scrapers: blocked — they generate ~25% of origin traffic and zero value.

# ─── Crawl trap: point-in-time + reform-history pages ────────────────────
# Every law links to /history, and /history links to one /date/<d> per reform,
# and each /date/ page links to every other date. That's laws × reforms URLs
# (millions), so the CDN can never warm them — every crawl hit is a structural
# origin miss at ~1.7 s each. Measured 2026-08-01: 45% of requests, 90% of the
# Cloud Run CPU bill. Law pages, country pages and the sitemap stay indexable.
#
# The rest is auth-gated or tokenized, so crawling it buys nothing. /email/
# matters most: an unsubscribe link acts on GET, so a crawler that follows one
# opts a real user out.
# Denmark: 45,400 laws whose bodies are in no repository since 2026-04-16. Their
# pages answered 503 with Retry-After, an invitation to come back hourly forever;
# since 2026-10-21 they answer 410. Measured 2026-08-30: 155 of 1,000 origin requests
# were /dk/, 152 of them 503, and every 5xx ClaudeBot got that day was one. They
# are out of the sitemap too (see routes/seo.py). Lift both together the day the
# corpus is republished.

# Crawl-delay 5 -> 1 (2026-08-30). The 5 was set on 2026-08-01, when a law page
# cost ~1.7 s at the origin and crawlers were 90% of the Cloud Run CPU bill. Both
# halves of that are gone: the reforms query that took the site down is fixed,
# concurrency went from 80 to 16 requests per instance, and p90 is 1.17 s with
# the CDN still cold. Google and OpenAI ignore Crawl-delay anyway; the ones that
# honour it are Bing, Yandex and Amazon — and, per its own documentation,
# ClaudeBot. At 5 s the ceiling was 17,280 URLs/day against ~825,000 laws, which
# is 48 days for one pass of a corpus we want read and cited. At 1 s it is 86,400.
# Review the origin before raising anything else: see the calendar entry for
# 2026-09-04, and `make verify` for whether this file is even live.
User-agent: *
Crawl-delay: 1
Disallow: /*/law/*/date/
Disallow: /*/law/*/history
Disallow: /api/
Disallow: /dashboard
Disallow: /admin
Disallow: /billing/
Disallow: /email/
Disallow: /dk/

# No Googlebot group on purpose: Google ignores Crawl-delay, and its rules
# would be a copy of the ones above — a copy that drifts.

# No bingbot group either, since 2026-08-30: at Crawl-delay 1 for everyone it
# was a character-for-character copy of the group above, and a copy drifts.
# Bing does honour Crawl-delay — it is the index behind ChatGPT and Copilot — so
# if this file ever wants to treat it differently, that is the group to add back.

# Amazonbot honours robots.txt and Crawl-delay. It was the second most expensive
# crawler here — 75 requests and 55 s of origin in fifteen minutes — and Rufus
# and Alexa do not cite their sources, so it buys none of the visibility the
# rest of this file is written for. Slowed, not blocked.
User-agent: Amazonbot
Crawl-delay: 10
Disallow: /*/law/*/date/
Disallow: /*/law/*/history
Disallow: /api/
Disallow: /dashboard
Disallow: /admin
Disallow: /billing/
Disallow: /email/
Disallow: /dk/

# ─── Blocked: SEO/analytics scrapers (no search or AI visibility value) ───

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: DataForSeoBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: serpstatbot
Disallow: /

# Bytespider (TikTok/ByteDance) — aggressive crawler, ignores Crawl-delay,
# no assistant surface that cites sources. Blocked.
User-agent: Bytespider
Disallow: /

# Baidu indexes for a search market with no audience for European legislation.
User-agent: Baiduspider
Disallow: /

User-agent: Baiduspider-render
Disallow: /

# No product behind either that cites what it read.
User-agent: AionBot
Disallow: /

User-agent: GolemreachTrustBot
Disallow: /

Sitemap: https://legalize.dev/sitemap.xml