Robots protocol

Root robots.txt behaviours on dedicated profile origins, plus meta robots, X-Robots-Tag, and canonical-versus-blocked conflict fixtures on the primary site.

Fixtures for the robots exclusion protocol and its page-level cousins. Robots rules are origin-scoped, so every root robots.txt behaviour is owned by a dedicated supporting-profile origin rather than simulated under a subpath of the primary site.

Families:

• Root robots.txt profiles: fifteen origins covering allow-all, block-all, blank Disallow, Allow overrides, Crawl-delay variants, sitemap references, wildcard and end-anchor patterns, multiple user-agent groups, AI crawler-specific rules, malformed syntax, 404 and 500 responses, a UTF-8 BOM, an oversized file, and a redirecting robots.txt. The registry records the contract each profile origin must serve; the profile services themselves are deployed separately.
• Meta robots: head-placed noindex, combined directive lists, uppercase directives, conflicting tags, a bot-specific googlebot tag, and a body-placed tag that conforming parsers must ignore.
• X-Robots-Tag: the header form of the directives, on an HTML response and on a generated PDF.
• Conflicts: a crawlable page whose canonical link targets a page that robots.txt disallows.

The block-all, AI-specific, malformed, 500, and oversized robots origins plus the conflict fixture are risky: they live in the risky manifest only and are direct-entry-only so broad crawls never trip them.

Fixtures in this category

• Root robots.txt — allow all — robots-root-allow-all (safe)
• Root robots.txt — blank Disallow — robots-root-blank-disallow (safe)
• Root robots.txt — Allow overrides Disallow — robots-root-allow-override (safe)
• Root robots.txt — Crawl-delay variants — robots-crawl-delay-variants (safe)
• Root robots.txt — valid and missing sitemap references — robots-sitemap-valid-and-missing (safe)
• Root robots.txt — wildcard and end-anchor patterns — robots-wildcard-and-end-anchor (safe)
• Root robots.txt — multiple user-agent groups — robots-multiple-user-agents (safe)
• Root robots.txt — HTTP 404 — robots-404 (safe)
• Root robots.txt — UTF-8 BOM — robots-bom (safe)
• Root robots.txt — redirect before content — robots-redirect (safe)
• Meta robots — noindex — robots-meta-noindex (safe)
• Meta robots — combined nofollow, none, noarchive — robots-meta-nofollow-none-noarchive (safe)
• Meta robots — uppercase NOINDEX — robots-meta-uppercase (safe)
• Meta robots — conflicting tags — robots-meta-conflict (safe)
• Meta robots — bot-specific googlebot tag — robots-meta-bot-specific (safe)
• Meta robots — body placement is ignored — robots-meta-body-ignored (safe)
• X-Robots-Tag — noindex on HTML — robots-x-robots-html (safe)
• X-Robots-Tag — noindex on PDF — robots-x-robots-pdf (safe)

Back to the fixture index