AI modern crawlers

AI crawler fixtures covering per-bot robots.txt rules, bot-scoped meta robots and X-Robots-Tag directives, llms.txt discovery variants, JavaScript-gated versus server-rendered content, and a self-labelled bot-specific content marker.

Fixtures that let a crawler demonstrate how it handles AI-agent-specific signals.

Families:

• Per-bot robots directives: nine 200 target pages, each contributing a generated robots.txt stanza that disallows its path for exactly one AI user-agent token (GPTBot, ChatGPT-User, ClaudeBot, Claude-Web, Amazonbot, Google-Extended, OAI-SearchBot, PerplexityBot, cohere-ai). Whole-origin AI policies live on the robots-ai-specific profile origin.
• Bot-scoped directives: one meta robots element scoped to the GPTBot agent name and one X-Robots-Tag: ClaudeBot: noindex header, establishing the per-bot directive pattern once each.
• llms.txt: the primary origin’s generated baseline plus four profile origins serving valid, optional-section, missing (404), and malformed variants at their roots.
• JavaScript versus server-rendered content: a JS-gated article whose content and rendered marker only exist after script execution, its server-rendered control serving the same article raw, and a reference page cross-referencing the JavaScript rendering group’s proof fixtures.
• Bot-specific content marker: one endpoint whose marker names the AI bot identity matched in the User-Agent, always with Vary: User-Agent and a visible label declaring the deliberate fixture — never deceptive cloaking.
• Policy reference pages: the Google-Extended-versus-Googlebot and OAI-SearchBot-versus-GPTBot contrasts, both real in the generated robots.txt.

The UA-conditional fixtures are moderate because they need the visible self-label to stay non-deceptive; the full-policy origin and the malformed llms.txt are risky and direct-entry only.

Fixtures in this category

• AI robots blocked target — GPTBot — ai-robots-gptbot (safe)
• AI robots blocked target — ChatGPT-User — ai-robots-chatgpt-user (safe)
• AI robots blocked target — ClaudeBot — ai-robots-claudebot (safe)
• AI robots blocked target — Claude-Web — ai-robots-claude-web (safe)
• AI robots blocked target — Amazonbot — ai-robots-amazonbot (safe)
• AI robots blocked target — Google-Extended — ai-robots-google-extended (safe)
• AI robots blocked target — OAI-SearchBot — ai-robots-oai-searchbot (safe)
• AI robots blocked target — PerplexityBot — ai-robots-perplexitybot (safe)
• AI robots blocked target — cohere-ai — ai-robots-cohere-ai (safe)
• Bot-scoped meta robots — GPTBot noindex — ai-meta-noai-gptbot (moderate)
• Bot-scoped X-Robots-Tag — ClaudeBot noindex — ai-xrobots-claudebot (moderate)
• llms.txt — valid — ai-llms-valid (safe)
• llms.txt — optional section — ai-llms-optional-section (safe)
• llms.txt — missing — ai-llms-missing (safe)
• JavaScript-gated article content — ai-js-gated-content (safe)
• Server-rendered equivalent control — ai-ssr-equivalent-control (safe)
• JavaScript execution detection reference — ai-js-execution-detection (safe)
• Bot-specific content marker — ai-bot-specific-content-marker (moderate)
• Training-control disallow reference — ai-training-disallow-reference (safe)
• Search-bot allow reference — ai-search-bot-allow-reference (safe)

Back to the fixture index