Sitebulb Crawler Test Site

Deterministic fixtures for exercising the crawler.

Documentation

• Introduction — What the Sitebulb Crawler Test Site is and how to use it.
• Safe crawling — How to crawl this site without falling into deliberate traps.
• Crawler behaviour — How a crawler's observations are compared against fixture expectations.
• Infrastructure requirements — Environment capabilities and how unavailable fixtures are reported.
• Sitebulb hints — How fixtures link to the Sitebulb hint engine.

Categories

• Foundation example — The minimal proving fixture for the registry-to-route pipeline.
• Utility endpoints — Crawler introspection endpoints that make crawler behaviour observable.
• HTTP status codes — Exact status fixtures across informational, success, redirect, client-error, and server-error ranges, plus soft 404 variants.
• Redirects — HTTP 3xx redirects, chains, loops, meta refresh, JavaScript redirects, and redirect edge cases.
• Robots protocol — Root robots.txt behaviours on dedicated profile origins, plus meta robots, X-Robots-Tag, and canonical-versus-blocked conflict fixtures on the primary site.
• Canonical tags — Canonical link and HTTP Link header fixtures across self, conflicting, blocked, redirected, relative, protocol, fragment, and external scenarios.
• Links — Link discovery fixtures covering standard anchors, nofollow variants, JavaScript-built links, hidden links, iframes, malformed markup, and a 10,000-anchor page.
• URLs and normalisation — URL format, encoding, duplicate-parameter, tracking-parameter, internationalisation, host, and path normalisation fixtures.
• JavaScript Rendering — Fixtures that distinguish crawlers that render JavaScript from crawlers that only parse the initial HTML response, covering injected content, delays, metadata mutation, interaction-gated content, shadow DOM, and service workers.
• Content and meta tags — Title, meta description, heading, charset, and content-quality fixtures with deterministic markers for extraction checks.
• Structured data — Format-level JSON-LD, Microdata, and RDFa fixtures covering valid types, invalid payloads, placement, multiplicity, JavaScript injection, and content conflicts.
• Mobile Responsive — Mobile-crawling fixtures covering a responsive single-URL baseline, alternate mobile URL pairing, User-Agent dynamic serving, AMP relationship discovery, and viewport and rendering defects.
• HTTP Headers — Cache directives, conditional validators, content negotiation, compression and transfer encodings, security headers, and response-timing fixtures that exercise how a crawler reads the wire.
• Sitemaps — XML sitemap fixtures covering parsing, the 50,000-URL protocol limit, gzip handling, malformed XML, sitemap index traversal, extension namespaces, and sitemap-only discovery.
• Spider traps — Bounded spider-trap and honeytrap fixtures covering path depth, incrementing parameters, calendar walks, session-id URLs, faceted explosion, unique-token chains, hidden links, and a 10 MB body.
• Authentication and access control — HTTP auth challenges, cookie-gated content, expired sessions, SameSite scoping, login surfaces, and redirect-to-login flows.
• Pagination — Paginated-series fixtures covering anchor traversal, rel-next/prev head links, correct and incorrect canonicalisation, sitemap-only discovery, noindex inside a series, client-side loading, numbered-range boundaries, empty final pages, and duplicate page content.
• Non-HTML Resources — Fixtures covering how a crawler fetches, classifies, and reports PDFs, images, text and structured-data files, feeds, large binaries, mismatched content types, download dispositions, range requests, and broken resource references.
• Structured data features — Per-feature validation fixtures for every Google search-gallery rich-result feature, with known pass and fail states.
• International micro-sites — Six self-contained hreflang micro-sites for end-to-end international audits — three correct, two broken, one edge-case.
• International and hreflang — Per-hint hreflang fixtures covering the three declaration channels, reciprocity checking, the standard failure modes, and RTL documents.
• AI modern crawlers — AI crawler fixtures covering per-bot robots.txt rules, bot-scoped meta robots and X-Robots-Tag directives, llms.txt discovery variants, JavaScript-gated versus server-rendered content, and a self-labelled bot-specific content marker.
• Exotic advanced tests — Advanced crawler fixtures covering sandboxed iframes and legacy framesets, non-GET HTTP method handling, browser-environment fingerprint exposure, TLS fingerprint echo, web components, bounded streaming, and parser stress recovery.
• Images and Media Optimisation — Fixtures covering alt-text extraction, image dimension and scaling analysis, encoding-quality and next-gen format opportunities, animated-GIF-versus-video classification, hidden and offscreen image detection, and favicon handling.
• Asset Weight and Duplication — Fixtures covering minification detection on CSS/JS/HTML, duplicate and identical resource grouping, single-asset and combined per-category size thresholds, same-origin request counting, and enormous-network-payload detection.
• Loading, Webfonts and Render-Blocking — Loading-behaviour fixtures covering render-blocking head resources, critical-CSS placement, synchronous script chains, single points of failure, print stylesheets, webfont count/weight/font-display behaviour, and unused CSS/JavaScript via Chrome coverage.
• Web Vitals and Performance Lab — Capability fixtures producing known, bounded Web Vitals and performance-budget values so crawler tests can validate LCP, CLS, TBT/TTI, FCP, and TTFB capture plus per-category transferred-size buckets.
• Accessibility — Deliberate axe-core rule violations grouped one rule family per page, plus the helper frame and media assets they embed.
• Content quality and language — Calibrated word counts, readability and sentiment extremes, template subtraction, rendered word divergence, multilingual spelling, and a near-duplicate similarity gradient.
• Technology and Infrastructure Fingerprints — Self-labelled pages and endpoints that emit one technology or CDN/WAF signature set each, exercising Sitebulb's Wappalyzer technology detection and header-based CDN/WAF identification without any real stack installed.
• Custom Extraction and Search Targets — Frozen, exact-valued pages that validate Sitebulb's three user-configurable data-collection pipelines - content extraction rules, content search terms, and custom JavaScript snippets.
• Crawl scope and host discrimination — Fixtures that make scope classification itself the subject under test - subdomain scope, external links, CDN-hosted first-party assets, tracker classification, and scheme/port variants.

Fixtures

Machine-readable outputs

• /manifest.json — canonical manifest
• /manifest.safe.json — safe crawl inputs
• /manifest.risky.json — direct-entry-only fixtures
• /manifest.csv — flat export
• /manifest.schema.json — manifest JSON schema
• /sitemap.xml — default sitemap
• /robots.txt
• /llms.txt
• /health