Safe crawling
How to crawl this site without falling into deliberate traps.
A default crawl of this site is always safe. The default sitemap, the navigation, and the safe manifest only ever contain fixtures that behave like ordinary pages.
Risky fixtures — redirect loops, spider traps, oversized responses, deliberate slowness — are never linked from safe surfaces. They are published only in the risky manifest and must be entered directly, one URL at a time, by a crawler operator who intends to test them.
Helper routes (intermediate hops of chains and clusters) appear only in the
canonical manifest, marked with routeClass: helper. Never use a helper route
as a crawl entry point.
Fixtures that intentionally exit this site’s infrastructure always target
https://example.com/ and are flagged external: true in the registry; they
are excluded from the default sitemap so an unattended crawl never leaves the
fixture domain.