Crawl scope and host discrimination
Fixtures that make scope classification itself the subject under test - subdomain scope, external links, CDN-hosted first-party assets, tracker classification, and scheme/port variants.
Fixtures for crawl-scope behaviour: whether a crawler classifies each URL it discovers as internal, subdomain, external, CDN-hosted first-party, or blockable third-party - and how its configuration changes that outcome.
Families:
- Subdomain scope: a primary page linking to a sibling-subdomain page with a backlink into primary scope. The crawler’s subdomain setting decides whether the target is crawled as internal, status-checked only, or merely recorded.
- External links: a page whose single anchor is the shared external target constant. External URLs may be status-checked but never have their links extracted.
- CDN-hosted first-party resources: a page embedding a stylesheet, a script, and an image from a CDN-style asset host. Adding that host to the crawler’s allowed-CDN list makes the assets first-party page resources instead of third-party references.
- Tracking stubs: a page embedding a self-labelled analytics script and a 1x1 pixel from a dedicated stub host. Adding that host to the crawler’s custom blocked domains classifies both as blocked trackers. The stubs perform no tracking.
- Scheme and port variants: three anchors - plain, explicit
:443, andhttp://- to one primary-host path. Host comparison ignores scheme and default port, so all three are one internal URL. - Scope audit: a single page whose five anchors cover every scope class above, so a crawler’s scope report can be diffed against the registry in one crawl. It links off-infrastructure, so it stays out of every sitemap and is reachable only from this page.
Fixtures in this category
- Subdomain scope link —
scope-subdomain-link(safe) - Subdomain scope target —
scope-subdomain-target(safe) - CDN-hosted first-party resources —
scope-cdn-embedded-resources(safe) - CDN asset — stylesheet —
scope-cdn-asset-stylesheet(safe) - CDN asset — script —
scope-cdn-asset-script(safe) - CDN asset — image —
scope-cdn-asset-image(safe) - Third-party tracking stubs —
scope-tracking-stub-page(safe) - Tracking stub — analytics script —
scope-tracking-analytics-script(safe) - Tracking stub — beacon pixel —
scope-tracking-beacon-pixel(safe) - Scheme and default-port scope variants —
scope-protocol-port-variants(safe) - Scheme and port variants target —
scope-protocol-variants-target(safe)
