Non-HTML Resources
Fixtures covering how a crawler fetches, classifies, and reports PDFs, images, text and structured-data files, feeds, large binaries, mismatched content types, download dispositions, range requests, and broken resource references.
Fixtures that let a crawler prove it classifies non-HTML responses by their
declared type instead of guessing, attributes header-only directives to
URLs that have no <head>, and degrades gracefully on delivery edge cases.
Families:
- Documents: a committed one-page PDF (no robots directives), the same
bytes re-served with
X-Robots-Tag: noindex, and a 302 whose single hop lands on the PDF. PDF text extraction varies widely between crawlers; the marker sits in an uncompressed content stream so it is grep-able in raw bytes. - Images: a committed 64x64 PNG whose marker lives in a
tEXtchunk, plus the HTML page that embeds it. The PNG is kept out of every sitemap so embed-based discovery is measurable. - Data formats: committed plain-text, CSV, JSON, and generic XML files, each served with its exact declared content type. None of them should be parsed for links or misclassified as HTML, feeds, or sitemaps.
- Feeds: minimal valid RSS 2.0 and Atom feeds whose two item links are absolute URLs to feed-item target pages excluded from every sitemap, so reaching those pages proves feed processing. Feed handling varies widely between crawlers.
- Delivery edge cases: an 8 MB streamed binary with a concurrency cap of
4 (excess requests get a deterministic 429 with Retry-After), an unknown
MIME type served verbatim, PNG bytes deliberately declared as
text/html, an attachment content disposition, and a 1 MB file supporting range requests (206 for valid ranges, 416 beyond EOF). - Broken references: an HTML page referencing one missing image and one missing stylesheet; both targets are registered 404 fixtures so the broken subresource fetches are deliberate and validated.
Everything is safe to crawl except the large binary, which is risky and direct-entry only so casual crawls never trigger the expensive generator; the mismatched and unknown content types are moderate because they probe error handling rather than ordinary fetching.
Fixtures in this category
- PDF resource — basic —
resource-pdf-basic(safe) - PDF resource — X-Robots-Tag noindex —
resource-pdf-x-robots-noindex(safe) - Redirect to a PDF resource —
resource-redirect-to-pdf(safe) - PNG image resource —
resource-image-png(safe) - Image embedded from HTML —
resource-image-linked-from-html(safe) - Plain text resource —
resource-text-plain(safe) - CSV resource —
resource-csv(safe) - JSON resource —
resource-json(safe) - Generic XML resource —
resource-xml(safe) - RSS 2.0 feed —
resource-rss(safe) - Atom feed —
resource-atom(safe) - Feed item target — one —
resource-feed-item-one(safe) - Feed item target — two —
resource-feed-item-two(safe) - Unknown MIME type —
resource-unknown-mime(moderate) - Wrong content type —
resource-wrong-content-type(moderate) - Download attachment disposition —
resource-download-content-disposition(safe) - Range request file —
resource-range-request-file(moderate) - Broken resource references —
resource-404-file-reference(safe)
