Utility endpoints

Crawler introspection endpoints that make crawler behaviour observable.

Diagnostic endpoints, not test fixtures: they mirror what the server observed about a request so crawler behaviour can be verified without a reporting database. All are deterministic, safe to crawl, stateless, and never cached.

GET /user-agent-mirror

Returns the requesting user agent as JSON; append ?format=text (or send Accept: text/plain) for a plain-text body.

GET /headers-mirror

Returns the request headers as JSON. Authorization, Proxy-Authorization, Cookie, and Set-Cookie values are redacted.

GET /ip-mirror

Returns the requester IP information visible to the app, including forwarded-header values. Forwarded values depend on the deployment path and should not be treated as absolute truth.

GET /timing

Returns server receive/respond timestamps and a duration marker.

GET /.well-known/security.txt

Stable RFC 9116 security contact content for discovery tests.

GET /robots.txt

The primary robots policy. Generated from registry contributions — never hand-edited.

GET /llms.txt

AI-crawler guidance, generated from the registry under the same policy.

Fixtures in this category

User agent mirror — user-agent-mirror (safe)
Headers mirror — headers-mirror (safe)
IP mirror — ip-mirror (safe)
Timing — timing (safe)
security.txt — security-txt (safe)
robots.txt — robots-txt (safe)
llms.txt — llms-txt (safe)

Back to the fixture index