AI Crawler Index
Which crawler a User-Agent header belongs to, who operates it, whether it documents obeying robots.txt, how to verify it is genuine against the operator's own published IP ranges, and what blocking it costs you. Every endpoint is a GET returning JSON, CSV or plain text.HOST NOTE: the entire surface is served from www.pathwren.workers.dev, a Cloudflare Workers default subdomain rather than a custom domain. That is NOT an ephemeral tunnel -- unlike a trycloudflare quick tunnel, a workers.dev subdomain is stable and tied to the account -- but it is a platform-owned address, so the operator does not control it and it cannot be moved. Recorded rather than treated as disqualifying: everything it serves is real and complete. Verified 2026-09-01.
AI Crawler Index is published by AI Crawler Index on the APIs.io network, described by a machine-readable OpenAPI specification.
This API exposes 1 JSON Schema definition.
Tagged areas include AI crawlers, web crawlers, robots.txt, user agents, and bot detection. The published artifact set on APIs.io includes an OpenAPI specification, API documentation, authentication docs, rate-limit docs, a getting-started guide, an API reference, and 1 JSON Schema.
Requests are made against the base URL https://www.pathwren.workers.dev.
Metadata
The identity and technical contract details declared by the specification.
Specification
The full machine-readable OpenAPI contract behind this narrative.
Source
This is an independent, third-party profile of AI Crawler Index, published by API Evangelist. We do not operate, host, resell, or support these APIs, and we are not affiliated with or endorsed by the company unless stated above. Everything here is built from publicly available information — the company's own site, developer portal, documentation, public repositories, and the specifications it publishes for public use. Nothing is obtained by breaching a system, defeating an access control, or using credentials.
The Kin Score and Agent Readiness rating are independently calculated assessments of a company's public API artifacts, scored against a published rubric. They are not certifications, endorsements, security assessments, or audits.
Corrections, re-scores, and removal are free — no partnership or purchase required, and you do not need to justify the request. A removed company is recorded as unrated, never scored zero for having asked. Acknowledgement within one business day; removal within two.
info@apievangelist.com
·
Read the full data-sourcing policy →
On a security or compliance team? Put security in the subject line and
you will get a person, not a form — we will tell you exactly which public URLs this profile was built from.