How AI is applied across API Evangelist and APIs.io. Read my AI disclosure →
API Evangelist API Evangelist
Discovery
Learnings
Guidance
Toolbox
Alignment
API Evangelist LLC

AI Crawler Index Crawlers API

One record per crawler.

AI Crawler Index Crawlers API is one of 5 APIs that AI Crawler Index publishes on the APIs.io network, described by a machine-readable OpenAPI specification.

This API exposes 1 JSON Schema definition.

Tagged areas include Crawlers. The published artifact set on APIs.io includes an OpenAPI specification, API documentation, authentication docs, rate-limit docs, a getting-started guide, an API reference, and 1 JSON Schema.

This API exposes 57 operations across 57 paths, and defines 1 schema. It is described by OpenAPI 3.2.0, at version 2026-09-01.

Requests are made against a single base URL, https://www.pathwren.workers.dev.

57 operations 57 paths 1 schemas 57 GET

Metadata

The identity and technical contract details declared by the specification.

Specification
OpenAPI 3.2.0
API Version
2026-09-01
Server
https://www.pathwren.workers.dev
License
Resource Areas
1

Paths & Operations 57

Across 57 paths, the API surfaces 57 operations — 57 GET. Each is listed below with its method, path, parameters, and response codes.

Crawlers 57

One record per crawler.

GET
/crawler/{slug}.json
One crawler record by slug
getCrawler 1 param → 200404
GET
/crawler/gptbot.json
Record for GPTBot
crawler_gptbot → 200
GET
/crawler/oai-searchbot.json
Record for OAI-SearchBot
crawler_oai_searchbot → 200
GET
/crawler/chatgpt-user.json
Record for ChatGPT-User
crawler_chatgpt_user → 200
GET
/crawler/claudebot.json
Record for ClaudeBot
crawler_claudebot → 200
GET
/crawler/claude-searchbot.json
Record for Claude-SearchBot
crawler_claude_searchbot → 200
GET
/crawler/claude-user.json
Record for Claude-User
crawler_claude_user → 200
GET
/crawler/anthropic-ai.json
Record for anthropic-ai
crawler_anthropic_ai → 200
GET
/crawler/claude-web.json
Record for Claude-Web
crawler_claude_web → 200
GET
/crawler/google-extended.json
Record for Google-Extended
crawler_google_extended → 200
GET
/crawler/googlebot.json
Record for Googlebot
crawler_googlebot → 200
GET
/crawler/googleother.json
Record for GoogleOther
crawler_googleother → 200
GET
/crawler/google-cloudvertexbot.json
Record for Google-CloudVertexBot
crawler_google_cloudvertexbot → 200
GET
/crawler/google-inspectiontool.json
Record for Google-InspectionTool
crawler_google_inspectiontool → 200
GET
/crawler/googlebot-image.json
Record for Googlebot-Image
crawler_googlebot_image → 200
GET
/crawler/googlebot-news.json
Record for Googlebot-News
crawler_googlebot_news → 200
GET
/crawler/storebot-google.json
Record for Storebot-Google
crawler_storebot_google → 200
GET
/crawler/bingbot.json
Record for bingbot
crawler_bingbot → 200
GET
/crawler/applebot.json
Record for Applebot
crawler_applebot → 200
GET
/crawler/applebot-extended.json
Record for Applebot-Extended
crawler_applebot_extended → 200
GET
/crawler/perplexitybot.json
Record for PerplexityBot
crawler_perplexitybot → 200
GET
/crawler/perplexity-user.json
Record for Perplexity-User
crawler_perplexity_user → 200
GET
/crawler/ccbot.json
Record for CCBot
crawler_ccbot → 200
GET
/crawler/bytespider.json
Record for Bytespider
crawler_bytespider → 200
GET
/crawler/tiktokspider.json
Record for TikTokSpider
crawler_tiktokspider → 200
GET
/crawler/meta-externalagent.json
Record for meta-externalagent
crawler_meta_externalagent → 200
GET
/crawler/meta-externalfetcher.json
Record for meta-externalfetcher
crawler_meta_externalfetcher → 200
GET
/crawler/facebookexternalhit.json
Record for facebookexternalhit
crawler_facebookexternalhit → 200
GET
/crawler/facebookbot.json
Record for FacebookBot
crawler_facebookbot → 200
GET
/crawler/amazonbot.json
Record for Amazonbot
crawler_amazonbot → 200
GET
/crawler/duckassistbot.json
Record for DuckAssistBot
crawler_duckassistbot → 200
GET
/crawler/duckduckbot.json
Record for DuckDuckBot
crawler_duckduckbot → 200
GET
/crawler/ai2bot.json
Record for AI2Bot
crawler_ai2bot → 200
GET
/crawler/ai2bot-dolma.json
Record for Ai2Bot-Dolma
crawler_ai2bot_dolma → 200
GET
/crawler/cohere-ai.json
Record for cohere-ai
crawler_cohere_ai → 200
GET
/crawler/cohere-training-data-crawler.json
Record for cohere-training-data-crawler
crawler_cohere_training_data_crawler → 200
GET
/crawler/mistralai-user.json
Record for MistralAI-User
crawler_mistralai_user → 200
GET
/crawler/youbot.json
Record for YouBot
crawler_youbot → 200
GET
/crawler/diffbot.json
Record for Diffbot
crawler_diffbot → 200
GET
/crawler/omgilibot.json
Record for omgilibot
crawler_omgilibot → 200
GET
/crawler/omgili.json
Record for omgili
crawler_omgili → 200
GET
/crawler/webzio-extended.json
Record for Webzio-Extended
crawler_webzio_extended → 200
GET
/crawler/imagesiftbot.json
Record for ImagesiftBot
crawler_imagesiftbot → 200
GET
/crawler/timpibot.json
Record for Timpibot
crawler_timpibot → 200
GET
/crawler/semrushbot.json
Record for SemrushBot
crawler_semrushbot → 200
GET
/crawler/semrushbot-ocob.json
Record for SemrushBot-OCOB
crawler_semrushbot_ocob → 200
GET
/crawler/ahrefsbot.json
Record for AhrefsBot
crawler_ahrefsbot → 200
GET
/crawler/archive-org-bot.json
Record for archive.orgbot
crawler_archive_org_bot → 200
GET
/crawler/ia-archiver.json
Record for iaarchiver
crawler_ia_archiver → 200
GET
/crawler/yandexbot.json
Record for YandexBot
crawler_yandexbot → 200
GET
/crawler/baiduspider.json
Record for Baiduspider
crawler_baiduspider → 200
GET
/crawler/seznambot.json
Record for SeznamBot
crawler_seznambot → 200
GET
/crawler/yeti.json
Record for Yeti
crawler_yeti → 200
GET
/crawler/petalbot.json
Record for PetalBot
crawler_petalbot → 200
GET
/crawler/firecrawlagent.json
Record for FirecrawlAgent
crawler_firecrawlagent → 200
GET
/crawler/scrapy.json
Record for Scrapy
crawler_scrapy → 200
GET
/crawler/img2dataset.json
Record for img2dataset
crawler_img2dataset → 200

Schemas 1

The contract defines 1 schema that model the data the API accepts and returns. The most detailed is Crawler (17 properties). Each schema is shown below with its type and property counts.

Crawler
object
17 properties 5 required

Specification

The full machine-readable OpenAPI contract behind this narrative.

Source

pathwren-crawlers-api-openapi.yml Raw ↑

Other APIs AI Crawler Index publishes across the network.

AI Crawler Index Bulk API
AI Crawler Index Ip Ranges API
AI Crawler Index Robots API
AI Crawler Index Status API
Where this information came from

This is an independent, third-party profile of AI Crawler Index Crawlers API, published by API Evangelist. We do not operate, host, resell, or support these APIs, and we are not affiliated with or endorsed by the company unless stated above. Everything here is built from publicly available information — the company's own site, developer portal, documentation, public repositories, and the specifications it publishes for public use. Nothing is obtained by breaching a system, defeating an access control, or using credentials.

The Kin Score and Agent Readiness rating are independently calculated assessments of a company's public API artifacts, scored against a published rubric. They are not certifications, endorsements, security assessments, or audits.

Corrections, re-scores, and removal are free — no partnership or purchase required, and you do not need to justify the request. A removed company is recorded as unrated, never scored zero for having asked. Acknowledgement within one business day; removal within two.

info@apievangelist.com · Read the full data-sourcing policy →
On a security or compliance team? Put security in the subject line and you will get a person, not a form — we will tell you exactly which public URLs this profile was built from.