Algolia Crawler API
Manages Algolia's hosted web crawler that extracts content from websites and pushes it into indices on a schedule. The only Algolia API that authenticates with HTTP Basic rather than the x-algolia-* header pair.
This API exposes 20 operations across 17 paths, and defines 117 schemas. It is described by OpenAPI 3.2.0, at version 1.0.0.
Requests are made against a single base URL, https://crawler.algolia.com/api.
20 operations
17 paths
117 schemas
1 DELETE9 GET2 PATCH8 POST
Metadata
The identity and technical contract details declared by the specification.
Authentication & Security 1
Algolia Crawler API declares
1 security scheme
for authenticating requests.
It accepts HTTP basic authentication (BasicAuth).
By default, every request must be authenticated.
Paths & Operations 20
Across 17 paths, the API surfaces 20 operations — 1 DELETE, 9 GET, 2 PATCH, 8 POST. Each is listed below with its method, path, parameters, and response codes.
crawler 20
GET
/1/crawlers
List crawlers
POST
/1/crawlers
Create a crawler
GET
/1/crawlers/{id}
Retrieve crawler details
PATCH
/1/crawlers/{id}
Replace crawler configuration
DELETE
/1/crawlers/{id}
Delete a crawler
POST
/1/crawlers/{id}/run
Unpause a crawler
POST
/1/crawlers/{id}/pause
Pause a crawler
POST
/1/crawlers/{id}/reindex
Start a crawl
POST
/1/crawlers/{id}/test
Test crawl a URL
POST
/1/crawlers/{id}/urls/crawl
Crawl URLs
GET
/1/crawlers/{id}/stats/urls
Retrieve crawler stats
GET
/1/crawlers/{id}/crawl_runs
List crawler runs
GET
/1/crawlers/{id}/{logId}/download
Crawler run file
POST
/1/crawlers/{id}/delete_runs
Crawler run record deletion
PATCH
/1/crawlers/{id}/config
Update crawler configuration
GET
/1/crawlers/{id}/config/versions
List configuration versions
GET
/1/crawlers/{id}/config/versions/{version}
Retrieve a configuration version
GET
/1/crawlers/{id}/tasks/{taskID}
Retrieve task status
POST
/1/crawlers/{id}/tasks/{taskID}/cancel
Cancel a blocking task
GET
/1/domains
List registered domains
Schemas 117
The contract defines 117 schemas that model the data the API accepts and returns. The most detailed are indexSettingsAsSearchParams (41 properties), Configuration (26 properties), baseIndexSettings (20 properties), Action (12 properties). Each schema is shown below with its type and property counts.
facetOrdering
object
Order of facet names and facet values in your UI.
2 properties
crawlersResponse
loginRequestOptions
object
Options for the HTTP request for logging in.
4 properties
CrawlerName
string
Name of the crawler.
widgets
object
Widgets returned from any rules that are applied to the current search.
1 property
UrlsCrawledGroup
object
Processed URLs and their status.
5 properties
cache
object
Whether the crawler should cache crawled pages. For more information, see [Partial crawls with caching](https://www.algolia.com/doc/tools/crawler/getting-start…
1 property
maxValuesPerFacet
integer
Maximum number of facet values to return for each facet.
reRankingApplyFilter
Restrict [Dynamic Re-Ranking](https://www.algolia.com/doc/guides/algolia-ai/re-ranking) to records that match these filters.
parameters_order
string
Order of the query.
IndexSettings_advancedSyntaxFeatures
array
Advanced search syntax features you want to support - exactPhrase. Phrases in quotes must match exactly. For example, sparkly blue "iPhone case" only returns r…
urlsCrawledGroupStatus
string
Crawled URL status. For more information, see [Troubleshooting by crawl status](https://www.algolia.com/doc/tools/crawler/troubleshooting/crawl-status).
ignorePlurals
Treat singular, plurals, and other forms of declensions as equivalent. Only use this feature for the languages used in your index.
replaceSynonymsInHighlight
boolean
Whether to replace a highlighted word with the matched synonym By default, the original words are highlighted even if a synonym matches. For example, with home…
restrictHighlightAndSnippetArrays
boolean
Whether to restrict highlighting and snippeting to items that at least partially matched the search query. By default, all items are highlighted and snippeted.
disableTypoToleranceOnAttributes
array
Attributes for which you want to turn off [typo tolerance](https://www.algolia.com/doc/guides/managing-results/optimize-search-results/typo-tolerance). Attribu…
until
string
Unix string 'until' date.
mode
string
Search mode the index will use to query for results. This setting only applies to indices, for which Algolia enabled NeuralSearch for you.
removeStopWords
Removes stop words from the search query. Stop words are common words like articles, conjunctions, prepositions, or pronouns that have little or no meaning on…
safetyChecks
object
Checks to ensure the crawl was successful. For more information, see the [Safety checks](https://www.algolia.com/doc/tools/crawler/getting-started/crawler-conf…
1 property
baseIndexSettings
object
20 properties
relevancyStrictness
integer
Relevancy threshold below which less relevant results aren't included in the results You can only set relevancyStrictness on [virtual replica indices](https://…
page
integer
Current page of the paginated API response.
accessTokenRequest
object
Parameters required to make the [access token request](https://datatracker.ietf.org/doc/html/rfc6749section-4.4.2).
6 properties
4 required
minProximity
integer
Minimum proximity score for two matching words This adjusts the [Proximity ranking criterion](https://www.algolia.com/doc/guides/managing-results/relevance-ove…
urlsCrawledGroupCategory
string
Step where the status information was generated. For more information, see [Troubleshooting by crawl status](https://www.algolia.com/doc/tools/crawler/troubles…
allowTyposOnNumericTokens
boolean
Whether to allow typos on numbers in the search query Turn off this setting to reduce the number of irrelevant matches when searching in large sets of similar…
browserRequest
object
Information for using a web browser for authorization. The browser loads a login page and enters the provided credentials.
4 properties
3 required
sortRemainingBy
string
Order of facet values that aren't explicitly positioned with the order setting. - count. Order remaining facet values by decreasing count. The count is the num…
bannerImageUrl
object
URL for an image to show inside a banner.
1 property
domainsResponse
disableExactOnAttributes
array
Searchable attributes for which you want to [turn off the Exact ranking criterion](https://www.algolia.com/doc/guides/managing-results/optimize-search-results/…
CrawlerID
string
Universally unique identifier (UUID) of the crawler.
advancedSyntaxFeatures
string
indexSettingsAsSearchParams
object
41 properties
renderingContent
object
Extra data that can be used in the search UI. You can use this to control aspects of your search UI, such as the order of facet names and values without changi…
3 properties
renderJavaScript
If true, use a Chrome headless browser to crawl pages. Because crawling JavaScript-based web pages is slower than crawling regular HTML pages, you can apply th…
Configuration
object
Crawler configuration.
26 properties
3 required
updatedAt
string
Date and time when the object was updated, in RFC 3339 format.
requestOptions
object
Lets you add options to HTTP requests made by the crawler.
4 properties
facets
object
Order of facet names.
1 property
responseFields
array
Properties to include in the API response of search and browse requests By default, all response properties are included. To reduce the response size, you can…
supportedLanguage
string
ISO code for a supported language.
offset
integer
Offset of the query results.
headers
object
Headers to add to all requests.
3 properties
TaskID
string
Universally unique identifier (UUID) of the task.
ignoreCanonicalTo
Action
object
How to process crawled URLs. Each action defines: - The targeted subset of URLs it processes. - What information to extract from the web pages. - The Algolia i…
12 properties
2 required
pathAliases
object
Key-value pairs to replace matching paths with new values. It doesn't replace: - URLs in the startUrls, sitemaps, pathsToMatch, and other settings - Paths foun…
distinct
Determines how many records of a group are included in the search results. Records with the same value for the attributeForDistinct attribute are considered a…
crawlerConfigVersionsResponse
authorId
string
Universally unique identifier (UUID) of the user who created this version of the configuration.
order
array
Explicit order of facets or facet values. This setting lets you always show specific facets or facet values at the top of the list.
redirectURL
object
The redirect rule container.
1 property
from
string
Unix string 'from' date.
booleanString
string
bannerImage
object
Image to show inside a banner.
2 properties
banner
object
Banner with image and link to redirect users.
2 properties
hostnameAliases
object
Key-value pairs to replace matching hostnames found in a sitemap, on a page, in canonical links, or redirects. During a crawl, this action maps one hostname to…
grantType
string
OAuth 2.0 grant type.
minWordSizefor1Typo
integer
Minimum number of characters a word in the search query must contain to accept matches with [one typo](https://www.algolia.com/doc/guides/managing-results/opti…
BaseResponse
object
10 properties
8 required
IndexSettings_alternativesAsExact
array
Determine which plurals and synonyms should be considered an exact matches By default, Algolia treats singular and plural forms of a word, and single-word syno…
GetCrawlerResponse
oauthRequest
object
Authorization information for using the [OAuth 2.0 client credentials](https://datatracker.ietf.org/doc/html/rfc6749section-4.4) authorization grant. OAuth aut…
1 property
1 required
snippetEllipsisText
string
String used as an ellipsis indicator when a snippet is truncated.
fetchRequest
object
Information for making a HTTP request for authorization.
2 properties
1 required
maxFacetHits
integer
Maximum number of facet values to return when [searching for facet values](https://www.algolia.com/doc/guides/managing-results/refine-results/faceting/search-f…
values
object
Order of facet values. One object for each facet.
alternativesAsExact
string
bannerLink
object
Link for a banner defined in the Merchandising Studio.
1 property
total
integer
Total number of retrievable items.
waitTime
object
Timeout for the HTTP request.
2 properties
queryLanguages
array
Languages for language-specific query processing steps such as plurals, stop-word removal, and word-detection dictionaries. This setting sets a default list of…
PartialConfig
Crawler configuration to update. You can only update top-level configuration properties. To update a nested configuration, such as actions.recordExtractor, you…
configurationRecordExtractorType
string
attributesToHighlight
array
Attributes to highlight By default, all searchable attributes are highlighted. Use to highlight all attributes or use an empty array [] to turn off highlightin…
minWordSizefor2Typos
integer
Minimum number of characters a word in the search query must contain to accept matches with [two typos](https://www.algolia.com/doc/guides/managing-results/opt…
advancedSyntax
boolean
Whether to support phrase matching and excluding words from search queries Use the advancedSyntaxFeatures parameter to control which feature is supported.
decompoundQuery
boolean
Whether to split compound words in the query into their building blocks For more information, see [Word segmentation](https://www.algolia.com/doc/guides/managi…
enableRules
boolean
Whether to enable rules.
optionalWords
Words that should be considered optional when found in the query. By default, records must match all words in the search query to be included in the search res…
applicationID
string
Algolia application ID where the crawler creates and updates indices.
CrawlerLogID
string
Universally unique identifier (UUID) of the crawler log.
userData
An object with custom data. You can store up to 32kB as custom data.
schedule
string
Schedule for running the crawl. Instead of manually starting a crawl each time, you can set up a schedule for automatic crawls. [Use the visual UI](https://www…
highlightPostTag
string
HTML tag to insert after the highlighted parts in all highlighted results and snippets.
removeWordsIfNoResults
string
Strategy for removing words from the query when it doesn't return any results. This helps to avoid returning empty search results. - none. No words are removed…
createdAt
string
Date and time when the object was created, in RFC 3339 format.
typoTolerance
Whether [typo tolerance](https://www.algolia.com/doc/guides/managing-results/optimize-search-results/typo-tolerance) is enabled and how it is applied. If typo…
queryType
string
Determines if and how query words are interpreted as prefixes. By default, only the last query word is treated as a prefix (prefixLast). To turn off prefix sea…
Pagination
object
Pagination information.
3 properties
login
Authorization method and credentials for crawling protected content. The Crawler supports these authentication methods: - Basic authentication. The Crawler obt…
enableReRanking
boolean
Whether this search will use [Dynamic Re-Ranking](https://www.algolia.com/doc/guides/algolia-ai/re-ranking) This setting only has an effect if you activated Dy…
indexSettings
Index settings.
hitsPerPage
integer
Number of hits per page.
urlPattern
string
Use [micromatch](https://github.com/micromatch/micromatch) for negation, wildcards, and more.
limit
integer
Limit of the query results.
exactOnSingleWordQuery
string
Determines how the [Exact ranking criterion](https://www.algolia.com/doc/guides/managing-results/optimize-search-results/override-search-engine-defaults/in-dep…
itemsPerPage
integer
Number of items per page of the paginated API response.
attributesToRetrieve
array
Attributes to include in the API response To reduce the size of your response, you can retrieve only some of the attributes. Attribute names are case-sensitive…
enablePersonalization
boolean
Whether to enable Personalization.
sortFacetValuesBy
string
Order in which to retrieve facet values - count. Facet values are retrieved by decreasing count. The count is the number of matching records containing this fa…
highlightPreTag
string
HTML tag to insert before the highlighted parts in all highlighted results and snippets.
typoToleranceEnum
string
- min. Return matches with the lowest number of typos. For example, if you have matches without typos, only include those. But if there are no matches without…
value
object
3 properties
beforeIndexPublishing
object
Checks triggered after the crawl finishes but before the records are added to the Algolia index.
2 properties
attributesToSnippet
array
Attributes for which to enable snippets. Attribute names are case-sensitive Snippets provide additional context to matched words. If you enable snippets, they…
banners
array
Banners defined in the Merchandising Studio for a given search.
hide
array
Hide facet values.
optionalWordsArray
array
List of [optional words](https://www.algolia.com/doc/guides/managing-results/optimize-search-results/empty-or-insufficient-results/creating-a-list-of-optional-…
extraParameters
object
Extra parameters for the authorization request.
1 property
version
integer
Version of the configuration. Version 1 is the initial configuration you used when creating the crawler.
WithConfiguration
9 required
attributeCriteriaComputedByMinProximity
boolean
Whether the best matching attribute should be determined by minimum proximity This setting only affects ranking if the Attribute ranking criterion comes before…
fileTypes
string
For more information, see [Extract data from non-HTML documents](https://www.algolia.com/doc/tools/crawler/extracting-data/non-html-documents).
semanticSearch
object
Settings for the semantic search part of NeuralSearch. Only used when mode is neuralSearch.
1 property
Specification
The full machine-readable OpenAPI contract behind this narrative.
Source
#Crawler#Ingestion#REST
Where this information came from
This is an independent, third-party profile of Algolia Crawler API, published by API Evangelist. We do not operate, host, resell, or support these APIs, and we are not affiliated with or endorsed by the company unless stated above. Everything here is built from publicly available information — the company's own site, developer portal, documentation, public repositories, and the specifications it publishes for public use. Nothing is obtained by breaching a system, defeating an access control, or using credentials.
The Kin Score and Agent Readiness rating are independently calculated assessments of a company's public API artifacts, scored against a published rubric. They are not certifications, endorsements, security assessments, or audits.
Corrections, re-scores, and removal are free — no partnership or purchase required, and you do not need to justify the request. A removed company is recorded as unrated, never scored zero for having asked. Acknowledgement within one business day; removal within two.
info@apievangelist.com
·
Read the full data-sourcing policy →
On a security or compliance team? Put security in the subject line and
you will get a person, not a form — we will tell you exactly which public URLs this profile was built from.