AI bots & crawlers

Collects public content to train OpenAI's AI models, such as the ones behind ChatGPT.

Visits your site live when a ChatGPT user asks for information that requires checking your page.

Indexes websites so they can appear as results and links in ChatGPT search.

Collects public content to train and improve Anthropic's Claude models.

Accesses your site live when a Claude user asks a question that requires checking it.

Indexes web content to improve the quality and relevance of Claude's search results.

Crawls the web for Google Search, including AI Overviews.

A robots.txt control token that decides whether your content can be used to train Google's Gemini models.

A general-purpose crawler Google's product teams use to download public content for various purposes, including AI research and development.

Indexes websites to show and link them in Perplexity search results.

Visits your site live when a user asks a question on Perplexity that requires checking your page.

Crawls the web for the Bing search engine and also feeds Microsoft Copilot's answers.

Crawls the web to train Meta's AI models (Llama) and to index content for its products.

Downloads specific links requested by users and assists Meta AI's agentic capabilities.

Crawls the web for Apple's search features: Siri, Spotlight and Safari suggestions.

A control token that decides whether Apple can use your content to train Apple Intelligence models.

Crawls the web to improve Amazon products and services, such as Alexa's answers, and can be used to train its AI models.

Collects web content at scale to train ByteDance's AI models — the company behind TikTok and Doubao.

Builds an open archive of the web that serves as training data for many AI models.

Collects public web content to train Cohere's language models, aimed at businesses.

Visits your site when a user of Vibe, Mistral's assistant, asks a question that requires checking it.

Collects content for DuckAssist, DuckDuckGo's AI-generated answers.

Discovers and indexes web pages so You.com's search engine and AI assistants can give up-to-date answers.

Extracts structured data from web pages to build a knowledge base used by businesses and AI systems.

Collects web documents to build open datasets used to train and evaluate Ai2's language models.

Retrieves web content for Grok, xAI's AI assistant built into X (formerly Twitter).

Crawls the web for Petal Search, Huawei's search engine, and its ecosystem services.

Collects public images and their context for Hive's web intelligence and AI products.

Downloads the content of a URL when a NotebookLM user — on Google's AI-powered note-taking app — manually adds it as a source in their project.

Powers Google's Read Aloud feature: when a user asks to have a web page read to them, this bot fetches the text to convert it into audio.

Indexes your website content to power AI agents and enterprise search applications built on Google Cloud Vertex AI.

Fetches your page on demand when you actively use the URL Inspection tool or the Rich Results Test in Google Search Console.

Browses the web on behalf of real users so Google's AI agent (Project Mariner) can carry out specific tasks, such as finding information or taking actions on web pages.

Crawled web pages to build the Bing (and formerly MSN Search) index, helping sites appear in Microsoft's search results.

Crawls public web content to improve search relevance across Amazon products, including Alexa.

Generates the link preview when someone shares your website on Facebook, Instagram, or Messenger.

Downloads public web content to build training datasets for Meta's voice recognition technology.

Crawls external pages to review Meta ads and generate the link previews displayed on Facebook, Instagram, and WhatsApp.

Crawls public web pages to index them in Baidu, China's largest search engine.

Crawls public web content to train Moonshot AI's foundation models, which power the Kimi assistant.

Fetches your web content on demand when a Kimi user asks for real-time information or to summarise an article.

Analyzes web pages for relevance to build the search index that powers Kimi, Moonshot AI's AI assistant.

Crawls public web pages to index content that powers the search engine inside Mistral's Vibe platform.

Validates that landing pages submitted for ChatGPT Ads comply with OpenAI's advertising policies and are relevant to the ad.

Validates which publicly available web content can legally be used to train AI models, labeling it for Webz.io's data pipeline clients.

Crawls news sites, forums and user-generated content to power Webz.io's data API, sold to media monitoring platforms and AI training companies alike.

Crawls public websites to build business datasets and train the machine learning models that power Hunter.io.

Collects public web data on behalf of business clients that need price monitoring, competitive intelligence, or commercial databases.

Crawls public web content to power Semrush's Content Toolkit, an AI-assisted content marketing and SEO suite.

Crawls pages already indexed by Yandex to use as real-time sources for YandexGPT, Yandex's AI-powered search feature.

Crawls public websites to build the index for Timpi's decentralised search engine, with that index potentially also used to train AI models.

Crawls public websites to power Andi's generative search results, which answer questions with AI-generated summaries instead of a traditional list of links.

Crawls public web pages to build the index powering iAsk, an AI-based answer search engine.

Crawls websites on demand to power Phind's AI-based search engine, which is aimed at developers and technical users.

Downloads images from public websites at scale to build datasets for training AI vision and image-generation models.

Downloads public web content to train PanGu, Huawei's multimodal AI model.

Collects publicly available web content in Japanese to build training datasets for AI models.

Crawls public websites to collect brand and keyword mentions on behalf of Awario's monitoring customers.