Are you blocking AI without knowing it?
Check in seconds which AI crawlers can read your site and which ones you're blocking in your robots.txt.
Takes 5 seconds · No sign-up · We don't store your site
How it works
Enter your site
Type your web address. No sign-up, no email required.
We read your robots.txt
We fetch your site's robots.txt and compare it against the AI crawlers we track.
See what you block
We show which AIs can read your site, which ones you block, and what each case means for your business.
The basics
What is an AI crawler?
An AI crawler (or bot) is a program from companies like OpenAI, Google or Anthropic that roams the internet and saves what it finds to feed their AIs. If it can read your site, that AI knows your business; if you block it, you stop existing for it.
Each AI uses several bots with different jobs: some learn from your content to train, others index it to show it in their search, and others visit live when a user asks about you. They're all controlled from a single file on your site, the robots.txt.
Why it matters
AI answers already decide who finds you
of Google searches already show an AI summary, rising to 57.9% on searches phrased as a question.
Ahrefs · Search Engine Journal, 2025is the share of clicks to websites when an AI summary appears, versus 15% when it doesn't.
Pew Research Center, 2025people use ChatGPT every week. If you block it, none of them know you exist.
OpenAI · TechCrunch, 2026The reference
The main AI crawlers
Who each one is, what it's for and whether it respects your robots.txt.
| Bot | Company | Type | Respects robots.txt? |
|---|---|---|---|
| Googlebot | Mixed | Yes | |
| GPTBot | OpenAI | Training | Yes |
| OAI-SearchBot | OpenAI | Search | Yes |
| ChatGPT-User | OpenAI | User action | No |
| Google-Extended | Training | Yes | |
| ClaudeBot | Anthropic | Training | Yes |
| PerplexityBot | Perplexity | Search | Partial |
| Bingbot | Microsoft | Mixed | Yes |
| Applebot | Apple | Search | Yes |
| Meta-ExternalAgent | Meta | Training | Yes |
| CCBot | Common Crawl | Training | Yes |
| Bytespider | ByteDance | Training | No |
How to fix it
How to unblock AI bots in your robots.txt
The robots.txt is a text file at the root of your site (yourdomain.com/robots.txt) that tells bots where they can go. Unblocking the AIs takes three steps:
- 1
Open your robots.txt
Type your domain followed by /robots.txt in the browser. If nothing shows up, you have no file and you're blocking no one.
- 2
Look for Disallow rules
Check for "Disallow: /" lines under "User-agent: *" or under the name of an AI bot. Those are the ones shutting the door.
- 3
Let the AIs in
Remove the "Disallow: /" that affects them or add a rule that lets them in. Save the file and check it again above.
Example robots.txt that lets the main AI bots in:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /Heads-up: some bots that visit when a user asks about you (like ChatGPT-User or Perplexity-User) don't always obey robots.txt. For those, what counts is not blocking them at the server level.
Frequently asked questions
What is an AI bot or crawler?
It's a program from companies like OpenAI, Google or Anthropic that crawls the web to feed their AIs. If it can read your site, that AI is more likely to know and recommend your business.
Where are these bots blocked?
In a file called robots.txt, at the root of your site (yourdomain.com/robots.txt). A wrong rule there can block AIs without you knowing.
Should I unblock all of them?
For most businesses that want customers, yes: the more AIs can read your site, the more chances to appear in their answers. The exception is very aggressive bots that overload your server.
Do you store my site or email?
No. This tool is single-use: we read your robots.txt on the spot and show you the result. No email, nothing stored.
How do I monitor this continuously?
Surfeo tracks your AI visibility and alerts you to changes. This tool is a snapshot; Surfeo is the ongoing monitoring.
Does blocking AI bots improve my Google ranking?
No. Blocking them gives you no SEO points; it only makes AIs know less about your business. Your Google ranking depends on Googlebot, which is a separate bot.
What's the difference between GPTBot and ChatGPT-User?
GPTBot collects content to train OpenAI's models; ChatGPT-User visits your site live when a ChatGPT user asks a question that needs your page. One learns from you in general, the other visits for a specific query.
What if I block a bot but it keeps coming?
Some bots that act on a user's request, like ChatGPT-User or Perplexity-User, warn that they may skip robots.txt. And others, like Bytespider, are known to ignore it. To really stop them you have to block them at the server or your CDN (for example, Cloudflare).
How often should I check this?
Every time you touch your robots.txt or change web platform. A well-meant change can block AIs by accident. Surfeo watches it for you continuously.
More free tools
Last updated: June 2026
Do you actually show up in ChatGPT, Gemini and Perplexity?
Unblocking the bots is step one. Surfeo measures whether AIs mention you and tells you how to improve.
Try Surfeo free