ClaudeBot
ClaudeBot is Anthropic's training crawler — Anthropic is the company behind the Claude assistant. It goes through public websites and collects their content so future Claude models can learn from it. If you block ClaudeBot, Anthropic excludes your future content from its training data, and Claude will know less about your business.
- User-agent
ClaudeBot- Does it respect robots.txt?
- Yes
How to allow it in your robots.txt
User-agent: ClaudeBot
Allow: /How to block it (not recommended)
User-agent: ClaudeBot
Disallow: /Frequently asked questions
Should I block ClaudeBot?
If you want Claude to know about and recommend your business, no. Blocking it excludes your site from the data Anthropic's models are trained on, so Claude will know less about you and more about your competitors.
Does ClaudeBot respect robots.txt?
Yes. Anthropic states that all three of its bots (ClaudeBot, Claude-User and Claude-SearchBot) respect standard robots.txt rules, including Disallow and Crawl-delay directives.
How do I know if ClaudeBot visits my site?
Search for "ClaudeBot" in your server access logs or your CDN analytics. Every request is identified by that name in the user-agent.