SemrushBot-OCOB
SemrushBot-OCOB is Semrush's dedicated crawler for its Content Toolkit, a product that uses AI to help businesses and agencies create and optimise content. It visits websites to analyse their content and benchmark it against competitors. It doesn't index pages for a public search engine or train its own language models, but the data it collects is used by marketing professionals worldwide to understand what content performs well in each sector. If someone uses Semrush to research your market, your website could appear as a reference point in that analysis.
- User-agent
SemrushBot-OCOBMozilla/5.0 (compatible; SemrushBot-OCOB/1; +https://www.semrush.com/bot/)- Does it respect robots.txt?
- Yes
- Official documentation
- https://www.semrush.com/bot/
How to allow it in your robots.txt
User-agent: SemrushBot-OCOB
Allow: /How to block it (not recommended)
User-agent: SemrushBot-OCOB
Disallow: /Frequently asked questions
Should I block SemrushBot-OCOB?
It depends on your goals. It doesn't feed ChatGPT or index for Google, so blocking it won't directly hurt your AI visibility. That said, if you want your content to serve as a benchmark when agencies or marketing professionals analyse your sector with Semrush, allowing it can be worthwhile. If you'd rather competitors not use your site as a reference in their analyses, you can block it without major consequences.
How does it affect my visibility in AI tools?
Indirectly. SemrushBot-OCOB doesn't feed models like ChatGPT or Gemini, so it won't directly improve or reduce how often those AI tools mention your business. However, the data it collects is used by marketing professionals to decide what content to create. If your site appears as a strong reference in your sector, it can influence the content strategies others develop using your market as a benchmark.
How do I know if SemrushBot-OCOB is visiting my site?
Look for "SemrushBot-OCOB" in your server access logs or your CDN dashboard, such as Cloudflare. Every visit is recorded with that name in the user-agent field. Keep in mind that Semrush doesn't use fixed IP ranges, so filtering by IP address is not a reliable way to identify it.