YouBot
YouBot is the crawler for You.com, a search engine with built-in AI assistants. It discovers and indexes pages to offer real-time results to its users, citing sources in its answers. It's a smaller player than Google or Bing, but allowing it costs nothing and adds another path through which potential customers can reach your site.
- User-agent
YouBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36- Does it respect robots.txt?
- Yes
- Official documentation
- https://you.com/docs/youbot
How to allow it in your robots.txt
User-agent: YouBot
Allow: /How to block it (not recommended)
User-agent: YouBot
Disallow: /Frequently asked questions
Should I block YouBot?
There's no reason to. Its crawling is respectful, and allowing it gives you presence in another AI search engine that cites sources. The more assistants that can read your site, the more chances of appearing in their answers.
Does YouBot respect robots.txt?
Yes, fully. Its official documentation confirms it respects robots.txt directives and crawl-speed preferences, including crawl-delay, with the user-agent "YouBot".
How do I know if YouBot visits my site?
Search for "YouBot" in your server logs. Its user-agent includes a link to You.com's official documentation that identifies it.