Googlebot
Googlebot is Google's classic crawler, but today it does double duty: it feeds both traditional search results and the AI-generated summaries (AI Overviews) that appear at the top. For a small business it's still the single most important bot: without Googlebot you don't exist in Google or in its AI answers.
- User-agent
GooglebotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36- Does it respect robots.txt?
- Yes
- Official documentation
- https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
How to allow it in your robots.txt
User-agent: Googlebot
Allow: /How to block it (not recommended)
User-agent: Googlebot
Disallow: /Frequently asked questions
Should I block Googlebot?
Absolutely not. Blocking Googlebot removes you from Google Search entirely: classic results, Google Images, Discover and AI Overviews. It's the fastest way to become invisible online.
Does Googlebot feed Google's AI Overviews?
Yes. AI Overviews are built on top of the search index Googlebot creates. If you want to opt out of AI summaries only while staying in Google, Googlebot isn't the lever to pull — there's no robots.txt control that does that without also losing search.
How do I know if Googlebot visits my site?
It shows up as "Googlebot" in your server logs, and you can see its crawl stats for free in Google Search Console, under Settings > Crawl stats.