Bytespider
Bytespider is ByteDance's crawler — ByteDance is TikTok's parent company — and it collects data to train its AI models like Doubao. It's a special case: ByteDance doesn't publish official documentation for the bot, and there are multiple documented reports of it ignoring robots.txt rules, with very aggressive crawl volumes. If it's causing you load problems, the only reliable way to stop it is blocking at the server or CDN level.
- User-agent
Bytespider- Does it respect robots.txt?
- No — it ignores the rules
- Official documentation
- https://crawlercheck.com/directory/scrapers/bytespider
How to allow it in your robots.txt
User-agent: Bytespider
Allow: /How to block it (not recommended)
User-agent: Bytespider
Disallow: /Frequently asked questions
Should I block Bytespider?
It depends. If you want visibility in ByteDance's AI assistants, allow it. But if its aggressive crawling overloads your server, it's one of the few AI bots where a block (at the server level, not just robots.txt) can be justified.
Does Bytespider respect robots.txt?
Not reliably. There are documented, repeated reports of Bytespider crawling pages explicitly disallowed in robots.txt. If you need to stop it for real, use firewall rules on your server or CDN (e.g. Cloudflare).
How do I know if Bytespider visits my site?
Search for "Bytespider" in your server logs. It's known for generating very high request volumes, so if your site suddenly slows down, it's worth checking.