VelenPublicWebCrawler
VelenPublicWebCrawler is the web crawler built by Velen.io to feed Hunter.io's data infrastructure, an email prospecting and business data platform. Its stated goal is to build business datasets and train machine learning models to better understand the web. It only accesses publicly available pages, respects robots.txt directives, and waits at least two seconds between requests to the same domain. If your website is included in its data, your business has a better chance of being recognised by the AI systems Hunter.io powers.
- User-agent
VelenPublicWebCrawlerMozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io)- Does it respect robots.txt?
- Yes
- Official documentation
- https://velen.io/
How to allow it in your robots.txt
User-agent: VelenPublicWebCrawler
Allow: /How to block it (not recommended)
User-agent: VelenPublicWebCrawler
Disallow: /Frequently asked questions
Should I block VelenPublicWebCrawler?
If you want your business to be known by AI systems and platforms that rely on Hunter.io data, blocking it works against that goal. The crawler only accesses public content, follows your robots.txt rules, and does not put any meaningful load on your server.
How does this crawler affect my business visibility in AI?
Velen collects public content to train machine learning models. The clearer and more complete your website content is, the more likely those models are to learn who you are and what you offer. It does not guarantee direct mentions, but it does increase the chances of your business being part of those training datasets.
How can I tell if VelenPublicWebCrawler is visiting my site?
Check your server access logs or your CDN dashboard, for example in Cloudflare. Search for VelenPublicWebCrawler in the user-agent field. Every visit is recorded with that identifier.