100% Free Tool
Bot Crawl Directives & Syntax Validator
Free Robots.txt Tester
Test your site’s crawler directives in real time. Ensure Googlebot and AI crawlers can access your important pages while blocking private directories.
Status for
Matched Line :
Common Search Engine & AI Bot Directives
Essential User-Agent rules for blocking or allowing search crawlers and generative AI bots.
| Bot Name | User-Agent String | Impact & Type | Recommended Access |
|---|---|---|---|
| Googlebot | Googlebot | Google Search Indexing | Allow All |
| Bingbot | Bingbot | Bing Search Indexing | Allow All |
| GPTBot | GPTBot | OpenAI Model Training & Search | Optional |
| PerplexityBot | PerplexityBot | AI Search Engine Citations | Allow for GEO |
🚫
Avoid Disallowing CSS/JS
Modern Googlebot needs full access to rendering scripts and stylesheets to evaluate layout and mobile responsiveness.
📍
Specify Sitemap URLs
Always append absolute URLs to your XML sitemap at the bottom of the file to accelerate search discovery.
⚠️
Robots.txt vs Noindex
Disallowing a page in robots.txt only stops crawling, not indexing. Use meta noindex tags to completely de-index URLs.
Automated Crawl Error Detection
Start Free Audit →
Detect Technical SEO Bugs Daily
ZevSEO crawls your website automatically, alerting you instantly if search bots encounter 404 dead links or accidental disallows.