Free GEO tool
Robots.txt Generator for AI
Create a robots.txt file that allows AI models like ChatGPT, Claude, and Perplexity to crawl your website.
About this tool
robots.txt tells crawlers what they can index. By default, some AI crawlers may be blocked or unsure. This generator creates a robots.txt that explicitly allows GPTBot, Claude-Web, PerplexityBot, Google-Extended, and other major AI crawlers — so they can index and recommend your pages.
How to use
- Enter your domain (e.g. www.yourcompany.com)
- Copy the generated robots.txt content
- Save as robots.txt and upload to your site root
- Your file should be reachable at https://yoursite.com/robots.txt
Generated robots.txt
# robots.txt for www.example.com # Allows AI crawlers (ChatGPT, Claude, Perplexity, etc.) to understand your site User-agent: * Allow: / User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-Web Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Anthropic-AI Allow: / User-agent: cohere-ai Allow: /
Host at https://www.example.com/robots.txt
Pair with LLMs.txt
Robots.txt tells crawlers they can index; LLMs.txt tells them who you are. For best results, add both. Use our LLMs.txt Generator — then track visibility with GeoArk AI.