Back to LLM Ready
Free Tool

robots.txt AI Configurator

Control which AI crawlers can access your content. Generate a robots.txt that welcomes the right bots and blocks the rest.

Choose a Preset

Fine-tune Individual Crawlers

8 Allowed
0 Blocked

GPTBot

OpenAI - OpenAI ChatGPT web browsing

Allowed

ChatGPT-User

OpenAI - ChatGPT user-initiated browsing

Allowed

Claude-Web

Anthropic - Claude web browsing

Allowed

ClaudeBot

Anthropic - Claude training and indexing

Allowed

PerplexityBot

Perplexity - Perplexity AI search

Allowed

Google-Extended

Google - Google Gemini and AI features

Allowed

cohere-ai

Cohere - Cohere AI models

Allowed

anthropic-ai

Anthropic - Anthropic AI indexing

Allowed

Additional Settings

Don't have one? Create your llms.txt

Preview

robots.txt
# robots.txt
# AI Crawler Configuration
# Generated by ReplyRaptor - https://replyraptor.com/llm-ready/robots-generator

# Allow traditional search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /

# AI Crawler Rules
# Allowed AI Crawlers
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: anthropic-ai
Allow: /

# Important URLs

# ---
# Generated by ReplyRaptor
# Create your own: https://replyraptor.com/llm-ready/robots-generator
# Check your AI discoverability: https://replyraptor.com/llm-ready/checker

Next Steps

  1. 1. Download or copy your robots.txt file
  2. 2. Replace the existing robots.txt at your site root
  3. 3. Verify your settings with our checker

Complete your AI discoverability

robots.txt is just the start. Create an llms.txt file to give AI assistants accurate information about your business.

Create llms.txtCheck Your Site