guides
Robots.txt for AI Crawlers: Search, Training and Bot Access
How should robots.txt handle AI crawlers without accidentally blocking search discovery?
Read guide →Free online crawler-control tool
Robots.txt Generator
Create a robots.txt file, validate common mistakes as you edit, then copy or download it. The generator runs in your browser and does not require an account.
Choose your editing mode
Simple mode is safer for quick setup. Advanced mode exposes every crawler group and rule.
Crawler purpose policy
Start neutral, then opt in to AI crawler restrictions only when they match your policy.
Neutral default: keep search, AI search and AI-control crawlers open unless your path rules explicitly block them.
Paths to block
Start with a suggested path or add your own. Only block paths that you intentionally want compliant crawlers to avoid.
Suggested paths
Add custom path
Add any site path. Bare paths get a leading slash; full HTTP(S) URLs are reduced to their path and query.
Search
Allowed by default
AI Search
Allowed by default
AI Control / Training
Allowed by default
Before you publish
- 1. Generate
Review the live output. - 2. Validate
Check syntax and compatibility. - 3. Test URLs
Verify important paths and crawlers. - 4. Publish
Upload it as /robots.txt at your site root.
Generated robots.txt
Live output; generation and validation stay in your browser.
User-agent: * 0 errors0 warnings0 notes
No syntax or compatibility warnings detected.
Generate safely
Build User-agent, Allow, Disallow, and Sitemap directives without hand-formatting every line.
Catch risky rules
Get warnings for whole-site blocking, invalid sitemap URLs, non-standard directives, and common indexing misunderstandings.
Test before publishing
Move from generation to validation, live checking, and URL-rule testing without leaving the site.
How to create a robots.txt file
- 1
Choose a starting preset
Use a standard, WordPress, Blogger, or staging preset, then review every rule for your own site.
- 2
Add crawler rules
Enter one Allow or Disallow path per line and specify the main User-agent group the rules apply to.
- 3
Add crawler-specific exceptions
Advanced controls can create separate groups for crawlers that should behave differently from the main rules.
- 4
Add your sitemap
Use an absolute HTTP(S) sitemap URL such as https://example.com/sitemap.xml.
- 5
Review validation warnings
A valid-looking file can still contain a dangerous rule. Fix errors and review warnings before publishing.
- 6
Publish at the site root
The standard location is /robots.txt, for example https://example.com/robots.txt.
robots.txt does not make a page private
robots.txt controls crawler access. It is not an authentication system, and blocking a URL from crawling is not the same thing as guaranteeing that the URL cannot appear in search results.
Control crawlingrobots.txt
Request no indexingnoindex
Protect private contentauthentication / authorization
Search crawlers and AI-related crawler controls are not the same
The Advanced section keeps separate crawler tokens separate. For example, a provider may document one crawler for search discovery and another token for model-development or AI-product controls. Choose a crawler-specific rule only when you intentionally want it to differ from the main User-agent rules.
Search crawling
Traditional search crawlers such as Googlebot and Bingbot discover and crawl content for search products.
AI search discovery
Examples include OAI-SearchBot, Claude-SearchBot, and PerplexityBot. Provider documentation should be checked before changing access.
AI product controls
Tokens such as GPTBot, ClaudeBot, and Google-Extended have provider-specific purposes that should not be treated as identical.
Robots.txt presets for common platforms
Platform presets are editable starting points, not universal best practices. Review the paths your own site actually uses before publishing them.
Related robots.txt tools
Use a separate page when your task changes from creating a file to validating, checking, or testing it.
Knowledge base
Robots.txt Guides
Practical guides for crawler rules, indexing controls, host scope, parameter URLs, deployment behavior, and robots.txt troubleshooting.
Examples
Copy practical robots.txt examples for allowing all crawlers, blocking paths, adding sitemaps, wildcards, and crawler-specific rules.
→Robots.txt Guides
Practical robots.txt guides covering crawler policy, noindex, sitemap directives, subdomain scope, query parameters, caching, file limits, and rule behavior.
→Platform Robots.txt Guides
Platform-specific robots.txt guides for Shopify, Next.js, Wix, Squarespace, Webflow, and Magento / Adobe Commerce, with configuration and verification workflows.
→Related guides
guides
Robots.txt vs Noindex: Crawling and Indexing Explained
Can robots.txt prevent a page from being indexed, and when should noindex be used instead?
Read guide →guides
Sitemap in Robots.txt: How we Add and Verify the Directive
Where should we add a Sitemap directive in robots.txt, and how do we verify multiple sitemap references?
Read guide →guides
Robots.txt Wildcards and Rule Precedence: *, $ and Longest Match
How do robots.txt wildcards and conflicting Allow or Disallow rules actually match?
Read guide →guides
Robots.txt for Subdomains: How we Check Scope Before Changing Rules
Does the root-domain robots.txt apply to subdomains, HTTP, HTTPS, or another port?
Read guide →platforms
Next.js Robots.txt: How we Configure App Router Crawling Safely
Should we use app/robots.txt or app/robots.ts in Next.js, and how do we keep previews from blocking production?
Read guide →Technical references
Rules and crawler-specific guidance are checked against primary standards and provider documentation rather than copied from other SEO tools.
- RFC 9309
Robots Exclusion Protocol syntax, matching, grouping, and percent-encoding behavior.
- Google robots.txt specification
Google-specific robots.txt behavior and supported fields.
- OpenAI crawler controls
OAI-SearchBot and GPTBot crawler purposes and controls.
- Anthropic crawler controls
ClaudeBot, Claude-SearchBot, and Claude-User crawler behavior.
- Perplexity robots.txt guidance
PerplexityBot indexing behavior and robots.txt compliance.
- Google-Extended documentation
Google-Extended Gemini training and grounding controls.
Common questions
Where should robots.txt be located?
At the root of the host it controls, normally /robots.txt.
Can robots.txt remove a URL from Google?
It primarily controls crawling. If your goal is to prevent indexing, use the appropriate indexing control rather than relying on Disallow alone.
Does Google support Crawl-delay?
Googlebot does not support the Crawl-delay directive, so this validator warns when it appears.
Should I block every AI-related crawler?
Not automatically. Different crawler tokens can serve search, user-requested retrieval, model development, or other provider-specific purposes. Decide by crawler and verify the provider's current documentation.
Is this generator free?
Yes. Core generation and validation happen in the browser and do not require an account.