Before an AI assistant can describe your business, its crawler has to be allowed to read your site. A blanket rule or a security plugin can block them, and you may never know.
On this page
What robots.txt does
robots.txt is a plain-text file at the root of your site that tells automated crawlers which parts they may visit. Well-behaved crawlers follow it voluntarily. It works like a request, not a lock. Never use it to hide private data.
AI crawlers you may see
- GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI)
- ClaudeBot, Claude-SearchBot and Claude-User (Anthropic)
- PerplexityBot and Perplexity-User (Perplexity)
- Google-Extended (a robots.txt token that controls whether content Google crawls may be used for its Gemini models; it does not affect inclusion in Google Search)
- Applebot-Extended (Apple; lets publishers opt out of their content being used to train Apple's AI models)
Names and behavior change, so check each provider's current documentation before relying on this list.
A simple, permissive example
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
You can see a live example at swahaai.com/robots.txt.
Not every agent follows robots.txt
Crawlers that fetch a page because a user asked for it, such as ChatGPT-User and Perplexity-User, may not follow robots.txt the way search and training crawlers do, and providers document this differently. Check each provider's current documentation.
Anything private belongs behind a login.
Deciding who gets in
Allowing AI crawlers is a business decision. A local business that wants to be found usually benefits from being readable. A publisher whose content is the product may reasonably choose differently. Some crawlers fetch pages for training, others fetch live to answer a user's question, and providers offer different controls for each.
The risk to avoid is blocking by accident. Common causes are a blanket Disallow: / left over from a staging site, a security plugin with a bot-blocking default, or a CDN rule.
How do you check whether AI crawlers are blocked?
- Type your website address followed by /robots.txt into a browser. If you see the line "Disallow: /" with nothing after the slash, ask whoever manages your site: "Are we blocking AI crawlers on purpose?"
- Look for
Disallow: /underUser-agent: *or under a named AI crawler. - Check your hosting, CDN and security plugin settings for bot rules.
- Re-check after any site migration.
For the rest of the AEO checklist see what is AEO.
Common questions
Does allowing GPTBot guarantee I appear in ChatGPT?
No. It only permits reading. Appearing depends on many other factors.
Can I block training but allow live answers?
Some providers offer separate crawlers for each. Check their current documentation.
Where does the robots.txt file go?
At the root of your domain, for example yourdomain.com/robots.txt. Crawlers look for it at that exact address.
Want help putting this into practice? See our AEO, SEO and UGC service or request a free read.
This article is general information for small-business owners, not legal, financial, medical or professional advice. Rules and platform policies change; confirm anything that matters with a qualified professional.
Tell us what is slow, manual or missing
Free, plain-English conversation. We reply within one business day with a read and a scoped, written quote.
Get a free readKeep reading
- AEO
What Is llms.txt and Do You Need One?
- AEO
What Is AEO? Answer Engine Optimization Explained
- AEO
Schema Markup for Small Business: What to Add