AI crawler access prompt
An AI crawler access prompt reads your robots.txt and explains, crawler by crawler, which AI tools can find your pages for answers and which can use them for training. You block what you mean to block and nothing else.
- Stage
- Check, step 34 of 40
- Use it
- Before launch, after changing host or CDN, and whenever a security plugin is installed.
- You need
- Your robots.txt (yoursite.com/robots.txt)
- Your decision on AI training use
- You get
- A crawler-by-crawler table and a proposed robots.txt.
The prompt
Here is my robots.txt: [paste].
For each of these, say whether it is allowed, blocked or not mentioned,
and what that means for my site:
Googlebot, Google-Extended, OAI-SearchBot, GPTBot, ChatGPT-User,
Claude-SearchBot, ClaudeBot, PerplexityBot, Bingbot.
Separate search and answer access from training use.
Then propose a robots.txt that allows search and answers,
and leaves the training choice to me as a clear yes or no.JSON version, for teams and automations
Fill each value under "inputs", keep the rest. It follows the prompt schema. When to use JSON.
{
"$schema": "https://websitesdesigner.com/insights/prompts/prompt.schema.json",
"id": "ai-crawl-check",
"name": "AI crawler access prompt",
"stage": "check",
"step": 34,
"of": 40,
"version": "2026-11-19",
"url": "https://websitesdesigner.com/insights/prompts/ai-crawl-check/",
"inputs": {
"paste": "[paste]"
},
"instructions": [
"Here is my robots.txt: {{paste}}.",
"For each of these, say whether it is allowed, blocked or not mentioned, and what that means for my site: Googlebot, Google-Extended, OAI-SearchBot, GPTBot, ChatGPT-User, Claude-SearchBot, ClaudeBot, PerplexityBot, Bingbot.",
"Separate search and answer access from training use.",
"Then propose a robots.txt that allows search and answers, and leaves the training choice to me as a clear yes or no."
],
"output": "A crawler-by-crawler table and a proposed robots.txt.",
"checks": [
"Search crawlers and training crawlers are listed separately.",
"The proposed file keeps the sitemap line.",
"Nothing that feeds answers is blocked by accident."
]
}What good output looks like
- Search crawlers and training crawlers are listed separately.
- The proposed file keeps the sitemap line.
- Nothing that feeds answers is blocked by accident.
What experts miss
- Blocking GPTBot doesn't remove you from ChatGPT search. OpenAI uses OAI-SearchBot for search, and says sites that opt out of it won't be shown in ChatGPT search answers.1
- Google-Extended doesn't control AI Overviews. Google says the token doesn't affect inclusion or ranking in Google Search, and AI Overviews are part of Search.2
- Check the host and CDN too. Firewalls and bot-protection settings can block AI crawlers before robots.txt is ever read.
Why it works
Each AI company runs separate crawlers for search, for user-requested fetches and for training. Blocking the wrong one can remove you from answers while leaving training untouched, or the reverse.
How to adapt it
- Add your sitemap URL to the file if it's missing.
- Ask for a separate rule set for staging sites, which should block everything.
Mistakes to avoid
- Copying a robots.txt from another site.
- Blocking everything with one wildcard rule to stop training.
Go deeper
How to get cited by ChatGPT explains crawl access in context. llms.txt is a separate, optional file. The AI shopping readiness checker tests what a crawler can read. This prompt is part of the check stage. It follows the AI website design prompts guide and the six steps in how to use AI to design a website.