AI crawler access prompt

An AI crawler access prompt reads your robots.txt and explains, crawler by crawler, which AI tools can find your pages for answers and which can use them for training. You block what you mean to block and nothing else.

Stage
Check, step 34 of 40
Use it
Before launch, after changing host or CDN, and whenever a security plugin is installed.
You need
  • Your robots.txt (yoursite.com/robots.txt)
  • Your decision on AI training use
You get
A crawler-by-crawler table and a proposed robots.txt.

The prompt

Here is my robots.txt: [paste].
For each of these, say whether it is allowed, blocked or not mentioned,
and what that means for my site:
Googlebot, Google-Extended, OAI-SearchBot, GPTBot, ChatGPT-User,
Claude-SearchBot, ClaudeBot, PerplexityBot, Bingbot.
Separate search and answer access from training use.
Then propose a robots.txt that allows search and answers,
and leaves the training choice to me as a clear yes or no.
JSON version, for teams and automations

Fill each value under "inputs", keep the rest. It follows the prompt schema. When to use JSON.

{
  "$schema": "https://websitesdesigner.com/insights/prompts/prompt.schema.json",
  "id": "ai-crawl-check",
  "name": "AI crawler access prompt",
  "stage": "check",
  "step": 34,
  "of": 40,
  "version": "2026-11-19",
  "url": "https://websitesdesigner.com/insights/prompts/ai-crawl-check/",
  "inputs": {
    "paste": "[paste]"
  },
  "instructions": [
    "Here is my robots.txt: {{paste}}.",
    "For each of these, say whether it is allowed, blocked or not mentioned, and what that means for my site: Googlebot, Google-Extended, OAI-SearchBot, GPTBot, ChatGPT-User, Claude-SearchBot, ClaudeBot, PerplexityBot, Bingbot.",
    "Separate search and answer access from training use.",
    "Then propose a robots.txt that allows search and answers, and leaves the training choice to me as a clear yes or no."
  ],
  "output": "A crawler-by-crawler table and a proposed robots.txt.",
  "checks": [
    "Search crawlers and training crawlers are listed separately.",
    "The proposed file keeps the sitemap line.",
    "Nothing that feeds answers is blocked by accident."
  ]
}

What good output looks like

  • Search crawlers and training crawlers are listed separately.
  • The proposed file keeps the sitemap line.
  • Nothing that feeds answers is blocked by accident.

What experts miss

  • Blocking GPTBot doesn't remove you from ChatGPT search. OpenAI uses OAI-SearchBot for search, and says sites that opt out of it won't be shown in ChatGPT search answers.1
  • Google-Extended doesn't control AI Overviews. Google says the token doesn't affect inclusion or ranking in Google Search, and AI Overviews are part of Search.2
  • Check the host and CDN too. Firewalls and bot-protection settings can block AI crawlers before robots.txt is ever read.

Why it works

Each AI company runs separate crawlers for search, for user-requested fetches and for training. Blocking the wrong one can remove you from answers while leaving training untouched, or the reverse.

How to adapt it

  • Add your sitemap URL to the file if it's missing.
  • Ask for a separate rule set for staging sites, which should block everything.

Mistakes to avoid

  • Copying a robots.txt from another site.
  • Blocking everything with one wildcard rule to stop training.

Go deeper

How to get cited by ChatGPT explains crawl access in context. llms.txt is a separate, optional file. The AI shopping readiness checker tests what a crawler can read. This prompt is part of the check stage. It follows the AI website design prompts guide and the six steps in how to use AI to design a website.

Read the prompts guideAll 40 prompts

Sources

  1. "Overview of OpenAI Crawlers," OpenAI. developers.openai.com
  2. "Google-Extended Bot (Token) Does Not Affect Google Search," Search Engine Roundtable, quoting Google's crawler documentation. seroundtable.com

Related prompts

Schema markup prompt

A schema markup prompt asks the AI to write JSON-LD structured data for a page, using only facts visible on that page. It returns one linked graph and a list showing where each property comes from, so you can check it.

Get the prompt

Fact consistency prompt

A fact consistency prompt reads every page and lists each fact that appears in more than one place, such as name, address, numbers, services and dates, and flags any that disagree. Conflicting facts make AI tools hedge, or describe you wrongly.

Get the prompt

AI answer audit prompt

An AI answer audit prompt reads answers from ChatGPT, Perplexity and Google's AI to your buyers' questions and records whether you're named, how you're described, who else is named, and why. It shows what to fix rather than a rank.

Get the prompt

Page speed prompt

A page speed prompt asks the AI to list every script, font and image over a size limit, say what each does and whether the page needs it, and propose what to remove. AI builders tend to add more than a page needs.

Get the prompt

Want this done for you?

Unsure whether AI tools can read your site? Book a call and I'll check it with you.

Book a 20-minute call