How to let AI crawlers reach your site (robots.txt)

AI Search Score checks whether your robots.txt allows the major AI crawler user-agents. If you’re blocking them, you can’t be cited — no matter how good your content is.

Why this matters

AI engines send their own crawlers to read your pages. If robots.txt disallows them, you’ve shut the door on AI citation entirely. Many sites block AI bots by accident — via a security plugin default or a copied robots.txt — without realising they’ve opted out of generative search.

How to fix it

Check your robots.txt (at yourdomain.com/robots.txt) and make sure it doesn’t disallow the major AI user-agents unless you intend to. The notable ones in 2026 include:

• GPTBot (OpenAI)
• ClaudeBot (Anthropic)
• PerplexityBot (Perplexity)
• Google-Extended (Google’s AI training/Gemini)

If you want to be cited, ensure none of these are disallowed. A simple permissive robots.txt allows all crawlers by default. If you’ve a Disallow targeting these agents and you want citations, remove it.

(Deciding whether to allow AI crawlers is a genuine business choice — some sites opt out deliberately. This check assumes you want AI visibility; if you don’t, it’s fine to leave it.)

How to check it worked

Re-scan. “AI crawlers allowed” should turn green. Also open yourdomain.com/robots.txt directly and confirm the AI user-agents aren’t disallowed.

Prefer this reviewed for you? See our services.