Check my robots.txt for AI crawlers

See exactly which AI crawlers your robots.txt allows, blocks, or leaves unmentioned — with the matching rule quoted. Runs entirely in your browser. Nothing is uploaded anywhere.

1. Get your robots.txt

Site URL (we'll try to fetch /robots.txt directly) Fetch robots.txt

Direct fetch usually fails: browsers block cross-origin requests unless the site sends CORS headers, and almost no site does for /robots.txt. If it fails, paste the file below.

2. Or paste it

robots.txt contents Check crawlers Load an example

What the states mean

• allowed — a rule explicitly permits this crawler (or nothing in the file blocks it).
• blocked — a rule disallows it. The exact matching line is quoted.
• unmentioned — the file never names this crawler. Crawlers default to allowed when unblocked, so this effectively means allowed — but you haven't made an explicit choice.

Matching is simplified RFC 9309: most-specific user-agent group wins, * is the fallback, most-specific rule wins, Allow beats Disallow on ties. Evaluated for the site root (/).

Going deeper

robots.txt is one of four checks that decide whether AI engines can use your pages. The AI Search Readiness Audit ($59) also checks raw-HTML coverage, AI Overviews eligibility directives, and dead schema — in one click, 100% client-side.

FAQ

Does blocking GPTBot stop my content from appearing in ChatGPT?

Not necessarily. GPTBot trains models; ChatGPT-User fetches pages for live user queries, and other crawlers may still index the page. Blocking training crawlers and blocking answer-time crawlers are different decisions with different tokens.

What does "unmentioned" mean?

The file has no group naming that crawler. Crawlers default to allowed when nothing blocks them, so unmentioned effectively means allowed — but you haven't made an explicit choice about that crawler.

Why can't the checker fetch my robots.txt directly?

Browsers block cross-origin fetches unless the site sends CORS headers, and almost no site does for /robots.txt. Paste the file instead — it never leaves your browser.

Is robots.txt enough to control AI crawler access?

It's a voluntary signal that reputable crawlers honor, not enforcement. And it's only one gate: raw-HTML coverage, robots directives, and schema all matter too.