ai visibility
Reads your live robots.txt and tells you, per engine, whether ChatGPT, Claude and Perplexity can actually reach your pages — and separates the settings that cost you citations from the ones that don’t.
What actually matters in this field
01Being allowed is not the same as being readableAI crawlers don’t run JavaScript. If your content builds in the browser, they see an empty page.
This is the one that catches most sites. A server-log study by Vercel and MERJ covering more than 500 million crawler fetches found that none of the major AI crawlers execute JavaScript — they fetch the raw HTML, read what is in it, and move on. GPTBot downloads JavaScript files in roughly one request in ten and never runs them. So a React, Vue or Angular site that builds its content in the browser can rank perfectly well on Google, whose crawler does render JavaScript, while being effectively blank to ChatGPT, Claude and Perplexity. There is no partial credit here: the content is either in the HTML your server sends or it is invisible.
02Test it the right way, in thirty secondsUse View Source, not the Elements panel — Elements shows the page after JavaScript has run.
Open the page and use View Source — not the Elements panel in developer tools. Elements shows you the page after JavaScript has run, which is not what a crawler receives. View Source shows the raw response, which is exactly what it receives. Search that source for a sentence of your main content. If it is there, you are readable. If it only appears in Elements, you are not. Turning JavaScript off in your browser and reloading gets you to the same answer.
03Blocking a training crawler does not remove you from the modelA disallow rule only looks forward. It doesn’t remove you from a model already trained on you.
A disallow rule is forward-looking. Content already ingested stays in the model that was trained on it; blocking GPTBot today affects future training runs, not what has already happened. This matters because people often add the rule expecting it to undo something. If your concern is what is already out there, robots.txt is the wrong instrument entirely.
04robots.txt is a request, not a lockThe big operators honour it; some crawlers ignore it. Real blocking happens at the firewall.
The major operators — OpenAI, Anthropic, Google, Perplexity — publish their crawler names and honour the file, and they have reputational reasons to keep doing so. Others have a documented history of ignoring it; ByteDance’s Bytespider is the usual example. If you genuinely need to keep a crawler out rather than politely ask, that has to be enforced at your firewall or CDN edge. Treat robots.txt as policy, not as security.
05Know which Google token you are touchingGoogle-Extended opts out of Gemini training. Googlebot is search itself — blocking it removes you from Google.
Google-Extended is an opt-out token for Gemini training. It has no crawler behind it and blocking it does not affect your position in ordinary Google Search. Googlebot is the search crawler itself. Disallowing Googlebot removes you from Google entirely — a far more expensive mistake than any AI setting, and one we see made by people who meant to do the first thing. Applebot-Extended works the same way for Apple.
06Keep redirects shortAI crawlers give up on redirect chains sooner than Googlebot, and they do it silently.
Search crawlers tolerate long redirect chains; the specification allows up to twenty hops and Googlebot follows around ten. AI retrieval crawlers have been observed giving up considerably sooner, and they abandon quietly with no error for you to find. If a page is worth citing, it should be one hop from its canonical URL at most.
07Do not lean on llms.txtIt isn’t an access-control mechanism. Harmless to publish, unwise to rely on.
The proposed llms.txt convention is often described as a way to control or guide AI access. It is not an access-control mechanism, and as of now the crawler operators have either said nothing about it or declined to support it. Publishing one is harmless. Relying on it instead of robots.txt and readable HTML is not.
About this check
·Why this check exists now
People increasingly ask an AI instead of searching — and the answer they get is assembled from pages the AI was able to fetch. If its crawler can’t reach your site, you are not in the running, however good your content is. That access is governed by one file most sites haven’t revisited in years: robots.txt.
·The part almost everyone gets wrong
There are two completely different kinds of AI crawler. Training crawlers — GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended — collect pages to train models. Blocking them is a legitimate rights decision and it does not affect whether you get cited. Retrieval crawlers — OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User — build the answer indexes and fetch your page live when someone asks a question. Block those and you cannot be cited at all.
·The common accident
Many sites blocked “the AI bots” in a single sweep and removed themselves from AI answers while believing they had only opted out of training. Others still carry a blanket Disallow left over from a staging site. This check tells you which of the two you have actually done, per engine, and exactly which line to change.
·What this check does not claim
We cannot tell you how often ChatGPT mentions you, and neither can anyone else — those systems are non-deterministic, answers vary by person and phrasing, and any tool selling a “prominence score” is showing you a number it invented. We check the part that has a verifiable answer: whether the door is open.
Free ai visibility check — the whole tool, no signup
Check free whether AI answer engines can reach your site. Reads your live robots.txt and shows which of ChatGPT, Claude and Perplexity’s crawlers are allowed — and which are blocked. Runs live in your browser; nothing is stored. Part of checkmysite.pro — a free suite of website checks with no account and no paid tier.