A crawlability checker fetches a URL the way a crawler would and reports whether named bots are allowed to read it — robots.txt, headers, and often a fetch or rendering probe. For AI, that means GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and related tokens, not only Googlebot. A pass means the bot was not obviously locked out. It does not mean ChatGPT or Perplexity will cite you. Use a checker to find access gaps; fix robots and HTML next; measure citations separately.
What does a crawlability checker test?
It answers 'can this bot fetch this URL?' Our free crawlability checker tests robots.txt, opt-out signals, rendering, and discovery files against a set of AI crawlers and returns pass or fail per bot with fixes. That is the access layer. It is not an extractability score and not a citation count.
Classic SEO crawl tools often stop at Googlebot and Bingbot. A site can look healthy there and still disallow GPTBot or serve an empty shell to clients that do not run JavaScript. If you only 'check crawlability' in an SEO suite, you can miss the AI gap entirely.
Google's robots introduction is still the access-control file: it tells compliant crawlers which paths they may request. A checker that never opens /robots.txt is guessing.
- Job: per-bot access — allowed, blocked, or fetch failed.
- AI-specific: named user-agent tokens, not only Googlebot.
- A pass is eligibility to be read, not a promise to be quoted.
- JavaScript-only bodies can fail even when robots.txt looks open.
How do you check crawlability by hand if you do not use a tool?
Open https://yourdomain.com/robots.txt. Confirm retrieval tokens are allowed with exact casing: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended. OpenAI documents those crawlers separately — allowing one does not imply the others.
Fetch the URL with JavaScript disabled (or view the HTML source). If the answer lives only after a client render, many AI crawlers will see a stub. Check CDNs and WAFs for managed AI blocks that override a permissive file.
Then run the checker to catch the cases that are tedious by hand: per-bot mismatches, headers, and edge rules. Pair it with the robots.txt for AI crawlers guide when you need the allow/disallow split.
- 1Open /robots.txt and confirm retrieval bots are not disallowed (exact token casing).
- 2Confirm the page's answer exists in static or server-rendered HTML.
- 3Check CDN or WAF AI-block features that can override robots.txt.
- 4Re-run a crawlability checker after each change; do not assume a file save is live at the edge.
What should you do with a fail?
A fail is an access bug, not a content-quality score. Fix the rule or the render, then re-check the same URL. If GPTBot is blocked on purpose, say so — that is a training-access choice; do not confuse it with OAI-SearchBot, which OpenAI uses for ChatGPT search.
Once access is green, structure still decides whether a model can lift an answer. That is a different page: content structure for AI extraction. Do not expect the checker to grade your headings.
If you need site-wide technical scores or a citation board after access is healthy, that is a broader product. This checker stays a free access probe.
Frequently asked questions
Is a crawlability checker the same as a Google crawl test?+
No. URL Inspection in Search Console is about Google Search. An AI crawlability checker names AI user-agents. You can pass one and fail the other.
If the checker passes, will I show up in ChatGPT?+
No. A pass means a named bot could fetch. Citation still depends on the answer on the page, accuracy, and authority.
How often should I re-check?+
After any robots, CDN, or template change, and when you ship a new template. Access regressions are usually config, not copy.