Can ChatGPT, Claude and Perplexity Actually Crawl Your Site? A 10-Minute Check
A growing share of product discovery now happens inside an assistant: someone asks ChatGPT for "best EU-compliant invoicing tools for Shopify," or asks Perplexity to compare two SaaS vendors, and the answer cites three sites. If your pages can't be fetched by those assistants, you're not in the conversation — no ranking penalty, no error in Search Console, just silence.
The trap is that most blocks are accidental. A WAF default, a hosting platform's "bot fight" toggle, a geo-rule left over from a DDoS week, a staging password that leaked into production DNS. None of these were decisions anyone made about AI crawlers.
Below is the manual version of a check you can run in about ten minutes with a browser, a terminal, and read access to your CDN dashboard. It's written for cross-border, SaaS and store teams who run multi-locale sites behind a CDN — the exact setup where silent blocks happen most.

图示:来源页预览图(仅作编辑上下文;版权归属原发布方) · 来源引用:dev.to
Step 1: Know which bots you're actually dealing with (2 min)
AI vendors run multiple user agents with different jobs. Blocking one does not block the others, and this is where most teams misread their own config.
| Vendor | Training / index crawler | User-initiated fetch | Notes |
|---|---|---|---|
| OpenAI | GPTBot (training), OAI-SearchBot (ChatGPT search index) |
ChatGPT-User |
Three separate controls |
| Anthropic | ClaudeBot |
Claude-User, Claude-SearchBot |
Older anthropic-ai token may still appear |
| Perplexity | PerplexityBot (index) |
Perplexity-User |
|
Google-Extended (a control token, not a crawler) |
Googlebot |
Gemini grounding rides on normal Googlebot | |
| Common Crawl | CCBot |
— | Feeds many downstream models |
The practical implication: you can allow retrieval while blocking training. If legal wants "don't train on our docs" but marketing wants "quote us in ChatGPT," the answer is usually Disallow: GPTBot + Allow: OAI-SearchBot and ChatGPT-User. Decide this once, in writing, before you touch config — otherwise every engineer who touches robots.txt will guess differently.
Step 2: Audit robots.txt for the exact tokens (2 min)
curl -s https://www.yourdomain.com/robots.txt \
| grep -iE "gptbot|oai-searchbot|chatgpt-user|claudebot|anthropic|claude-user|perplexity|ccbot|google-extended|bytespider|amazonbot|meta-external"
Three things to look for:
Disallow: /under an AI user agent block. Legitimate if intentional; a disaster if copy-pasted from a 2023 template.- Wildcard traps.
User-agent: *+Disallow: /searchis harmless.User-agent: *+Disallow: /because someone wanted to hide a staging path is not. - Per-subdomain and per-locale files. Your
de.,fr.orshop.hostname almost certainly has its own robots.txt. Teams fix the primary domain and forget the regional storefronts — which is precisely where cross-border revenue lives.
Check the file over HTTPS on the exact hostname the assistant would fetch. And confirm it returns 200, not a 301 to a marketing page or a 404 behind the CDN.
Step 3: Check headers, not just robots.txt (1 min)
Robots directives also live in HTTP headers, which override nothing but can silently suppress indexing and AI summarization.
curl -sI https://www.yourdomain.com/pricing | grep -iE "x-robots-tag|noindex|vary"
curl -sI https://www.yourdomain.com/pricing | grep -iE "cf-mitigated|server"
A cf-mitigated: challenge header means the request never reached your origin — the CDN served a JS challenge instead of your page. That's the single most common accidental block I see on SaaS sites.
Step 4: Actually fetch your key URLs as each bot (2 min)
Don't trust documentation; test it. Start with three URLs that matter commercially: homepage, a pricing page, and your best-converting blog post.
for ua in "GPTBot" "OAI-SearchBot" "ChatGPT-User" \
"ClaudeBot" "Claude-User" "Claude-SearchBot" \
"PerplexityBot" "Perplexity-User" "CCBot"; do
code=$(curl -s -o /dev/null -w "%{http_code}" \
-A "$ua/1.0 (+https://yourdomain.com/bot)" \
https://www.yourdomain.com/pricing)
echo "$ua -> $code"
done
How to read the result:
200— good, but verify the body too:curl -s -A "GPTBot/1.0" https://… | wc -cand grep forJust a moment,Enable JavaScript, orAccess denied. A 200 challenge page is still a block.403— WAF or bot-management rule. Go to Step 5.429— rate limiting; assistants crawl slowly, so this usually means an aggressive global rule.503— "Under Attack Mode," maintenance page, or origin overload.301/302to a login or consent wall — the crawler sees a redirect chain, not your content.
Important caveat on this test: a 200 from curl only proves your user-agent string passes. Serious WAFs verify the source IP against the vendor's published ranges, so a spoofed UA can get through while the genuine bot is blocked. Treat curl as a fast negative test — a failure is real, a pass is provisional. Confirm in your CDN's firewall analytics by filtering on the UA string over the last 7 days.
Step 5: Interrogate your CDN and platform defaults (2 min)
This is where the accidental blocks hide. Check each layer your traffic passes through:
- Cloudflare: Bot Fight Mode, "Block AI Scrapers and Crawlers" one-click toggle, Super Bot Fight Mode's "definitely automated" action, Managed WAF rules, rate-limiting rules scoped to
*, and any country-block rules. Cross-border teams often block regions after fraud spikes — several AI crawler egress ranges resolve to a handful of countries, so a "block RU/CN" rule from 2022 can quietly catch infrastructure. - Vercel / Netlify: Deployment Protection and password protection applied to the production alias rather than just preview URLs.
- Shopify: bot-protection apps, password-protected storefront during launches, and app-injected
noindexon collection pages. - WordPress / Magento: security plugins that ship a UA blocklist including
GPTBotandCCBotby default. - Origin:
.htaccessor nginxdenyrules, plus any IP allowlist left from a migration.
For each, ask: does this rule match on user-agent, IP reputation, or ASN? UA rules are the ones that break AI crawling without breaking humans.
Step 6: Prove the content is reachable without JavaScript (1 min)
If your pricing, docs or product pages render client-side, most AI fetchers see an empty shell. Run:
curl -s -A "OAI-SearchBot/1.0" https://www.yourdomain.com/pricing | wc -w
A word count in the low hundreds for a page your team spent weeks on is the tell. Server-side render, or publish a static, crawlable variant for the pages you most want quoted.
Step 7: Verify it end-to-end (1 min)
Config passing is not the same as being cited. Do two things:
- Ask the assistants directly. Query ChatGPT, Claude and Perplexity with a phrase that should surface you ("alternatives to X for German VAT"). Check whether you're cited and whether the cited URL resolves.
- Check referrers. In GA4 or your analytics, filter sessions by referrer containing
chatgpt.com,perplexity.ai,claude.ai,copilot.microsoft.com,gemini.google.com. Referrer data is imperfect, so treat it as directional traffic evidence, not a scoreboard.
Step 8: Lock it in (1 min)
Add a weekly synthetic check that fetches your three key URLs with each AI user agent and alerts on any non-200. Re-run the full audit whenever you touch WAF rules, launch a new locale, or migrate CDNs — those are the three events that break this silently.
The 10-minute checklist
| # | Check | Pass looks like | Owner |
|---|---|---|---|
| 1 | Written policy: training vs retrieval | Decision documented | Marketing + Legal |
| 2 | robots.txt on every hostname | Explicit AI UA blocks, no Disallow: / |
SEO / Dev |
| 3 | X-Robots-Tag headers |
No stray noindex, no cf-mitigated: challenge |
Dev |
| 4 | curl fetch per UA on 3 URLs | 200 + real body content |
Dev |
| 5 | CDN / WAF / platform defaults | AI bots not caught by bot-fight or geo rules | Infra |
| 6 | JS-free content | Meaningful word count in raw HTML | Frontend |
| 7 | Live citation + referrer check | Cited in ≥1 assistant; referrers appear | Marketing |
| 8 | Weekly synthetic monitor | Alerting on non-200 | Infra |

图示:HEXOPS 原创配图(受选题启发,非复制原文图片) · 来源引用:dev.to
The pattern worth internalizing
Almost nobody blocks AI crawlers on purpose. They get blocked by a default that was enabled to stop a DDoS in Q3, a password on a staging alias, a regional firewall rule, or a robots.txt line copied from a template three years ago. The access you lose is invisible — no error, no dashboard warning — and it compounds quietly while competitors get quoted.
The ten-minute version above catches roughly the same failures a paid scanner would, because the failures are structural, not exotic. Run it quarterly, and always after a CDN migration or a new locale launch.
Sources
- Original post: Can ChatGPT, Claude and Perplexity actually crawl your site? A 10-minute check
- OpenAI crawler documentation (GPTBot, OAI-SearchBot, ChatGPT-User): https://platform.openai.com/docs/bots
- Anthropic crawler documentation: https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Perplexity crawler documentation: https://docs.perplexity.ai/guides/bots
- Google
Google-Extendeddocumentation: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers - Cloudflare bot management and AI crawler controls: https://developers.cloudflare.com/bots/
- BotReady — free AI crawler access checks: https://botready.app/



