Should You Block AI Crawlers? A Straight Answer for Businesses
Every few months a client forwards the same article: “AI is scraping your site — block it.” Then their traffic from ChatGPT, Perplexity and Google AI Overviews quietly goes to zero and nobody connects the two. Blocking AI crawlers is a real strategic choice, but for most businesses it is the fastest way to make yourself un-recommendable.
There are two different kinds of AI bot
Most of the confusion comes from treating all AI crawlers as one thing. They are not. Training crawlers collect content to train future models. Retrieval crawlers fetch a page right now because a user just asked a question and the assistant needs a source to cite. Blocking the second group is what removes you from answers.
| Bot | Owner | What it does | Recommended |
|---|---|---|---|
| GPTBot | OpenAI | Training crawl | Allow (optional) |
| OAI-SearchBot | OpenAI | Powers ChatGPT search results & citations | Always allow |
| ChatGPT-User | OpenAI | Live fetch when a user asks about your page | Always allow |
| PerplexityBot | Perplexity | Index + citation engine | Always allow |
| Google-Extended | Gemini training opt-out only — does not affect AI Overviews | Allow (or block safely) | |
| Googlebot | Search index and AI Overviews source | Always allow | |
| ClaudeBot / Claude-User | Anthropic | Training + live retrieval | Allow retrieval |
| Bytespider | ByteDance | Aggressive crawl, little referral value | Block if server load is a problem |
The real tradeoff
The honest version of the debate is this: allowing crawlers gives away content in exchange for being named as a source. If your business sells attention — a publisher whose revenue is ad impressions on the article itself — that trade can genuinely be bad. If your business sells a product or a service, being named is the whole point. A plumber, a clinic, a SaaS tool or a law firm loses nothing by having an assistant summarise their pricing page; they lose a customer by being absent when the assistant recommends three competitors.
So the question is not “block or allow.” It is “which bots earn me answers, and which just cost me bandwidth?”
What blocking actually costs you
- You disappear from citations. Assistants cite what they can fetch. A disallowed page cannot be quoted, linked or verified.
- Your competitors define you. If a review site or a directory is the only fetchable source about your category, their framing becomes the answer — including their outdated prices for you.
- You cannot be corrected. When a model gets a fact about your business wrong, the fix is a crawlable page stating the fact. Blocking removes your only repair mechanism.
- The block often does not do what you think. Blocking
Google-Extendeddoes not remove you from AI Overviews. BlockingGPTBotdoes not remove you from ChatGPT search. Teams routinely block the wrong bot and get the downside twice.
A sane default policy
For almost every commercial site, the right posture is open to retrieval, selective on training, hard on abuse. In practice:
- Allow every retrieval and search bot (OAI-SearchBot, ChatGPT-User, PerplexityBot, Googlebot, Claude-User).
- Decide training separately — blocking GPTBot or Google-Extended is a legitimate IP stance and costs you little visibility.
- Rate-limit or block only genuinely abusive crawlers that generate load without referrals.
- Never block your own money pages: pricing, services, locations, FAQ, about.
- Publish an llms.txt so the bots you welcome are pointed at the pages you want quoted.
- Back it with structured data so the fetch returns unambiguous facts, not guesses.
Check what you are already blocking
Plenty of sites are blocking AI crawlers without ever deciding to — a security plugin, a CDN bot-fight setting, or a WAF rule added it. Three things to verify this week: your robots.txt for AI user agents, your CDN or firewall bot rules (many now ship an “AI scrapers” toggle switched on by default), and whether your key pages render server-side rather than only in JavaScript. A crawler you technically allow but cannot render is still a crawler that cannot cite you.
If you are starting from scratch, work through the AEO checklist and the fundamentals in how to do AEO — crawler access is step one of a longer chain. And if you are new to the discipline entirely, start with what AEO is.
Bottom line
Blocking AI crawlers is a publisher strategy applied, usually by accident, to businesses that need to be found. Unless your product is the content on the page, let the retrieval bots in, choose your training stance deliberately, and spend your energy on being the clearest, most citable source in your category.
Not sure whether the assistants can even see your site — or what they say about you right now? Run the free AI Visibility Scan and see which engines name you, which name your competitors, and where your crawler access is quietly breaking things.