What is an AI crawler? The bots that matter and what to allow

An AI crawler is a bot run by an AI company that visits web pages and copies their text. Some AI crawlers collect pages to train models, some build the search index behind AI answers, and some fetch a page when a person asks. For most SaaS sites, the sensible default is to allow the search and user bots, and to treat training bots as a separate business decision.

This guide covers the three jobs AI crawlers do, the 12 names worth knowing, and how to set robots.txt and your firewall for each.

I lead search, content and AI visibility at Fourward, and we sell AI visibility work, so I have a stake in this topic. Every bot description below comes from the operator's own documentation, checked on 10 November 2026.

What an AI crawler is

A crawler is a program that visits web pages automatically and saves what it finds. An AI crawler does that job for an AI company, such as OpenAI, Anthropic, Perplexity or Google.

How a crawler identifies itself

Each crawler sends a name with every visit, called a user agent. Your robots.txt file is a plain text file at the root of your site, and it tells each named crawler which pages it may read.

How much crawling is now for AI

Cloudflare is a CDN (content delivery network) that many websites route their traffic through, so it can measure bot traffic across a large part of the web. Cloudflare's July 2026 report says "52% of crawler requests are now for AI training as of June 2026, up from 22% in Spring 2025".

The same Cloudflare report says mixed-use crawlers, which blend search, agent use and training, "represent over 36% of activity". Mixed-use crawlers make the allow-or-block decision harder, because one bot can do two jobs.

The three jobs AI crawlers do

AI companies split their bots by job. The job matters more than the company name, because each job gives your site something different in return.

• Training crawlers copy pages to train AI models. OpenAI says "GPTBot is used to make our generative AI foundation models more useful and safe". Training feeds the model itself, not a list of sources it links to.
• Search crawlers build the index that AI answers search and cite. OpenAI says "OAI-SearchBot is used to surface websites in search results in ChatGPT's search features".
• User fetchers visit a page when a person asks the AI to read it. OpenAI says of ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply".

Search bots and user fetchers are the ones that can show a buyer a link to your site. Training bots shape what a model already knows about your product when it answers without searching.

Which AI crawlers matter for a SaaS site

These 12 names cover OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl and Meta. Each description comes from the operator's own documentation.

Two names that are not crawlers

Google-Extended and Applebot-Extended are names you write in robots.txt, not separate bots. The two names say whether pages collected by Google's or Apple's normal crawlers may be used for AI training.

Why CCBot matters

CCBot belongs to Common Crawl, which describes itself as "a non-profit foundation". Common Crawl publishes "an open repository of web crawl data that is universally accessible and analyzable by anyone". Blocking CCBot keeps your new pages out of that public archive.

The search bots matter most for buyers. Anthropic says blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results". Perplexity says "PerplexityBot is designed to surface and link websites in search results on Perplexity".

AI crawlers vs Googlebot

Google handles AI differently from OpenAI and Anthropic. Google's AI Overviews and AI Mode draw on the normal Google Search index, so there is no separate Google AI search bot to allow.

• AI Overviews follow Search. Google says "a page must be indexed and eligible to be shown in Google Search with a snippet" to appear in them. Google adds that there are "no additional requirements" for AI Overviews or AI Mode.
• Google-Extended only covers Gemini training. Google says "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search".
• Snippet controls limit what AI features show. Google points to "nosnippet, data-nosnippet, max-snippet, or noindex controls" to limit what Search shows from your pages.
• Apple separates search and training the same way. Apple says "Webpages that disallow Applebot-Extended can still be included in search results".

How SaaS teams decide what to allow or block

Start from what each page is for. A SaaS marketing site exists to be found and understood, so most of the site should be readable by AI search bots and user fetchers.

Three common setups

Questions to ask before you block a training bot

• Is the content your product? A data company may want to license its pages rather than give them away. A SaaS feature page is usually marketing, not the product.
• Do you want AI models to describe you well without searching? Training bots shape what a model knows about your product, your pricing model and your category.
• Is anything private sitting on a public page? Blocking a bot will not protect a private page. The internet standard for robots.txt says "These rules are not a form of access authorization". Put private pages behind a login.
• Who signs off? Marketing, legal and the content owner should agree on the setup and write it down, so nobody changes it by accident.

My default for a B2B SaaS site is the open setup, with anything private behind a login. Blocking training bots is a fair choice when your content is what you sell.

How to allow or block AI crawlers in robots.txt

Each robots.txt group names one or more user agents, then lists the paths they may or may not read. This example keeps AI search bots and user fetchers in, and opts out of training:

# AI search bots and user fetchers: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# AI training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /

For the open setup, delete the second group. Three rules apply either way:

• Robots.txt is a request. The Robots Exclusion Protocol, the internet standard for robots.txt, says its rules "are not a form of access authorization".
• User fetchers may skip it. OpenAI says robots.txt rules "may not apply" to ChatGPT-User. Perplexity says Perplexity-User "generally ignores robots.txt rules".
• Never block Googlebot or Bingbot to stop AI. Blocking Googlebot or Bingbot takes you out of normal Google or Bing search as well.

OpenAI's bots get a full walkthrough in our guide on how to get cited by ChatGPT.

Check your firewall and CDN settings too

A firewall or CDN can block a bot before the bot ever reads robots.txt. Your robots.txt can say yes while your firewall says no, and the file will not show it.

• Cloudflare sorts AI bots by job. Cloudflare's AI crawler settings let you "set actions for Search, Agent, and Training traffic".
• New Cloudflare domains allow AI bots by default. Cloudflare's documentation, updated 2 November 2026, says "For new domains, Search, Agents, and Training use Allow (do not block) by default".
• A training block does not stop mixed-use bots. Cloudflare says "Mixed-purpose crawlers used for both Search and Training are excluded from this edge block".
• Old settings can linger. A bot rule someone switched on in the past can still block a bot you now want to allow. Compare your CDN's bot settings with your robots.txt, line by line.

Common mistakes with AI crawlers

• Copying a block-all template. A template that lists every AI bot can block OAI-SearchBot along with GPTBot. Blocking OAI-SearchBot removes you from ChatGPT search answers, which is rarely what a SaaS team wants.
• Blocking Google-Extended to escape AI Overviews. Google-Extended controls Gemini training use, not Search. Snippet controls such as nosnippet are the tool Google offers for AI Overviews.
• Treating robots.txt as security. Any public page can be fetched by a bot that ignores robots.txt. Customer-only docs and admin pages belong behind a login.
• Fixing robots.txt but not the firewall. Robots.txt and the firewall both have to allow a bot, or the bot stays out.
• Thinking llms.txt controls access. An llms.txt file is a plain text summary some sites publish for AI tools. An llms.txt file does not allow or block any bot, as our llms.txt examples show.
• Setting the rules once and forgetting them. AI companies add bot names as they launch products, so review your list every quarter.

How to check which AI crawlers can reach your site

1. Read your robots.txt. Open yoursite.com/robots.txt and look for each name in the table above.
2. Test robots.txt and the firewall together. Fourward's free AI crawler checker tests 20 AI crawlers against your robots.txt and your firewall.
3. Read your server or CDN logs. Search the logs for the user agent names above to see which bots actually visit.

Crawler access is the first step in our GEO guide for B2B SaaS. GEO (generative engine optimization) means making your site easy for AI engines to find, read and cite.

FAQs

Sources

1. Overview of OpenAI Crawlers, OpenAI, checked 10 November 2026.
2. Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic Claude Help Center, checked 10 November 2026.
3. Perplexity Crawlers, Perplexity, checked 10 November 2026.
4. List of Google's common crawlers, Google, checked 10 November 2026.
5. AI features and your website, Google Search Central, checked 10 November 2026.
6. About Applebot, Apple Support, published 4 September 2026, checked 10 November 2026.
7. CCBot, Common Crawl, checked 10 November 2026.
8. Meta Web Crawlers, Meta, checked 10 November 2026.
9. Robots Exclusion Protocol, Internet Engineering Task Force, September 2022, checked 10 November 2026.
10. Content Independence Day, one year on: building the business model for the agentic Internet, Cloudflare, 1 July 2026, checked 10 November 2026.
11. AI Crawler and Agent access policies, Cloudflare, updated 2 November 2026, checked 10 November 2026.
12. Fourward AI crawler checker, Fourward.