Are AI Crawlers Visiting My Site, And Why Does It Matter?

Which Crawlers Are There?

OpenAI’s GPTBot and OAI-SearchBot, Anthropic’s ClaudeBot and Perplexity’s PerplexityBot are crawlers these companies name in their own documentation. Some collect content to train models; others read current pages at the moment a user asks a question.

Google does not send a separate crawler for AI. It reads your site with its own search crawler and defines a separate permission name for AI use. So you will not see a separate “Google AI bot” visitor in the records.

Why It Matters

Assistants have to read pages to get current information. If your site is closed to these crawlers, the assistant cannot take current information about you from your site. It is left with either old information or what others have written about you.

This matters most for details that change often: price range, opening hours, new products, shipping terms. An assistant getting these wrong can be worse than it not mentioning them at all.

Another effect is citation. An assistant that can read your page in live search can link to you in its answer. It will not link to a page it cannot read; the answer points to another site instead.

The Door Can Close Without You Noticing

Some infrastructure and security services can block AI crawlers by default. Cloudflare, for example, announced in mid-2025 that it would start blocking these crawlers by default for newly added sites. A setting like that can take effect without the site owner knowing.

The result: your site looks open to visitors, everything works normally, but it is closed to the assistants. The only way to notice is to look.

How Can You Tell They Come?

Your server records every visit under the name the visitor gives itself. Looking for the crawlers’ names in these records shows how often they come and which pages they read. The records are usually reached through your hosting provider’s panel or your technical team.

When reading the records, two cases need to be told apart: a crawler that never comes, and one that comes and is turned away at the door. Both keep you out of the answers, but for different reasons.

Training Crawlers And Search Crawlers

AI crawlers serve two purposes. Some collect content to train models; that content becomes part of the model’s general knowledge, and its effect shows over time. Others read the relevant pages at the moment a user asks a question.

The effects of the two are felt at different times. A site closed to the search crawler disappears from current answers at once; the effect of a site closed to the training crawler shows up in later model versions.

What To Watch For In The Records

Not every visit carrying a crawler’s name really belongs to that crawler; some automated software introduces itself under the name of a known crawler. The companies publish information in their documentation for verifying where their own crawlers come from.

So when reading the records, look at the pattern as much as the numbers: when visits peak, which pages are read again and again, and which are never visited.

Do You Have To Be Open To All Of Them?

Which crawler to allow is a business decision. Some businesses do not want their content used in model training but do want to appear in current answers. It is possible to separate the two.

What matters is that the decision is made knowingly. Letting a default setting decide for you can make you invisible without meaning to.

Before deciding, seeing which crawlers come and which pages they read helps you understand what is being given up.

Which Pages Do They Read?

The pages the crawlers read most are the pages the assistants can rely on when they describe you. Often the home page and a few product or service pages come out on top; pages with important information may never be visited.

Following this spread every month shows which part of your site is visible to the assistants. You can see what this section looks like in our sample report.

The number of visits is neither good nor bad on its own. A crawler that reads a few of the right pages can be more useful than one that reads many irrelevant ones. The real question is whether the pages that describe your business are being read.