GEO & AI search

Is Your Website Blocking ChatGPT? robots.txt for AI Crawlers on B2B Sites

Before content, check access. Many export websites block AI search crawlers by accident through robots.txt, a CDN or a security plugin. Here's how to tell search bots from training bots and what to allow.

ToucHigh Research TeamPublished 6 min readEnglish
A friendly AI chat bubble with violet eyes lowers an orange bucket into a clean stone well surrounded by a lab flask, a ruler, a product bottle and a blank clipboard, while a dusty heap of dark chain links lies ignored nearby.
“
The question behind this article

Our IT team blocked "AI bots" on the website last year. Could that be why ChatGPT never shows our factory, and what should we allow?

Short answerQuite possibly. AI companies run separate bots for search and for model training. OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, while GPTBot only governs training. Allow the search crawlers (OAI-SearchBot, PerplexityBot, Googlebot, Bingbot), decide separately about training bots such as GPTBot and Google-Extended, and check that your CDN or firewall isn't blocking them regardless of robots.txt.

Key takeaways

  • OpenAI's OAI-SearchBot surfaces sites in ChatGPT search; GPTBot is for training. The settings are independent, so you can allow search and refuse training.
  • OpenAI says robots.txt changes can take about 24 hours to affect ChatGPT search.
  • PerplexityBot surfaces and links sites in Perplexity's results and, according to Perplexity, isn't used to train foundation models.
  • Google-Extended controls use of your content for Gemini training and grounding; Google says it does not affect inclusion or ranking in Google Search.
  • Firewalls and CDNs can block crawlers even when robots.txt allows them; Perplexity publishes IP ranges and WAF instructions for this reason.

A surprising number of export websites are invisible to AI search for a dull reason: the crawlers can't get in. Sometime in the last two years someone added a "block AI bots" rule, switched on a CDN's bot protection or installed a security plugin. The site still works for people, so nobody notices. Then the team spends months rewriting content that AI search engines were never allowed to read.

This guide is a 30-minute technical check for factory and B2B websites. It won't make you visible on its own, but if access is broken, nothing else you do will count.

Which AI crawlers are for search, and which are for training?

The key idea: AI companies separate the bot that finds pages to show in answers from the bot that collects content for model training. You can make different choices for each.

CompanyUser agentPurpose (per the company's documentation)
OpenAIOAI-SearchBotSurfaces websites in ChatGPT's search features. Sites opted out are not shown in ChatGPT search answers, though they can still appear as navigational links.
OpenAIGPTBotCrawls content that may be used to train OpenAI's foundation models. Disallowing it signals that content shouldn't be used for training.
OpenAIChatGPT-UserVisits pages for certain user actions. OpenAI says robots.txt rules may not apply and it isn't used to decide what appears in search.
PerplexityPerplexityBotSurfaces and links websites in Perplexity's search results; not used to crawl content for AI foundation models.
PerplexityPerplexity-UserFetches pages when a user asks a question; generally ignores robots.txt because a user requested the fetch.
GoogleGoogle-ExtendedA robots.txt token that controls whether content Google crawls may be used for Gemini training and grounding. It does not affect Google Search inclusion or ranking.

Google's own AI features in Search work differently. Google's guidance says AI is built into Search, so robots.txt rules for Googlebot are the control for how a site is crawled for Search, including AI Overviews and AI Mode. Blocking Googlebot to "opt out of AI" would also remove you from ordinary search results.

What should a factory website's robots.txt say?

For most export manufacturers the goal is to be found, so allow the search crawlers. Whether to allow training crawlers is a business decision; refusing them doesn't remove you from AI search. A simple starting point looks like this:

robots.txt lineEffect
User-agent: OAI-SearchBot / Allow: /Eligible for ChatGPT search answers
User-agent: PerplexityBot / Allow: /Eligible for Perplexity results
User-agent: Googlebot / Allow: /Google Search, including AI Overviews and AI Mode
User-agent: Bingbot / Allow: /Bing search
User-agent: GPTBot / Allow or DisallowYour choice about OpenAI model training
User-agent: Google-Extended / Allow or DisallowYour choice about Gemini training and grounding
Sitemap: https://www.example.com/sitemap.xmlHelps crawlers find every page

Keep private areas, such as a dealer portal or price lists, out of crawlers' way with login protection, not just robots.txt. Google's robots.txt documentation is explicit that robots.txt is not a mechanism for keeping a page out of Google; use noindex or password protection for that.

Why can a firewall block AI crawlers even when robots.txt allows them?

robots.txt is a request; your CDN, firewall or security plugin is a gate. Many bot-protection features challenge or block unfamiliar crawlers before they ever read robots.txt. Google lists "ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" among its basics for AI features. Perplexity goes further and publishes IP address lists and step-by-step web application firewall rules so site owners can allow its bots. OpenAI also publishes IP ranges for its crawlers.

Ask whoever manages your hosting three questions: is bot protection on, does it have an allow list, and are the search crawlers above on it? Then check your server logs for the user agents in the table. If they appear and get errors, or never appear at all, you have found the problem.

How do you check access in 30 minutes?

  1. Open yoursite.com/robots.txt. Look for Disallow: / under User-agent: * or under any AI user agent.
  2. Check the CDN or firewall for bot protection, country blocking or "AI scraper" toggles.
  3. Search your server logs for OAI-SearchBot, PerplexityBot and Googlebot, and the status codes they receive.
  4. Confirm indexing in Google Search Console for your key product and capability pages.
  5. Retest in ChatGPT after a day. OpenAI says robots.txt changes can take about 24 hours to reflect in search.
  6. Write the policy down, so the next security update doesn't undo it.

Does allowing crawlers mean AI will recommend you?

No. Access makes you eligible; it doesn't make you chosen. Once crawlers can read your site, the next questions are whether your pages answer buyers' questions with specific facts, and whether your company information is consistent across the web. But access comes first. It's the cheapest fix in GEO and the most often missed.

GEO Autopilot · Social Autopilot · Ads

How ToucHigh helps

ToucHigh's GEO Autopilot starts every engagement with this kind of check:

  • an audit of where your site is invisible across ChatGPT, Perplexity, Google AI and Gemini;
  • fixes written and shipped for you, including pages, answers and entity data;
  • AI-citation share tracked afterwards, so you can see the effect of opening the gate.

Book a 30-minute strategy call and we'll look at your site's AI visibility live.

Factories and B2B suppliers: see how ToucHigh handles supplier visibility in AI search. Need the visibility to turn into pipeline? See ToucHigh PRO, our strategy-first outbound service.

Frequently asked questions

If we block GPTBot, will ChatGPT stop showing our site?

Not according to OpenAI. GPTBot governs whether content may be used for model training; OAI-SearchBot governs appearance in ChatGPT search. The two settings are independent, so you can disallow GPTBot and still allow OAI-SearchBot.

Does blocking Google-Extended remove us from AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews and AI Mode are part of Search, controlled through Googlebot. To limit what appears in snippets and AI features, Google points to nosnippet, data-nosnippet, max-snippet or noindex.

Do we need an llms.txt file to be found by AI?

Google says you don't need new machine-readable files or AI text files to appear in its AI features. Some sites publish llms.txt as an extra summary for AI tools, but it is not a substitute for crawlable pages.

How long after fixing robots.txt will we see a change?

OpenAI says its systems can take about 24 hours to adjust to a robots.txt update for search. Being cited still depends on your content, so measure over the following weeks rather than the next day.

Sources

  1. Overview of OpenAI Crawlers — OpenAI
  2. Perplexity Crawlers — Perplexity
  3. Google's common crawlers (Google-Extended) — Google Search Central
  4. Introduction to robots.txt — Google Search Central
  5. AI features and your website — Google Search Central

Why we wrote thisWe wrote this because the first thing we check on a new factory website is whether AI crawlers can get in, and the answer is "no" more often than anyone expects.

ToucHigh

See where you're invisible in US AI search

Book a free 30-minute call and we'll run a live AI-visibility read on your store.

Book a 30-min strategy call →Log in to ToucHigh