AI Crawlers and robots.txt: What Your Business Should Allow

November 3, 2026 · Lizzard Solutions

AI Crawlersrobots.txtAI SearchAEOTechnical SEOSmall Business

You want ChatGPT, Perplexity or Gemini to mention your business when someone asks for a provider like you. But if your robots.txt or firewall quietly blocks the bots those tools send, your pages never make it into their answers. This guide shows you how AI crawlers and robots.txt work together, and exactly what a small business should allow or block.

You'll get a plain-language map of the main AI bots, a copy-ready robots.txt template, and a five-minute way to check what your own site tells them today.

---

What Are AI Crawlers and How Does robots.txt Control Them?

AI crawlers are automated bots that AI companies send to read web pages, either to build a search index for AI answers, to fetch a page when a user asks about it, or to collect training data. robots.txt is a plain text file at the root of your site that tells each bot, by its user-agent name, which paths it may visit.

The robots.txt format is an internet standard, RFC 9309. Each group starts with a User-agent line followed by Allow or Disallow rules. A bot follows the most specific group that matches its name, so a group for GPTBot overrides the general User-agent: * group for that bot.

Action for today: open yourdomain.com/robots.txt in a browser and note which bots it names.

---

The Main AI Bots: Search, User Requests and Training

The big AI companies now document separate user agents for separate jobs. This table summarises what their own documentation says:

Sources: OpenAI's crawler documentation, Anthropic's crawler help article and Google's list of common crawlers. OpenAI states that its settings are independent, so you can allow OAI-SearchBot to appear in search results while disallowing GPTBot. Perplexity's own bot page says the same for its two agents.

Google is a special case. Google-Extended has no user-agent string of its own, and Google says it doesn't affect inclusion or ranking in Google Search. Blocking Googlebot, on the other hand, removes you from Google Search altogether, so never do that by accident.

Action for today: compare the bots in your robots.txt with this table and mark any search or user bot that is blocked.

---

What Should a Small Business Allow?

Start from your goal. If you sell services and want AI assistants to recommend you, the search and user bots are the ones that matter, because they decide whether your pages can be found and quoted in answers.

Take a small accounting firm in Novi Sad. Its service pages, prices and FAQ are marketing material, so it has every reason to let OAI-SearchBot, PerplexityBot and Claude-SearchBot read them. Its client upload portal should be disallowed for every bot, and also protected by a login, because robots.txt alone hides nothing.

Action for today: write one sentence that states your policy: which bots you allow, which you block, and why.

---

A Copy-Ready robots.txt Template for AI Crawlers

This template allows AI search and user bots on public pages, keeps everyone out of private paths, and leaves one clear place to decide about training bots. Replace the example paths with your own:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /portal/

# AI search and user-requested visits
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
Disallow: /admin/
Disallow: /portal/

# Training: change Allow to Disallow if you opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /
Disallow: /admin/
Disallow: /portal/

Sitemap: https://www.example.com/sitemap.xml

Two details trip people up. First, a bot that matches a named group ignores the * group, so repeat your private-path rules in every group. Second, several User-agent lines in a row share the rules below them, which keeps the file short.

For a real example, our own robots.txt at lizzardsolutions.com names GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and Google-Extended and allows them, because we want AI tools to read our service pages.

Action for today: adapt the template, publish it, and reload the file in a browser to confirm the live version changed.

---

Do You Need llms.txt Too?

llms.txt is a proposed file, described at llmstxt.org, that gives AI agents a short Markdown summary of your site with links to the most useful pages. Unlike robots.txt, it doesn't allow or block anything. It only helps a bot understand what you do faster.

It is still a proposal, not an official standard, and AI companies don't promise to use it for ranking or citations. It's cheap to add, though: one file with your company description, services, location and key links. Keep it consistent with your About page and directory profiles, so every source says the same thing about you.

Action for today: draft five lines that describe your business, services, city and main contact page, which is the core of an llms.txt.

---

Check What Your Site Tells AI Crawlers in 5 Minutes

robots.txt is only one layer. Run this short check to make sure nothing else is blocking AI bots:

1. Read your robots.txt. Look for Disallow: / under * or under any AI bot you want to allow.
2. Check your firewall or CDN. Bot protection in hosting panels and CDNs can block AI crawlers even when robots.txt allows them. Look for AI bot or crawler settings.
3. Check page-level tags. A noindex robots meta tag or header keeps pages out of search indexes, including those AI search builds on.
4. Make content readable without JavaScript. If key text only appears after scripts run, some crawlers may see an empty page.
5. Confirm your sitemap. Link it at the bottom of robots.txt, so crawlers find every public page.

These checks overlap with classic technical SEO. Our SEO audit guide for small businesses covers indexing, sitemaps and page tags in more depth, and our AI visibility check guide explains the content side: clear answers, structured data and consistent company facts.

If you're curious how crawling looks from the other side, our guide to web scraping for business explains why responsible bots respect robots.txt and rate limits.

Action for today: run the five checks above and write down anything that blocks a bot you want to allow.

---

AI Crawlers and robots.txt: FAQ

Should I block GPTBot on my website?

Only if you don't want your content used for OpenAI model training. Blocking GPTBot doesn't stop OAI-SearchBot, which OpenAI uses for ChatGPT search results. Most service businesses that want AI visibility allow both, while publishers selling content often block GPTBot and keep OAI-SearchBot allowed.

Does blocking Google-Extended remove my site from Google Search?

No. Google says Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal. It only controls whether your content may be used for Gemini training and grounding. Never block Googlebot itself unless you want to disappear from Google Search.

Do AI crawlers have to obey robots.txt?

robots.txt is a voluntary standard. The major AI companies document their bots and say they respect it, but it isn't access control. Protect private pages with a login, and use firewall or CDN rules if you need to stop bots that ignore the file.

How long until robots.txt changes take effect for AI search?

It varies by company. OpenAI notes it can take about 24 hours for its systems to reflect a robots.txt change for search results, and Perplexity gives up to 24 hours. Other crawlers re-read the file on their own schedule, so allow a few days before judging results.

---

Getting AI crawlers and robots.txt right is one of the cheapest steps toward being cited by ChatGPT, Perplexity and Gemini: allow the search and user bots on public pages, decide training access on purpose, and check that no firewall overrides your choice. If you'd rather have someone look at it for you, our free AI visibility check reviews crawler access, llms.txt, structured data and content.