Should You Block AI Crawlers? robots.txt and llms.txt Explained

Every website owner now faces a question that did not exist a few years ago: should AI companies be allowed to crawl your content? Some publishers block every AI bot on principle; others actively court them for the visibility. Both positions are defensible – what matters is making the choice deliberately rather than by default. Here is what the controls actually do, and how to think about the trade-off for a UK small business, charity or club website.

Who is crawling you, and why

There are three distinct kinds of AI-related crawler, and they deserve different decisions:

• Training crawlers (OpenAI’s GPTBot, Google-Extended, Meta’s crawler, CCBot): collect content to train future models. Blocking these keeps your content out of training data but does not remove you from AI answers today.
• Live retrieval / search bots (OAI-SearchBot and ChatGPT-User for ChatGPT search, PerplexityBot, Bingbot, Googlebot): fetch pages at answer time so the assistant can cite them. Blocking these makes you invisible in AI search answers – including recommendations you might want.
• Ordinary search crawlers (Googlebot, Bingbot) now also feed AI features like AI Overviews. Blocking them means leaving search entirely, which almost no business wants.

How to block (or allow) them: robots.txt

robots.txt sits at your site’s root and names crawlers by user-agent. For example, to block training bots while staying visible in AI search:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

Points worth knowing: robots.txt is a request, not a lock – reputable companies honour it, bad actors ignore it (server-level blocking is possible for persistent offenders); Google-Extended controls AI training use without affecting your normal Google search presence; and the list of user-agents changes as new assistants launch, so review it a couple of times a year.

What is llms.txt?

llms.txt is a newer, informal convention: a markdown file at your site root that gives AI systems a curated summary of your site – what it is, and links to your most important pages in a machine-friendly format. Where robots.txt says “you may/may not crawl”, llms.txt says “if you are going to read my site, start here and get the facts right”.

Adoption by AI companies is still uneven, so treat it as a low-cost bet rather than a guarantee: it takes ten minutes to write, cannot hurt, and if assistants use it you have shaped how they describe you – accurate business details, correct prices, the pages you would want cited. For a small business the same information should already exist on a clear About page and in your structured data; llms.txt just serves it in one tidy file.

So – block or allow? A decision framework

• You sell products or services and want customers: allow retrieval/search bots at minimum. AI recommendations are a growing referral source, and our companion guide to getting recommended by ChatGPT and AI search covers making the most of it. Training bots are your call – blocking them costs little visibility today.
• Your content IS your product (paid tutorials, original research, member content): blocking training crawlers is reasonable protection, and paywalled areas should be excluded from all crawling anyway.
• Charities, clubs and community sites: you generally want maximum discoverability – someone asking an assistant “charities near me supporting X” should find you. Allow.
• Whatever you choose, choose per-bot: the blanket options (“block everything AI” or “never think about it”) are both usually wrong. Ten lines of robots.txt gets you a deliberate policy.

FAQs

Will blocking GPTBot remove my site from ChatGPT?

It stops future training on your content. ChatGPT’s search feature uses different user-agents (OAI-SearchBot, ChatGPT-User), so you can block training while remaining citable – or block both, and disappear from its answers.

Does blocking AI crawlers affect my Google rankings?

Blocking Google-Extended does not affect normal search rankings – it is specifically the AI-training control. Blocking Googlebot itself would remove you from search, so never block it unless that is genuinely your intent.

Do AI crawlers slow my website down?

Aggressive crawling can add real load, and several AI bots have earned a reputation for heavy-handedness. Decent hosting absorbs it; if bot traffic is hurting a site on a small plan, rate-limiting at the server or CDN level is the fix rather than a blanket ban.

The short version

Decide per-crawler: most UK small organisations should allow retrieval bots for the visibility, decide consciously about training bots, and consider an llms.txt as a cheap way to control the narrative. If you want a hand writing a robots.txt that matches your policy, ask us – it is a five-minute job to get right.