98% of UK small business websites let AI crawlers in

We read the robots.txt, sitemaps and indexing tags of 202 independent trades, hair and beauty, and dental websites. Almost none block ChatGPT, Claude or Perplexity. The bigger problem is search engines: about 1 in 14 homepages ask Google not to list them.

AI crawlers are almost never blocked

Only 5 of 202 sites block any of the eleven AI crawlers we checked. GPTBot (OpenAI's training crawler) and ClaudeBot are each blocked by 2%; every other AI crawler, including the ones that fetch pages live for ChatGPT, Claude and Perplexity answers, by 1%. For a local business that is probably the right default: being readable is how it gets recommended when someone asks an assistant for a plumber or a dentist nearby.

1 in 14 homepages are hidden from Google

7% of homepages carry a noindex directive in a robots meta tag or an X-Robots-Tag header. That instruction tells Google and Bing to leave the page out of search results. It is often left over from a site build or a "coming soon" setting and never removed, and the business owner has no way to notice it from the page itself.

Sitemaps: mostly there, often stale

Most sites have the basics, largely because website builders generate them. But of the 126 sitemaps that carry dates, 26% have not had a new or updated page in over a year, which tells crawlers the site has not changed.

llms.txt: more common than expected, mostly automatic

30% of sites serve an llms.txt file, the proposed summary file for language models. That is far higher than owner effort would explain. Wix, for example, now generates and manages llms.txt automatically. Many of these businesses likely have one without knowing.

By sector

Method and limits

• Businesses: random sample of independent businesses (no brand tag) with a website listed in OpenStreetMap, within ten UK city areas, 80 per sector. OpenStreetMap data © OpenStreetMap contributors, ODbL.
• Read: 202 of 240. The rest were dead domains, certificate errors, error pages or timeouts.
• robots.txt was evaluated for the homepage with Google's documented matching rules (longest match wins, Allow wins ties, a crawler's own group overrides the wildcard group). AI crawlers checked: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Google-Extended, PerplexityBot, CCBot, Applebot-Extended, Bytespider, meta-externalagent.
• Blocking measured here is robots.txt only. Some sites also block bots at the firewall, which this study does not measure.
• Sitemaps: those declared in robots.txt (up to three), otherwise /sitemap.xml. Freshness uses the newest lastmod date.
• Each site was fetched once, on 18 October 2026. Individual businesses are not named.

Run the same check