# AI bot policy

> An AI bot policy is the set of robots.txt rules for AI crawlers such as GPTBot, ClaudeBot or Google-Extended that allow or block them on a website.

- **Check ID:** A5
- **Area:** [AI Readiness](https://be1st.ai/glossary/ai-readiness)
- **Tool:** [AI Readiness](https://be1st.ai/tools/ai-readiness)
- **Page:** https://be1st.ai/glossary/ai-readiness/ai-bot-policy

## Why it matters

Strategic management of AI bot access is crucial. Complete blocking means your company won't exist in AI answers. Conversely, full access may lead to unwanted training on your content without compensation. An unclear AI bot policy leads to inconsistent crawler behavior.

## In practice

The New York Times blocked GPTBot in robots.txt, protecting its content from AI training but losing visibility in ChatGPT. Conversely, Stripe allows AI crawlers because it wants to be the primary source of information about payment APIs.

## How be1st checks it

Analyzes the robots.txt rules for AI crawlers: GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, PerplexityBot and Googlebot-Image. The check distinguishes whether the site blocks only the bots that collect model training data or also the bots that fetch content for answers. Blocking only the training bots is rated as a strategic tradeoff: it protects content but reduces AI visibility. If robots.txt is missing, the check passes only partially.

[AI Readiness](https://be1st.ai/tools/ai-readiness), Access tab.

Every check ends as: Pass, Warning, Fail.

## Sources

- [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) (OpenAI)
- [Anthropic Web Crawler Policy](https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (Anthropic)
- [AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) (Cloudflare)

## Related terms

- [robots.txt](https://be1st.ai/glossary/seo/robots-txt): robots.txt is a plain text file at the root of a website that tells crawlers which parts of the site they may and may not visit.
- [llms.txt existence](https://be1st.ai/glossary/ai-readiness/llms-txt-existence): llms.txt is a Markdown file at the root of a website that gives AI models a short overview of the site and links to its most important pages.
- [llms.txt content quality](https://be1st.ai/glossary/ai-readiness/llms-txt-content-quality): llms.txt content quality is how useful that file is to an AI model: a clear description of the site, working links to its sections and enough detail.
- [llms-full.txt](https://be1st.ai/glossary/ai-readiness/llms-full-txt): llms-full.txt is an extended version of llms.txt that holds the full text of a website's key documentation in one Markdown file.
- [BLUF (bottom line up front)](https://be1st.ai/glossary/ai-readiness/bluf-bottom-line-up-front): BLUF (Bottom Line Up Front) is a writing principle: put the answer or the key point first and add the details after it.
- [Content structure (lists, tables)](https://be1st.ai/glossary/ai-readiness/content-structure): Content structure is how a page organizes information into lists, tables and other formats that readers and AI models can scan quickly.