# robots.txt

> robots.txt is a plain text file at the root of a website that tells crawlers which parts of the site they may and may not visit.

- **Check ID:** S19
- **Area:** [SEO](https://be1st.ai/glossary/seo)
- **Tool:** [SEO + Security + Performance](https://be1st.ai/tools/ssp-audit)
- **Page:** https://be1st.ai/glossary/seo/robots-txt

## Why it matters

Robots.txt is the first file a search engine crawler reads when visiting a website. A properly configured robots.txt optimizes crawl budget and keeps crawlers out of sections they should not crawl (it does not keep a page out of the index, that is what noindex is for).

## In practice

Facebook has an extensive robots.txt that blocks crawling of profiles but allows public pages. An incorrect robots.txt can block the entire site - a well-known case where 'Disallow: /' accidentally blocked an entire e-shop for weeks.

## How be1st checks it

Checks for the existence of a robots.txt file in the root of the website. This file tells search engine crawlers which parts of the site they can and cannot crawl.

[SEO + Security + Performance](https://be1st.ai/tools/ssp-audit), SEO tab.

Every check ends as: Pass, Warning, Fail.

## Sources

- [Robots.txt Introduction and Guide](https://developers.google.com/search/docs/crawling-indexing/robots/intro) (Google Search Central)
- [How Google Interprets the robots.txt Specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) (Google Crawling Infrastructure)

## Related terms

- [sitemap.xml](https://be1st.ai/glossary/seo/sitemap-xml): An XML sitemap is a file that lists the important URLs of a website, so search engines can find them without relying on links alone.
- [HTTPS (SEO)](https://be1st.ai/glossary/seo/https): HTTPS is the encrypted version of HTTP. It protects data between the browser and the server, and browsers mark sites without it as not secure.
- [WWW consistency](https://be1st.ai/glossary/seo/www-consistency): WWW consistency means a website answers on one host name only, either with www or without it, and redirects the other version there.
- [Trailing slash consistency](https://be1st.ai/glossary/seo/trailing-slash-consistency): Trailing slash consistency means every URL is used in one form only, either /page or /page/, and the other form redirects to it.
- [HTML size (SEO)](https://be1st.ai/glossary/seo/html-size): HTML size is the weight of a page's HTML document in kilobytes, without images, styles and scripts. Search engines and browsers have to download and parse all of it.
- [Mobile-friendly indicators](https://be1st.ai/glossary/seo/mobile-friendly-indicators): Mobile-friendly indicators are the signs that a page adapts to small screens: a viewport tag, responsive CSS and content that fits without zooming.