# Be1stBot

Be1stBot audits websites for be1st.ai users.

Be1stBot is the bot of be1st.ai, a website intelligence service for developers and vibecoders. It checks a site's technology stack, SEO, security, performance and AI readiness. It is not a search engine crawler and not an AI-training crawler.

Page: https://be1st.ai/bot

## At a glance

- **Match on:** `Be1stBot`
- **Version:** 1.0
- **IP address:** `77.240.183.107`
- **robots.txt:** Followed, except for the contact lookup
- **Category:** SEO / site auditing
- **Contact:** support@be1st.ai

## When Be1stBot visits

Be1stBot does not crawl the web on its own. It fetches a site only when a be1st.ai user asks for it, in one of these ways:

- **A user runs an audit:** Someone enters your site in be1st.ai and starts an audit.
- **A recurring audit is due:** A user has scheduled regular audits of your site.
- **A user monitors your site:** Your site is on a be1st.ai user's monitoring list.
- **An AI agent asks for it:** A user's AI agent does any of the above through the be1st.ai MCP server or API.

In Cloudflare's bot categories, Be1stBot belongs to SEO / site auditing.

## User agent strings

Requests from Be1stBot carry the token `Be1stBot/1.0` and a link to this page. To recognize the bot, match on `Be1stBot`: the rest of the string can change.

### HTTP requests

Plain requests without a browser.

```text
Mozilla/5.0 (compatible; Be1stBot/1.0; +https://be1st.ai/bot)
```

### Chrome rendering, desktop

A real Chrome browser that renders the homepage for screenshots and performance.

```text
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<version>.0.0.0 Safari/537.36 (compatible; Be1stBot/1.0; +https://be1st.ai/bot)
```

### Chrome rendering, mobile

The same real Chrome, on a mobile screen.

```text
Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<version>.0.0.0 Mobile Safari/537.36 (compatible; Be1stBot/1.0; +https://be1st.ai/bot)
```

`<version>` is the Chrome version we currently run, so it changes when Chrome updates.

## What one audit fetches

One audit is a short, bounded visit, not a crawl of your whole site. It fetches:

- `robots.txt`, to learn what it may fetch
- The homepage
- The sitemap
- `llms.txt` and Markdown versions of pages
- A limited sample of subpages
- When the user asks for contacts: up to 8 pages such as Contact, About or Imprint, and up to 8 of the site's script files, to find a published email, phone number or contact form
- The homepage rendered in Chrome on desktop and mobile, including the page's own assets like a normal visitor, for screenshots and performance
- Lightweight HEAD requests that check the links found on the page

## robots.txt and Crawl-delay

Be1stBot follows robots.txt as defined in RFC 9309, plus the `Crawl-delay` directive.

- **Which group applies:** Be1stBot follows the `User-agent: Be1stBot` group. If there is none, it follows `User-agent: *`. Only one group applies, so if you add a Be1stBot group, repeat in it any `*` rules that should still apply.
- **Supported rules:** `Allow` and `Disallow`, including the `*` wildcard and the `$` end anchor, plus `Crawl-delay`. The longest matching rule wins; on a tie, `Allow` wins.
- **When changes apply:** robots.txt is cached for up to 1 hour, so a change applies within an hour.
- **When robots.txt fails:** A 4xx response (for example 404) means no restrictions. A server error (5xx) means the whole site is treated as disallowed, the same way Google handles it.
- **Disallowed pages:** The audit does not fetch them and tells the user that the site owner disallowed them. The contact lookup is the one exception, see below.
- **Contact lookup:** When the user asks for a site's contacts, Be1stBot reads up to 8 public pages such as Contact, About or Imprint, and up to 8 of the site's script files, to find a published email, phone number or contact form. It identifies itself as Be1stBot on these requests too, but per-page robots.txt rules and `Crawl-delay` do not apply to them. If robots.txt disallows Be1stBot on the homepage (for example `Disallow: /`), the audit stops before this step and the contact pages are not fetched either.
- **When a site blocks bots in general:** If your site turns away automated visitors in general (for example a 403 or a challenge for every bot), the audit tries again as a regular Chrome browser without the Be1stBot token, possibly through a different network, so the user who asked for the audit gets a result. robots.txt is respected in this case too: to keep Be1stBot out reliably, disallow it in robots.txt.

### Copy-paste examples

#### Block Be1stBot completely

Be1stBot reads robots.txt and nothing else.

```text
User-agent: Be1stBot
Disallow: /
```

#### Block one section

Keeps Be1stBot out of `/admin/`, the rest of the site stays open.

```text
User-agent: Be1stBot
Disallow: /admin/
```

#### Slow it down

Be1stBot waits at least 10 seconds between requests to your site.

```text
User-agent: Be1stBot
Crawl-delay: 10
```

## IP addresses

All requests with the Be1stBot user agent come from this IPv4 address. Be1stBot does not use IPv6.

- `77.240.183.107`

Machine-readable list: https://be1st.ai/bot/ips.json

```json
{
  "creationTime": "2026-11-07T00:00:00.000000",
  "prefixes": [
    {
      "ipv4Prefix": "77.240.183.107/32"
    }
  ]
}
```

The list can change. When it does, we update ips.json, and its `creationTime` shows when the list was generated.

To make sure a request really comes from Be1stBot, check both: the user agent contains `Be1stBot` and the IP address is on this list.

## Request rate

Requests to one site are spaced out, not sent all at once. Be1stBot slows down on its own when your server returns errors, and it honors `Crawl-delay` everywhere except the contact lookup.

## How fetched content is used

Fetched content is used only to produce the audit report for the be1st.ai user who requested it. It does not go into a search index or into AI training.

## Version history

- **1.0** (November 2026): First version of the unified identity.

## Questions or problems?

Write to us if Be1stBot causes trouble on your site, if you have a question, or if you want your site excluded.

E-mail: support@be1st.ai

Include your domain and the approximate time of the requests, it helps us find them quickly.