> ## Documentation Index
> Fetch the complete documentation index at: https://docs.perquery.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# robots.txt Checker & Generator

> See which AI bots your robots.txt lets in, and generate a corrected robots.txt file.

When your `robots.txt` blocks the bot of an AI engine, your site cannot appear on that AI engine. The **robots.txt Checker & Generator** shows the permission of each AI bot and builds a corrected file.

## Check the bot permissions

Open the tool. In the sidebar, under **Tools**, click **robots.txt Checker & Generator**.

The **Bot permission table** reads `https://<your domain>/robots.txt`. The card shows the URL of the file. When there is no file, the card shows the URL with **not found**.

The table has one row for each AI bot. Each row shows these values:

* **Bot**: the name of the bot.
* **Platform**: the AI platform that uses the bot.
* **Status**: **Allowed**, **Blocked** or **Not listed**. **Not listed** means that your file does not name the bot.

A row with the status **Blocked** also shows a **Fix** button.

To download the table, click **Export** and select **CSV** or **Excel (.xlsx)**.

## Unblock one bot

**1. Find the blocked bot**

Find a row with the status **Blocked**.

**2. Fix the bot**

Click **Fix** in that row. The **Generator** card builds a `robots.txt` that allows that bot.

**3. Copy or download**

Click **Copy** to copy the text, or click **Download** to save it as `robots.txt`.

The new file allows that bot on the whole site. It keeps your other rules. If your file declares no sitemap, the new file adds `Sitemap: https://<your domain>/sitemap.xml`.

## Generate a file without a fix

In the **Generator** card, click **Generate**.

* When your site has a `robots.txt`, Perquery keeps it. It adds a sitemap line only when the file declares no sitemap.
* When your site has no `robots.txt`, Perquery builds a file that allows all bots and declares `https://<your domain>/sitemap.xml`.

To build the file again, click **Regenerate**.

## Put the file on your site

> [!WARNING]
>
> Read the generated file before you publish it. The new file replaces your current `robots.txt`.

Upload the file to the root of your domain, so that it opens at `https://<your domain>/robots.txt`. Then open the tool again and examine the **Bot permission table**.

## Open the tool from an opportunity

An [opportunity](https://docs.perquery.ai/en/act/opportunities.md) can name a blocked bot. When you open the tool from that opportunity, the generator builds the file for that bot at once.

## Plan usage

The check and the generator do not count against your plan's allowance.

## Related pages

* [LLM bot logs](https://docs.perquery.ai/en/analyze/bot-logs.md)
* [Technical analysis](https://docs.perquery.ai/en/analyze/technical-analysis.md)
* [llms.txt Checker & Generator](https://docs.perquery.ai/en/tools/llms-txt.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.