robots.txt Checker & Generator

When your robots.txt blocks the bot of an AI engine, your site cannot appear on that AI engine. The robots.txt Checker & Generator shows the permission of each AI bot and builds a corrected file.

​

Check the bot permissions

Open the tool. In the sidebar, under Tools, click robots.txt Checker & Generator. The Bot permission table reads https://<your domain>/robots.txt. The card shows the URL of the file. When there is no file, the card shows the URL with not found. The table has one row for each AI bot. Each row shows these values:

• Bot: the name of the bot.
• Platform: the AI platform that uses the bot.
• Status: Allowed, Blocked or Not listed. Not listed means that your file does not name the bot.

A row with the status Blocked also shows a Fix button. To download the table, click Export and select CSV or Excel (.xlsx).

​

Unblock one bot

The new file allows that bot on the whole site. It keeps your other rules. If your file declares no sitemap, the new file adds Sitemap: https://<your domain>/sitemap.xml.

​

Generate a file without a fix

In the Generator card, click Generate.

• When your site has a robots.txt, Perquery keeps it. It adds a sitemap line only when the file declares no sitemap.
• When your site has no robots.txt, Perquery builds a file that allows all bots and declares https://<your domain>/sitemap.xml.

To build the file again, click Regenerate.

​

Put the file on your site

Upload the file to the root of your domain, so that it opens at https://<your domain>/robots.txt. Then open the tool again and examine the Bot permission table.

​

Open the tool from an opportunity

An opportunity can name a blocked bot. When you open the tool from that opportunity, the generator builds the file for that bot at once.

​

Plan usage

The check and the generator do not count against your plan’s allowance.

​

Related pages

• LLM bot logs
• Technical analysis
• llms.txt Checker & Generator