> ## Documentation Index
> Fetch the complete documentation index at: https://docs.perquery.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM bot logs

> Upload the access log of your web server to see which AI crawlers visit your site, what they fetch and what your server answers.

**LLM Bot Logs** shows the AI crawler traffic on your site. The data comes from the access log of your web server. You upload the log file, and Perquery keeps only the lines of AI crawlers.

Open it from the **Mapping** group in the menu: **LLM Bot Logs**.

## Prepare the log file

Perquery reads the access log that your web server writes for each request. Nginx and Apache write this file by default.

The file must meet these conditions:

* The format is the Combined Log Format.
* The file is plain text or gzipped. The upload accepts `.log`, `.txt` and `.gz` files.
* The file is not larger than the maximum size that the upload dialog shows. Split a larger file and upload each part.

A line in the Combined Log Format looks like this:

```text theme={null}
203.0.113.7 - - [04/Nov/2026:10:15:32 +0000] "GET /pricing HTTP/1.1" 200 5120 "-" "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)"
```

Perquery identifies each crawler by its user agent. It keeps only the lines of AI crawlers. It never stores the client addresses.

## Upload a log

**1. Open the dialog**

Click **Upload log**. The **Upload an access log** dialog opens.

**2. Choose the file**

Select the access log file on your computer.

**3. Upload the file**

Click **Upload**. The dialog shows the status while Perquery parses the file.

**4. Close the dialog**

When the dialog shows that the file is parsed, click **Close**.

Upload a new log file at regular intervals to keep the data current. When you upload the same file again, Perquery shows that the file was already uploaded and does not count it twice.

The log volume that you can upload each month depends on your plan. See [pricing](https://perquery.ai/en/pricing).

## Recent uploads

The **Recent uploads** list shows each file that you uploaded:

* The number of lines that matched an AI crawler, of all lines.
* The status: **Parsing**, **Parsed**, **Duplicate** or **Failed**.
* **Unmatched agents:** the most frequent user agents in the file that are not AI crawlers.

A failed upload shows the reason:

| Reason | What to do |
| - | - |
| **Could not be read** | Check the file, then upload it again. |
| **Unrecognised log format** | Use the Combined Log Format. |
| **No AI crawler lines** | Upload a log that covers a period with AI crawler traffic. |
| **Too large** | Split the file and upload each part. |

## Choose the date range

Use the **Date range** control at the top of the page to choose the period. Use **Compare to** to choose the period that the changes are measured against.

When the range has no bot traffic, the page shows **No bot traffic in this range.** and the period that your logs cover. Click **Show the range with data** to select that period.

## Read the report

| Section | Content |
| - | - |
| **Bot access overview** | **Total bot visits**, **200 OK rate** and **Busiest bot**, with the change from the comparison period. |
| **Bot visit trend** | One line for each crawler, for each day. |
| **Why the bot came** | The visits by the purpose of the crawler: **Citation**, **Indexing** or **Training**. |
| **HTTP status distribution** | The status codes that your server answered to the crawlers. |
| **Most and never fetched pages** | The bot hits for each path, and the crawled pages that no crawler fetched. |

The pages that no crawler fetched show the mark **never fetched**. To see them, Perquery needs a crawl of your site. See [Content analysis](https://docs.perquery.ai/en/analyze/content-analysis.md).

## The crawlers that Perquery counts

| AI engine | Crawlers |
| - | - |
| ChatGPT | `ChatGPT-User` (citation), `OAI-SearchBot` (indexing), `GPTBot` (training) |
| Claude | `Claude-User` (citation), `Claude-SearchBot` (indexing), `ClaudeBot` (training) |
| Perplexity | `Perplexity-User` (citation), `PerplexityBot` (indexing) |
| Gemini | `GoogleOther` (indexing), `Google-Extended` (training) |
| Other | `Amazonbot` (indexing), `Bytespider`, `meta-externalagent`, `CCBot` and `Applebot-Extended` (training) |

The same crawlers appear in **Bot Access Status** and **Crawl Budget** on [Technical analysis](https://docs.perquery.ai/en/analyze/technical-analysis.md).

> [!TIP]
>
> Use the export menu on the **Most and never fetched pages** panel to download the data as CSV or Excel.

## Related pages

> [card] [Technical analysis](https://docs.perquery.ai/en/analyze/technical-analysis.md)
>
> See the crawl budget and the robots.txt status of each crawler.

> [card] [robots.txt Checker & Generator](https://docs.perquery.ai/en/tools/robots-txt.md)
>
> Allow or block AI crawlers.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.