Skip to main content
LLM Bot Logs shows the AI crawler traffic on your site. The data comes from the access log of your web server. You upload the log file, and Perquery keeps only the lines of AI crawlers. Open it from the Mapping group in the menu: LLM Bot Logs.

​
Prepare the log file

Perquery reads the access log that your web server writes for each request. Nginx and Apache write this file by default. The file must meet these conditions:
  • The format is the Combined Log Format.
  • The file is plain text or gzipped. The upload accepts .log, .txt and .gz files.
  • The file is not larger than the maximum size that the upload dialog shows. Split a larger file and upload each part.
A line in the Combined Log Format looks like this:
Perquery identifies each crawler by its user agent. It keeps only the lines of AI crawlers. It never stores the client addresses.

​
Upload a log

1

Open the dialog

Click Upload log. The Upload an access log dialog opens.
2

Choose the file

Select the access log file on your computer.
3

Upload the file

Click Upload. The dialog shows the status while Perquery parses the file.
4

Close the dialog

When the dialog shows that the file is parsed, click Close.
Upload a new log file at regular intervals to keep the data current. When you upload the same file again, Perquery shows that the file was already uploaded and does not count it twice. The log volume that you can upload each month depends on your plan. See pricing.

​
Recent uploads

The Recent uploads list shows each file that you uploaded:
  • The number of lines that matched an AI crawler, of all lines.
  • The status: Parsing, Parsed, Duplicate or Failed.
  • Unmatched agents: the most frequent user agents in the file that are not AI crawlers.
A failed upload shows the reason:

​
Choose the date range

Use the Date range control at the top of the page to choose the period. Use Compare to to choose the period that the changes are measured against. When the range has no bot traffic, the page shows No bot traffic in this range. and the period that your logs cover. Click Show the range with data to select that period.

​
Read the report

The pages that no crawler fetched show the mark never fetched. To see them, Perquery needs a crawl of your site. See Content analysis.

​
The crawlers that Perquery counts

The same crawlers appear in Bot Access Status and Crawl Budget on Technical analysis.
Use the export menu on the Most and never fetched pages panel to download the data as CSV or Excel.

Technical analysis

See the crawl budget and the robots.txt status of each crawler.

robots.txt Checker & Generator

Allow or block AI crawlers.