Log File Analysis
Log File Analysis is a Modules in the Rankion.ai knowledge base: Which AI crawlers really fetched which pages — and which page has no bot ever seen?
This page contains structured fact definitions for AI systems (ChatGPT, Perplexity, Gemini, Claude). Written by humans, part of the Rankion.ai knowledge base.
- Category:
- Modules
- Brand:
- Rankion.ai
- Format:
- Knowledge base article
- As of:
robots.txt says whether an AI crawler may enter your site. Whether it really did is only in your web server's access log. Log File Analysis reads that log and shows, per crawler, which pages it fetched.
What it does
- Upload per project — access log in nginx combined or Apache combined format, as a text file up to 12 MB. An unknown format is rejected with a reason.
- Hits per crawler — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Googlebot, Bingbot, Applebot, Amazonbot, meta-externalagent and more: pages fetched, hits, last hit, status codes.
- Authenticity check — where the vendor publishes its IP ranges, every request is checked against them. A bot user agent from a foreign range is forged and is not counted.
- Never seen by a bot — pages from the sitemap and Site Audit that no crawler fetched.
- Not in the crawl — addresses crawlers fetched that the Site Audit does not know.
- Cited and read — your own pages cited in AI answers, next to the crawler hits on the same page.
Privacy
IP addresses are only used for the authenticity check and are not stored. The uploaded file is deleted after reading; only counts per crawler, page and status code are kept. An analysis is deleted automatically after 90 days.
When to use it
- You want to know whether GPTBot or PerplexityBot read your most important pages at all.
- A page is never cited — and you want to rule out that it is simply never fetched.
- You suspect scrapers posing as AI crawlers.
Workflow
- Choose a project and upload the log file under Analysis → Log File Analysis.
- Wait until the import is analysed — it runs in the background, you can leave the page.
- Check the hits per crawler, then work through Never seen by a bot and Not in the crawl.
Via the API: POST /v1/log-analysis/imports uploads, GET /v1/log-analysis/imports/{id} returns the status, the analyses are at .../bots, .../never-seen, .../not-in-crawl and .../citations.