Skip to main content
With Conductor Monitoring’s Log File Analysis feature, you can easily understand how search engine crawlers traverse the content on your website. Discover when Google, Bing, OpenAI, or Perplexity last visited each page, and compare visit frequencies between pages.

​
What is Log File Analysis?

A log file is a text file containing records of all the requests a server has received, from both humans and crawlers, and its responses to the requesters. Through log file analysis, AEO and SEO professionals can get a better understanding of what search engines are actually doing on their websites, in order to improve their performance.

​
Useful resources

Log File Analysis for SEO: An Introduction

​
Log Sources

Currently you can feed your logs to Conductor Monitoring using any of the following CDN platforms:

​
Can I integrate a data aggregation platform like Splunk instead?

No, currently Conductor Monitoring supports only direct integrations with the CDNs listed above.

​
Traffic that Conductor Monitoring detects and tracks

Conductor Monitoring detects and tracks visits of the following search engines to your websites:
  • Google Desktop
  • Google Mobile
  • Bing Desktop
  • Bing Mobile
  • OpenAI (OAI-SearchBot, ChatGPT-User, and GPTBot)
  • Perplexity (PerplexityBot and Perplexity-User)
You can learn more about these engines’ crawlers in our Log File Analysis FAQs article.

​
Setting up the Log File Analysis

The process of configuring the Log File Analysis differs based on the log sources that are used to feed them to Conductor Monitoring. Refer to the relevant support article based on your website’s log source:
  1. Setting up the Cloudflare Worker integration
  2. Setting up the Cloudflare Logpush integration
  3. Setting up the Akamai integration
  4. Setting up CloudFront Standard logging
  5. Setting up Fastly Real-Time Log Streaming

​
Working with the Log File Analysis data in Conductor Monitoring

​
Using the Log File Analysis data in Pages

Search engines will have two columns each on the Pages screen: Last Visited and Visit Frequency, which you can use for filtering: contentking-lfa-pages-screen@2x.png When filtering on the columns, you can easily create segments which can be very useful to do spot checks of your least visited pages. On the Page Detail screen, you can zoom in on how often crawlers are visiting a specific page. With a neat graph you can identify any trends in the search engine traffic on that page: contentking-lfa-page-detail-screen@2x.png And in the Tracked Changes tab of the Page Detail screen, you will be able to see whether search engines saw the latest important change that you’ve made. In combination with our Real-Time IndexNow feature you can see if Bing visited the page that was submitted to IndexNow: contentking-lfa-tracked-changes-screen@2x.png

​
Log File Analysis data in the Platform section

If you have made some changes to the robots.txt file or the XML sitemaps and you want to check if search engines visited them since, just head to the Platform section in Conductor Monitoring and easily check if the search engines visited the robots.txt: contentking-lfa-robotstxt@2x.png Or the XML sitemaps: contentking-lfa-xml-sitemap@2x.png

​
Disabling Log File Analysis

To learn how to disable Log File Analysis please refer to the relevant support article based on your website’s log source:

​
Using Log File Analysis

Curious why (and how) you can use Log File Analysis to inform your content strategy in the world of AI search? Check our AI Crawler Activity in Conductor Monitoring guide to learn!

​
FAQs

Review this feature’s frequently asked questions