Skip to main content
If your log file provider isn’t one of the CDNs Conductor Intelligence integrates with directly, you can use the Custom S3 log source to send log file data to the Pages report. With Custom S3, you convert your logs into one of Conductor’s supported log formats and upload them to an Amazon S3 bucket that Conductor Intelligence creates for you.
With Custom S3, you’re responsible for converting your log files and sending them to Conductor. Conductor Intelligence doesn’t connect to your log file provider.
Integrating Custom S3 with Conductor Intelligence consists of three phases:
  • Choosing the log format your files will use.
  • Creating an Amazon S3 bucket and credentials in Conductor Intelligence.
  • Uploading your converted log files to the bucket.

​
Choosing a log format

The log format is part of how your bucket is set up and can’t be changed later. To switch formats, remove the log source and connect Custom S3 again. This creates a new bucket and new credentials.
Conductor Intelligence reads Custom S3 log files in the same formats it accepts from its supported CDN integrations. You select the format during setup, and every file you upload for that subdomain must use it.
If your logs aren’t already in one of these formats, we recommend Fastly. It’s one JSON object per line with six fields, so it’s the easiest to produce from any web server or CDN log.
Field names are case sensitive and must match exactly. Conductor ignores any other fields. One JSON object per line.

​
Akamai

One JSON object per line, using DataStream 2 field names.

​
Cloudflare

One JSON object per line, using Logpush http_requests field names.

​
CloudFront

CloudFront legacy standard log format: tab-separated, all 26 columns in the standard order (date through fle-encrypted-fields). Lines starting with # are ignored. Use - for empty values.

​
Client IP addresses

Conductor Intelligence verifies bot visits by checking that the request came from the bot’s published IP ranges. The IP field in your logs must contain the visitor’s original IP address, not the address of a load balancer, proxy, or CDN in front of your server. If your server sits behind a proxy, take the IP from the X-Forwarded-For header (the first address in the list) or from your proxy’s equivalent header. Visits logged with a proxy’s IP address aren’t counted.

​
Creating an Amazon S3 bucket in Conductor Intelligence

In Conductor Intelligence:
  1. Go to the Pages & Crawling settings activity (Settings > Pages & Crawling).
  2. From the list of configured subdomains, identify the subdomain with the pages for which you want to see log file data.
  3. In the Log Source column for that subdomain, click Connect.
  4. In the Custom S3 section, from the Connect a log source window, click Connect.
  5. In the Connect Custom S3 window:
    1. If you have other subdomains under the same root domain without a log source, the Also set up Custom S3 for my other [root domain] subdomains option is selected by default. Leave it selected to set up all of them at once, or clear it to set up only this subdomain.
    2. From the Log format menu, select the format your log files will use: Akamai, Fastly, Cloudflare, or CloudFront.
    3. Click Create Bucket.
    If you need to choose a different log source, click Switch Provider instead.
  6. When the Bucket ready message appears, Conductor Intelligence displays the details you’ll need to upload your log files:
    • Bucket
    • Region
    • Access Key ID
    • Secret Access Key
    • Log format
    Use the copy icon next to each value to copy it.
    Copy and save the Access Key ID and Secret Access Key now. They aren’t shown again after you click Complete Integration. Treat them like a password: anyone with them can upload files to your bucket, so share them only with the people or systems responsible for sending your log files.
  7. Click Complete Integration.

​
Uploading your log files

Use the Bucket, Access Key ID, and Secret Access Key from Conductor Intelligence to upload your converted log files. The bucket is in the us-east-1 region. Any tool that can write to Amazon S3 with these credentials works. Your files must meet these requirements:
  • Encoding: UTF-8 text.
  • Compression: optional. Gzip is supported only when the file name ends in .gz. Other compression formats aren’t supported.
  • Size: at most 100 MB per file. Compressed files can expand to at most 1 GB. Split larger files before uploading.
  • Timestamps: every row needs a valid timestamp in the format’s timestamp field. Rows with a missing or malformed timestamp won’t appear in your reports.
  • File names: any name and folder path is fine.
For example, with the AWS CLI:
  1. Store the credentials in a named profile. This is a one-time setup step:
    When prompted, enter your Access Key ID and Secret Access Key, and enter us-east-1 as the default region name.
  2. Upload your files using one of these commands. Replace <Bucket> with your bucket name. To upload a single file:
    To upload every file in a folder:
    To upload only files that aren’t in the bucket yet:
Conductor Intelligence processes each file shortly after it lands in the bucket. To keep your Pages report current, upload new log files on a regular schedule (for example, hourly or daily) rather than as a one-time transfer.

​
Filtering your logs to bot traffic

To ensure you capture visits from all of the bots for which Conductor supports detection, make sure the log files you upload include requests from all of the following user agents:
  • bingbot
  • Googlebot
  • OAI-SearchBot
  • ChatGPT-User
  • GPTBot
  • PerplexityBot
  • Perplexity-User
  • ClaudeBot
  • Claude-SearchBot
  • Claude-User
  • GoogleOther
  • Google-Agent
  • OAI-AdsBot
  • GrokBot
Conductor matches these names anywhere in the User-Agent header, regardless of upper- or lower-case. Keep the full, unmodified header in your logs. Conductor Intelligence processes only search engine and AI bot visits. Because you control the conversion process, we recommend filtering out all other traffic before you upload your files so that no unnecessary data—including personally identifiable information—is sent to the bucket.

​
Disabling log file data in Conductor Intelligence’s Pages reports

  1. In Pages & Crawling, hover over the subdomain for which you want to remove Custom S3 and click Edit in the Log Source column.
  2. In the bottom-right of the Connect Custom S3 form, click Remove.
  3. Click Remove Log Source to confirm.
Removing the log source deletes the bucket, any files in it, and its credentials. Stop any scheduled uploads to the bucket. They’ll fail with an access error once the log source is removed.