> ## Documentation Index
> Fetch the complete documentation index at: https://www.conductor.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Integrate Custom S3 with Conductor Intelligence

> Send log file data from any log file provider to Conductor Intelligence's Pages report by uploading files in a supported format to a Conductor-managed Amazon S3 bucket.

If your log file provider isn't one of the CDNs Conductor Intelligence integrates with directly, you can use the **Custom S3** log source to send log file data to the Pages report. With Custom S3, you convert your logs into one of Conductor's supported log formats and upload them to an [Amazon S3 bucket](https://aws.amazon.com/s3/) that Conductor Intelligence creates for you.

> [!NOTE]
>
> With Custom S3, you're responsible for converting your log files and sending them to Conductor. Conductor Intelligence doesn't connect to your log file provider.

Integrating Custom S3 with Conductor Intelligence consists of three phases:

* Choosing the log format your files will use.
* Creating an Amazon S3 bucket and credentials in Conductor Intelligence.
* Uploading your converted log files to the bucket.

## Choosing a log format

> [!WARNING]
>
> The log format is part of how your bucket is set up and can't be changed later. To switch formats, remove the log source and connect Custom S3 again. This creates a new bucket and new credentials.

Conductor Intelligence reads Custom S3 log files in the same formats it accepts from its supported CDN integrations. You select the format during setup, and every file you upload for that subdomain must use it.

> [!TIP]
>
> If your logs aren't already in one of these formats, we recommend **Fastly**. It's one JSON object per line with six fields, so it's the easiest to produce from any web server or CDN log.

Field names are case sensitive and must match exactly. Conductor ignores any other fields.

### Fastly (recommended)

One JSON object per line.

| Field | Required | Description |
| - | - | - |
| `url` | Yes | Full request URL including scheme and query string, for example `https://www.example.com/products?id=1` |
| `timestampMs` | Yes | Request time as Unix epoch **milliseconds**, as a whole number, for example `1767780508000` |
| `ip` | Yes | The visitor's IP address (see [Client IP addresses](#client-ip-addresses)) |
| `userAgent` | Yes | The full `User-Agent` request header |
| `method` | No | HTTP method. Defaults to `GET` |
| `host` | No | Request host. Derived from `url` if missing |

```json theme={null}
{"url":"https://www.example.com/products?id=1","timestampMs":1767780508000,"method":"GET","ip":"66.249.66.1","userAgent":"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"}
```

### Akamai

One JSON object per line, using DataStream 2 field names.

| Field | Description |
| - | - |
| `reqHost` | Request host |
| `reqPath` | Request path |
| `queryStr` | Query string without `?`, or `-` if none |
| `tlsVersion` | TLS version, or `-` for plain HTTP (used to tell `http` from `https`) |
| `reqTimeSec` | Request time as Unix epoch **seconds** (decimals allowed) |
| `reqMethod` | HTTP method |
| `cliIP` | Visitor IP address |
| `UA` | `User-Agent` header |

### Cloudflare

One JSON object per line, using Logpush `http_requests` field names.

| Field | Description |
| - | - |
| `ClientRequestScheme` | `http` or `https` |
| `ClientRequestHost` | Request host |
| `ClientRequestURI` | Path and query string |
| `EdgeEndTimestamp` | ISO 8601 UTC time ending in `Z`, for example `2026-01-07T09:48:28Z` or `2026-01-07T09:48:28.000Z` |
| `ClientRequestMethod` | HTTP method |
| `ClientIP` | Visitor IP address |
| `ClientRequestUserAgent` | `User-Agent` header |

### CloudFront

CloudFront [legacy standard log](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/standard-logs-reference.html) format: tab-separated, all 26 columns in the standard order (`date` through `fle-encrypted-fields`). Lines starting with `#` are ignored. Use `-` for empty values.

## Client IP addresses

Conductor Intelligence verifies bot visits by checking that the request came from the bot's published IP ranges. The IP field in your logs must contain the visitor's original IP address, not the address of a load balancer, proxy, or CDN in front of your server.

If your server sits behind a proxy, take the IP from the `X-Forwarded-For` header (the first address in the list) or from your proxy's equivalent header. Visits logged with a proxy's IP address aren't counted.

## Creating an Amazon S3 bucket in Conductor Intelligence

In Conductor Intelligence:

1. Go to the [Pages & Crawling](https://app.conductor.com/u/pages-and-crawling) settings activity (**Settings** > **Pages & Crawling**).
2. From the list of configured subdomains, identify the subdomain with the pages for which you want to see log file data.
3. In the **Log Source** column for that subdomain, click **Connect**.
4. In the Custom S3 section, from the **Connect a log source** window, click **Connect**.
5. In the **Connect Custom S3** window:

   1. If you have other subdomains under the same root domain without a log source, the **Also set up Custom S3 for my other \[root domain] subdomains** option is selected by default. Leave it selected to set up all of them at once, or clear it to set up only this subdomain.
   2. From the **Log format** menu, select the format your log files will use: **Akamai**, **Fastly**, **Cloudflare**, or **CloudFront**.
   3. Click **Create Bucket**.

   If you need to choose a different log source, click **Switch Provider** instead.
6. When the **Bucket ready** message appears, Conductor Intelligence displays the details you'll need to upload your log files:

   * **Bucket**
   * **Region**
   * **Access Key ID**
   * **Secret Access Key**
   * **Log format**

   Use the copy icon next to each value to copy it.

   > [!WARNING]
   >
   > Copy and save the **Access Key ID** and **Secret Access Key** now. They aren't shown again after you click **Complete Integration**. Treat them like a password: anyone with them can upload files to your bucket, so share them only with the people or systems responsible for sending your log files.
7. Click **Complete Integration**.

## Uploading your log files

Use the **Bucket**, **Access Key ID**, and **Secret Access Key** from Conductor Intelligence to upload your converted log files. The bucket is in the `us-east-1` region. Any tool that can write to Amazon S3 with these credentials works.

Your files must meet these requirements:

* **Encoding**: UTF-8 text.
* **Compression**: optional. Gzip is supported only when the file name ends in `.gz`. Other compression formats aren't supported.
* **Size**: at most 100 MB per file. Compressed files can expand to at most 1 GB. Split larger files before uploading.
* **Timestamps**: every row needs a valid timestamp in the format's timestamp field. Rows with a missing or malformed timestamp won't appear in your reports.
* **File names**: any name and folder path is fine.

For example, with the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html):

1. Store the credentials in a named profile. This is a one-time setup step:

   ```bash theme={null}
   aws configure --profile conductor-lfa
   ```

   When prompted, enter your **Access Key ID** and **Secret Access Key**, and enter `us-east-1` as the default region name.
2. Upload your files using one of these commands. Replace `<Bucket>` with your bucket name.

   To upload a single file:

   ```bash theme={null}
   aws s3 cp access-2026-01-07.json.gz s3://<Bucket>/2026/01/07/ --profile conductor-lfa
   ```

   To upload every file in a folder:

   ```bash theme={null}
   aws s3 cp ./converted-logs/ s3://<Bucket>/2026/01/07/ --recursive --profile conductor-lfa
   ```

   To upload only files that aren't in the bucket yet:

   ```bash theme={null}
   aws s3 sync ./converted-logs/ s3://<Bucket>/ --profile conductor-lfa
   ```

Conductor Intelligence processes each file shortly after it lands in the bucket. To keep your Pages report current, upload new log files on a regular schedule (for example, hourly or daily) rather than as a one-time transfer.

## Filtering your logs to bot traffic

To ensure you capture visits from all of the bots for which Conductor supports detection, make sure the log files you upload include requests from all of the following user agents:

* bingbot
* Googlebot
* OAI-SearchBot
* ChatGPT-User
* GPTBot
* PerplexityBot
* Perplexity-User
* ClaudeBot
* Claude-SearchBot
* Claude-User
* GoogleOther
* Google-Agent
* OAI-AdsBot
* GrokBot

Conductor matches these names anywhere in the `User-Agent` header, regardless of upper- or lower-case. Keep the full, unmodified header in your logs.

Conductor Intelligence processes only search engine and AI bot visits. Because you control the conversion process, we recommend filtering out all other traffic before you upload your files so that no unnecessary data—including personally identifiable information—is sent to the bucket.

## Disabling log file data in Conductor Intelligence's Pages reports

1. In **Pages & Crawling**, hover over the subdomain for which you want to remove Custom S3 and click **Edit** in the **Log Source** column.
2. In the bottom-right of the **Connect Custom S3** form, click **Remove**.
3. Click **Remove Log Source** to confirm.

Removing the log source deletes the bucket, any files in it, and its credentials. Stop any scheduled uploads to the bucket. They'll fail with an access error once the log source is removed.