S3 Data Export

Purpose

The S3 Data Export service automatically delivers junction traffic statistics directly to your AWS S3 bucket. Instead of polling the API for data, this service pushes processed data files to your storage as soon as they become available.

The data flow is one-directional: TomTom exports data to your S3 bucket. No data is read from your bucket.

Data pipelines

The S3 Data Export service supports two distinct data pipelines. Depending on your use case, you can subscribe to one or both.

Both pipelines deliver data in the same file format and directory structure. The key difference lies in when and how the data is produced:

• Live pipeline: data is collected in 15-minute intervals and exported to S3 approximately 30 minutes after each interval ends.
• Historical pipeline: data is collected and processed in 6-hour cycles, and the export to S3 is triggered once processing for each cycle is complete.

Delivery schedule comparison

Prerequisites

To enable the S3 Data Export integration, you must provide the following information to your TomTom account representative.

Required IAM permissions

The provided AWS credentials must have at minimum the s3:PutObject permission on the target bucket. We recommend creating a dedicated IAM user specifically for this integration.

Example minimal IAM policy:

1
{
2
  "Version": "2012-10-17",
3
  "Statement": [
4
    {
5
      "Effect": "Allow",
6
      "Action": "s3:PutObject",
7
      "Resource": "arn:aws:s3:::your-bucket-name/*"
8
    }
9
  ]
10
}

Security recommendations

• Create a dedicated IAM user specifically for this integration.
• Apply the principle of least privilege - grant only s3:PutObject permission.
• Enable S3 bucket versioning if you need to retain previously overwritten data.

Delivered data

File format

Both pipelines deliver data in the same format: gzip-compressed Apache Avro Object Container files containing junction traffic statistics aggregated in 15-minute intervals. The Avro schema is embedded within each delivered file, allowing standard Avro tools and libraries to read the data without additional configuration.

Data schema

Each Avro file contains records with the following structure:

BatchJunctionTraceStats (root record)

BatchApproachTraces (element of approaches)

BatchApproachTraceStats (element of traces)

BatchArrival (element of arrivals)

BatchControlStats (value of controlStats)

Directory structure

Files are organized in your S3 bucket by date and time in 15-minute intervals. The structure is identical for both pipelines:

1
<prefix>/<day>/<hour>/<quarter>/data.bin

Where:

• <prefix> - your configured prefix path (if provided).
• <day> - date identifier.
• <hour> - hour of the day (0-23).
• <quarter> - quarter-hour interval (0-3), corresponding to minutes 0-14, 15-29, 30-44, and 45-59 respectively.

Example file paths:

1
live/2026-04-30/14/2/data.bin
2
archive/2026-04-30/14/2/data.bin

The examples above show data for April 30, 2026, between 14:30 and 14:44 UTC, using live/ and archive/ prefixes respectively. If you subscribe to both pipelines, using separate prefixes like this is recommended to keep the data clearly separated.

The layout can differ per export. Your export can instead be configured with a dated layout, in which the day is split into year, month and day folders and the hour is zero-padded:

1
<prefix>/<yyyy>/<MM>/<dd>/<HH>/<quarter>/data.bin

For example archive/2026/04/30/14/2/data.bin. Exports with completion markers always use the dated layout. Your TomTom account representative confirms which layout your export uses.

Completion markers

On request (see completionMarkers under Prerequisites), the Historical (Archive) pipeline also writes an empty marker object into each day folder once the whole day has been delivered, so your downstream job knows when a day’s set of files is complete. A quarter-hour without traffic produces no data.bin, so a day does not always contain 96 files: the marker, together with a LIST of the day folder, gives you the exact set.

Markers are written only for exports with the dated layout described under Directory structure. Enabling markers on an export that uses the default layout switches it to the dated layout. Days delivered before the switch stay in the default layout; a re-delivery or reprocessing after the switch writes to the dated layout. The markers sit at the root of the day folder, next to the hour folders:

1
<prefix>/<yyyy>/<MM>/<dd>/_COMPLETE
2
<prefix>/<yyyy>/<MM>/<dd>/_COMPLETE_<taskId>

For example archive/2026/04/30/_COMPLETE for the day whose data files are under archive/2026/04/30/.

Properties of the markers:

• A marker is written only after all files of the day have been built and uploaded. The marker is an empty file, gzip-encoded like the data files (Content-Encoding: gzip), so its stored size is 20 bytes, not 0. Only its presence and its LastModified carry information. Its content type is text/plain.
• A day on which your junctions produced no data files gets no marker, neither _COMPLETE nor _COMPLETE_<taskId>. Do not wait for a marker on such a day.
• A reprocessing run writes only its own _COMPLETE_<taskId>; the plain _COMPLETE is not touched. A day that has only been reprocessed, and not yet delivered by the daily processing, therefore holds data files and run markers but no _COMPLETE. Do not treat a missing _COMPLETE as “not ready” for such a day.
• Run markers accumulate and are never removed: a day reprocessed several times carries several of them. Discover them by listing the day folder, which you already do to get the set of files.
• A _COMPLETE_<taskId> means the run finished for that day, not that it changed anything. A run that found nothing to change still leaves its marker, and then no record carries its id.
• When a day is re-delivered or reprocessed, its data.bin files are overwritten. Reprocess the whole day and replace your local state for it rather than ingesting only the records stamped with the task id: a reprocessing run does not delete records it no longer produces, and a changed junction geometry can move a pass to the neighbouring quarter-hour while the previous copy remains where it was.

Write semantics

• Files are written using the S3 PutObject operation (full replace, not append).
• If the same time interval is reprocessed, the corresponding file will be overwritten with updated data.
• Each upload is idempotent - uploading the same data twice produces the same result.
• If you need to preserve previous versions of files, enable S3 bucket versioning on your bucket.
• Completion markers, if enabled, are written only after all files of the day have been uploaded. The daily _COMPLETE marker is overwritten on re-delivery; a reprocessing run adds its own _COMPLETE_<taskId> marker instead.

Onboarding checklist

Follow these steps to set up the S3 Data Export integration:

1. Decide which pipeline(s) you need: Live, Historical (Archive), or both.
2. Create an S3 bucket or designate an existing one for receiving data.
3. If subscribing to both pipelines, decide on separate prefix paths to keep data organized (e.g. live/ and archive/).
4. Create a dedicated IAM user with s3:PutObject permission on the bucket.
5. Generate an Access Key ID and Secret Access Key for the IAM user.
6. Send the required information to your TomTom account representative:
   • Bucket name
   • AWS region
   • Access Key ID
   • Secret Access Key
   • Selected pipeline(s)
   • Prefix path(s) (if desired)
   • Whether you want completion markers for the Historical pipeline (if desired)
7. TomTom configures the export and confirms activation.
8. Verify that data is arriving in your bucket:
   • Live pipeline: data should appear approximately 30 minutes after each 15-minute interval ends.
   • Historical pipeline: data should appear after the next processing cycle (~6 hours).

Troubleshooting