S3 Bulk Data Export

⚠️ Deprecated — use the Data Export API instead

Direct S3 access with per-tenant AWS credentials is deprecated and kept only for existing integrations. New integrations must use the Data Export API, which delivers the same daily snapshot files with the same schema through OAuth 2.0 authentication and presigned download URLs — no AWS credentials required.

Existing S3 credentials continue to work for now, but plan your migration: the switch only replaces the listing/download mechanism, your file parsing and warehouse loading logic stays the same.

Steer publishes a daily snapshot of your tenant’s data to a dedicated Amazon S3 bucket. The payload schema matches the data model — the same columns, the same semantics — delivered as gzipped JSON files you can pull on your own schedule.

How It Works

1️⃣ Receive Credentials

Steer provisions a dedicated IAM user for your tenant and emails you the AWS access key and secret key.

2️⃣ List Your Prefix

Every object lives under s3://s3-steer-dwh-prod-export/{tenant_id}/. List it to discover what’s available.

3️⃣ Download & Decompress

Files are gzip-compressed JSON. Download, gunzip, and parse with any standard tooling.

Access & Credentials

Steer will send your AWS credentials to the email address on file during onboarding.

🔐 What You Receive

• AWS_ACCESS_KEY_ID — unique per tenant
• AWS_SECRET_ACCESS_KEY — store in a secret manager, never commit to source control
• Your tenant_id — the UUID that prefixes all of your objects (e.g. fb296814-1cd0-4417-b3ea-d76092b82841)

Granted Permissions

The IAM user tied to your credentials is scoped to read-only access on your tenant’s prefix only. The attached policy grants:

s3:ListBucket
s3:GetObject
s3:GetObjectVersion
s3:GetBucketLocation

No write, delete, or cross-tenant access is possible with these keys.

Key Rotation

Keys are rotated on request. If you suspect a key has been compromised, email Steer support immediately and we will issue a replacement and revoke the old credentials.

Bucket Layout

All objects follow a predictable path structure:

s3://s3-steer-dwh-prod-export/
└── {tenant_id}/
    ├── appointment/
    │   └── 2026-04-16/
    │       ├── appointment_0_0_0.json.gz
    │       └── appointment_0_1_0.json.gz
    ├── campaign/
    │   ├── 2026-04-15/
    │   │   └── campaign_0_0_0.json.gz
    │   └── 2026-04-16/
    │       └── campaign_0_0_0.json.gz
    ├── invoice/
    │   └── 2026-04-16/
    │       ├── invoice_0_0_0.json.gz
    │       ├── invoice_0_2_0.json.gz
    │       └── invoice_0_3_0.json.gz
    ├── customer/
    ├── message_directmail/
    ├── message_email/
    ├── message_text/
    ├── review/
    ├── shop/
    ├── survey/
    └── vehicle/

• Prefix pattern: {tenant_id}/{data_type}/{YYYY-MM-DD}/ — one prefix per day
• Case sensitivity: tenant_id values are case-sensitive in S3 keys — preserve the exact casing of the UUID you receive.
• Cadence: one snapshot per data type, per day, written as several objects under that day’s prefix
• Contents: the objects under one day’s prefix together form a full snapshot of the tenant’s data for that type as of the export time — not a delta
• Format: gzip-compressed newline-delimited JSON (NDJSON), schema-identical to the corresponding Analytics API endpoint
• Retention: objects are retained indefinitely; historical dates remain downloadable

List the day prefix, never build object names

The numbers in invoice_0_2_0.json.gz identify the process that wrote the object, not a position in a sequence. They are not contiguous — the tree above jumps from _0_0_0 to _0_2_0 — and the object count varies per tenant and per day. Always list {tenant_id}/{data_type}/{YYYY-MM-DD}/ and read every key it returns.

Available Data Types

Folder	Schema match	Primary key	Updated-at column
appointment/	Appointment data model	STEER_APPOINTMENT_ID	STEER_UPDATED_AT
campaign/	Campaign data model	STEER_CAMPAIGN_ID	STEER_UPDATED_AT
customer/	Customer data model	STEER_CUSTOMER_ID	STEER_UPDATED_AT
invoice/	Invoice data model	STEER_INVOICE_ID	STEER_UPDATED_AT
message_directmail/	Message (direct mail) data model	STEER_MESSAGE_ID	STEER_UPDATED_AT
message_email/	Message (email) data model	STEER_MESSAGE_ID	STEER_UPDATED_AT
message_text/	Message (text) data model	STEER_MESSAGE_ID	STEER_UPDATED_AT
review/	Review data model	STEER_REVIEW_ID	STEER_UPDATED_AT
shop/	Shop data model	STEER_SHOP_ID	SHOP_UPDATED_AT
survey/	Survey data model	STEER_SURVEY_ID	STEER_UPDATED_AT
vehicle/	Vehicle data model	STEER_VEHICLE_ID	STEER_UPDATED_AT

Field-level definitions live on the individual Data Models pages in the sidebar (Customer, Vehicle, Invoice, Appointment, Campaign, Message, Review, Survey, and Shop).

Field availability varies by shop management system (SMS)

Some fields are populated by every SMS; others are SMS-specific or depend on Steer feature configuration. Treat all fields as nullable in your DWH ingest, and do not enforce NOT NULL on anything except the documented primary keys (STEER_*_ID) and STEER_TENANT_ID / STEER_SHOP_ID.

Quick Start

List objects

Replace the placeholder values with your credentials and tenant_id.

AWS_ACCESS_KEY_ID=YOUR_KEY \
AWS_SECRET_ACCESS_KEY=YOUR_SECRET \
aws s3 ls s3://s3-steer-dwh-prod-export/YOUR_TENANT_ID/ --recursive

Example output:

2026-04-15 14:23:51     201516 YOUR_TENANT_ID/campaign/2026-04-15/campaign_0_0_0.json.gz
2026-04-16 11:53:43     201109 YOUR_TENANT_ID/campaign/2026-04-16/campaign_0_0_0.json.gz
2026-04-16 11:53:58   23604112 YOUR_TENANT_ID/customer/2026-04-16/customer_0_0_0.json.gz
2026-04-16 11:53:58   23570220 YOUR_TENANT_ID/customer/2026-04-16/customer_0_1_0.json.gz
2026-04-16 11:56:54  281978862 YOUR_TENANT_ID/invoice/2026-04-16/invoice_0_0_0.json.gz
2026-04-16 11:56:54  281978863 YOUR_TENANT_ID/invoice/2026-04-16/invoice_0_2_0.json.gz
...

Download one day

Copy the whole day prefix — a day is several objects:

AWS_ACCESS_KEY_ID=YOUR_KEY \
AWS_SECRET_ACCESS_KEY=YOUR_SECRET \
aws s3 cp --recursive \
  s3://s3-steer-dwh-prod-export/YOUR_TENANT_ID/invoice/2026-04-16/ \
  ./invoice_2026-04-16/

Decompress the day into one NDJSON file and inspect it:

cat ./invoice_2026-04-16/*.json.gz | gunzip -c > invoice_2026-04-16.json
head -n 1 invoice_2026-04-16.json

Python (boto3)

Configure credentials via environment variables, IAM role, or ~/.aws/credentials. Do not hardcode access keys.

import boto3
import gzip
import json

s3 = boto3.client("s3")  # picks up credentials from env / IAM / ~/.aws/credentials

BUCKET = "s3-steer-dwh-prod-export"
TENANT = "YOUR_TENANT_ID"
PREFIX = f"{TENANT}/customer/2026-04-16/"

records = []
for page in s3.get_paginator("list_objects_v2").paginate(Bucket=BUCKET, Prefix=PREFIX):
    for s3_object in page.get("Contents", []):
        obj = s3.get_object(Bucket=BUCKET, Key=s3_object["Key"])
        with gzip.GzipFile(fileobj=obj["Body"]) as gz:
            for line in gz:
                records.append(json.loads(line))

print(f"{len(records):,} customers")

Large snapshots

A day of invoice/ can exceed 500 MB compressed (multi-GB uncompressed) once its objects are combined. Stream-decompress with gzip.GzipFile and parse one line at a time rather than holding a whole day in memory.

Computing Deltas Between Two Days

Because each day is a full snapshot, clients compute deltas themselves. The recommended approach is to download two consecutive day prefixes and join on the data type’s primary key. Load every object of a day before comparing — a partially loaded day looks like mass deletions.

Strategy 1 — Filter by STEER_UPDATED_AT (fastest)

Every record carries an update timestamp. To find what changed on 2026-04-16, you only need the later snapshot:

import glob, gzip, json
from datetime import datetime, timezone

def load_day(day_directory):
    for path in sorted(glob.glob(f"{day_directory}/*.json.gz")):
        with gzip.open(path, "rt", encoding="utf-8") as f:
            for line in f:
                yield json.loads(line)

cutoff = datetime(2026, 4, 15, tzinfo=timezone.utc)
changed = [
    invoice for invoice in load_day("invoice_2026-04-16")
    if datetime.fromisoformat(invoice["STEER_UPDATED_AT"]) > cutoff
]
print(f"{len(changed):,} invoices changed since {cutoff.date()}")

This catches inserts and updates but not deletes.

Strategy 2 — Diff two snapshots on the primary key (complete)

When you need to detect deletes too, load both days and diff on the primary key:

import glob, gzip, json

def load_day(day_directory, pk):
    records = {}
    for path in sorted(glob.glob(f"{day_directory}/*.json.gz")):
        with gzip.open(path, "rt", encoding="utf-8") as f:
            for line in f:
                record = json.loads(line)
                records[record[pk]] = record

    return records

PK = "STEER_INVOICE_ID"
prev = load_day("invoice_2026-04-15", PK)
curr = load_day("invoice_2026-04-16", PK)

inserted = curr.keys() - prev.keys()
deleted  = prev.keys() - curr.keys()
updated  = {
    k for k in curr.keys() & prev.keys()
    if curr[k]["STEER_UPDATED_AT"] != prev[k]["STEER_UPDATED_AT"]
}

print(f"+{len(inserted)}  ~{len(updated)}  -{len(deleted)}")

See the primary-key table above for the correct PK column per data type.

Incremental warehousing

If you load these snapshots into Snowflake, BigQuery, or Postgres, a common pattern is: (1) truncate-and-replace for small types (shop, survey, message_text), and (2) MERGE on primary key + STEER_UPDATED_AT for the large ones (invoice, customer, vehicle, appointment).

Operational Notes

• Timezone: the date in the prefix reflects the UTC date of export. Object names carry no date.
• Publish time: a day’s objects typically land by ~12:00 UTC and are written together. If a day’s prefix is missing, re-check later or fall back to the previous day — no partial days are published.
• No notifications: Steer does not currently emit S3 event notifications. Poll the prefix or schedule downloads against expected publish time.
• Versioning: object versions are preserved (s3:GetObjectVersion is granted), but in normal operation a given day’s prefix is written once.
• Missed-day recovery: if a day’s prefix is missing past the publish window, contact Steer support — exports can be regenerated within the retention window.

Troubleshooting

Symptom	Likely cause	Fix
AccessDenied on aws s3 ls	Wrong tenant_id in the path, or listing the bucket root	List exactly s3://s3-steer-dwh-prod-export/{your_tenant_id}/ (case-sensitive)
Empty listing for today’s prefix	Export hasn’t landed yet (before ~12:00 UTC)	Retry after the publish window, or use yesterday’s prefix
NoSuchKey for an object name you built yourself	Object names cannot be derived — the numbers in them are writer ids with gaps	List {tenant_id}/{data_type}/{YYYY-MM-DD}/ and use the keys it returns
Record count far below expectations	Only some of the day’s objects were read	Read every key under the day prefix — a day is several objects
InvalidAccessKeyId / SignatureDoesNotMatch	Credentials mistyped or rotated	Re-check env vars; request rotation via Steer support if needed
Python MemoryError reading a day	Holding the whole day in memory	Stream with gzip.GzipFile and parse one line at a time

Need help? Contact Steer support with your tenant_id and the S3 key you were trying to access.