S3 Bulk Data Export

⚠️ Deprecated — use the Data Export API instead

Direct S3 access with per-tenant AWS credentials is deprecated and kept only for existing integrations. New integrations must use the Data Export API, which delivers the same daily snapshot files with the same schema through OAuth 2.0 authentication and presigned download URLs — no AWS credentials required.

Existing S3 credentials continue to work for now, but plan your migration: the switch only replaces the listing/download mechanism, your file parsing and warehouse loading logic stays the same.

Steer publishes a daily snapshot of your tenant’s data to a dedicated Amazon S3 bucket. The payload schema matches the data model — the same columns, the same semantics — delivered as gzipped JSON files you can pull on your own schedule.

How It Works

1️⃣ Receive Credentials

Steer provisions a dedicated IAM user for your tenant and emails you the AWS access key and secret key.

2️⃣ List Your Prefix

Every object lives under s3://s3-steer-dwh-prod-export/{tenant_id}/. List it to discover what’s available.

3️⃣ Download & Decompress

Files are gzip-compressed JSON. Download, gunzip, and parse with any standard tooling.

Access & Credentials

Steer will send your AWS credentials to the email address on file during onboarding.

🔐 What You Receive

• AWS_ACCESS_KEY_ID — unique per tenant
• AWS_SECRET_ACCESS_KEY — store in a secret manager, never commit to source control
• Your tenant_id — the UUID that prefixes all of your objects (e.g. fb296814-1cd0-4417-b3ea-d76092b82841)

Granted Permissions

The IAM user tied to your credentials is scoped to read-only access on your tenant’s prefix only. The attached policy grants:

s3:ListBucket
s3:GetObject
s3:GetObjectVersion
s3:GetBucketLocation

No write, delete, or cross-tenant access is possible with these keys.

Key Rotation

Keys are rotated on request. If you suspect a key has been compromised, email Steer support immediately and we will issue a replacement and revoke the old credentials.

Bucket Layout

All objects follow a predictable path structure:

s3://s3-steer-dwh-prod-export/
└── {tenant_id}/
    ├── appointment/
    │   └── appointment_2026-04-16.gzip
    ├── campaign/
    │   ├── campaign_2026-04-15.gzip
    │   └── campaign_2026-04-16.gzip
    ├── customer/
    │   └── customer_2026-04-16.gzip
    ├── invoice/
    │   └── invoice_2026-04-16.gzip
    ├── message_directmail/
    ├── message_email/
    ├── message_text/
    ├── review/
    ├── shop/
    ├── survey/
    └── vehicle/

• Object key pattern: {tenant_id}/{data_type}/{data_type}_{YYYY-MM-DD}.gzip
• Case sensitivity: tenant_id values are case-sensitive in S3 keys — preserve the exact casing of the UUID you receive.
• Cadence: one file per data type, per day
• Contents: a full snapshot of the tenant’s data for that type as of the export time — not a delta
• Format: gzip-compressed JSON, schema-identical to the corresponding Analytics API endpoint
• Retention: objects are retained indefinitely; historical dates remain downloadable

Available Data Types

Folder	Schema match	Primary key	Updated-at column
appointment/	Appointment data model	STEER_APPOINTMENT_ID	STEER_UPDATED_AT
campaign/	Campaign data model	STEER_CAMPAIGN_ID	STEER_UPDATED_AT
customer/	Customer data model	STEER_CUSTOMER_ID	STEER_UPDATED_AT
invoice/	Invoice data model	STEER_INVOICE_ID	STEER_UPDATED_AT
message_directmail/	Message (direct mail) data model	STEER_MESSAGE_ID	STEER_UPDATED_AT
message_email/	Message (email) data model	STEER_MESSAGE_ID	STEER_UPDATED_AT
message_text/	Message (text) data model	STEER_MESSAGE_ID	STEER_UPDATED_AT
review/	Review data model	STEER_REVIEW_ID	STEER_UPDATED_AT
shop/	Shop data model	STEER_SHOP_ID	SHOP_UPDATED_AT
survey/	Survey data model	STEER_SURVEY_ID	STEER_UPDATED_AT
vehicle/	Vehicle data model	STEER_VEHICLE_ID	STEER_UPDATED_AT

Field-level definitions live on the individual Data Models pages in the sidebar (Customer, Vehicle, Invoice, Appointment, Campaign, Message, Review, Survey, and Shop).

Field availability varies by shop management system (SMS)

Some fields are populated by every SMS; others are SMS-specific or depend on Steer feature configuration. Treat all fields as nullable in your DWH ingest, and do not enforce NOT NULL on anything except the documented primary keys (STEER_*_ID) and STEER_TENANT_ID / STEER_SHOP_ID.

Quick Start

List objects

Replace the placeholder values with your credentials and tenant_id.

AWS_ACCESS_KEY_ID=YOUR_KEY \
AWS_SECRET_ACCESS_KEY=YOUR_SECRET \
aws s3 ls s3://s3-steer-dwh-prod-export/YOUR_TENANT_ID/ --recursive

Example output:

2026-04-15 14:23:51     201516 YOUR_TENANT_ID/campaign/campaign_2026-04-15.gzip
2026-04-16 11:53:43     201109 YOUR_TENANT_ID/campaign/campaign_2026-04-16.gzip
2026-04-15 14:24:04   47138770 YOUR_TENANT_ID/customer/customer_2026-04-15.gzip
2026-04-16 11:53:58   47174332 YOUR_TENANT_ID/customer/customer_2026-04-16.gzip
2026-04-15 14:26:57  563377780 YOUR_TENANT_ID/invoice/invoice_2026-04-15.gzip
2026-04-16 11:56:54  563957725 YOUR_TENANT_ID/invoice/invoice_2026-04-16.gzip
...

Download one day

AWS_ACCESS_KEY_ID=YOUR_KEY \
AWS_SECRET_ACCESS_KEY=YOUR_SECRET \
aws s3 cp \
  s3://s3-steer-dwh-prod-export/YOUR_TENANT_ID/invoice/invoice_2026-04-16.gzip \
  ./invoice_2026-04-16.gzip

Decompress and inspect. Note the non-standard .gzip extension — pass -S .gzip so gunzip accepts the suffix:

gunzip -k -S .gzip invoice_2026-04-16.gzip
head -c 500 invoice_2026-04-16

Python (boto3)

Configure credentials via environment variables, IAM role, or ~/.aws/credentials. Do not hardcode access keys.

import boto3
import gzip
import json

s3 = boto3.client("s3")  # picks up credentials from env / IAM / ~/.aws/credentials

BUCKET = "s3-steer-dwh-prod-export"
TENANT = "YOUR_TENANT_ID"
KEY = f"{TENANT}/customer/customer_2026-04-16.gzip"

obj = s3.get_object(Bucket=BUCKET, Key=KEY)
with gzip.GzipFile(fileobj=obj["Body"]) as gz:
    records = json.load(gz)

print(f"{len(records):,} customers")

Large files

The invoice/ dumps can exceed 500 MB compressed (multi-GB uncompressed). Stream-decompress with gzip.GzipFile rather than loading the full file into memory, and consider a streaming JSON parser (e.g. ijson) for row-by-row iteration.

Computing Deltas Between Two Days

Because each file is a full snapshot, clients compute deltas themselves. The recommended approach is to download two consecutive days and join on the data type’s primary key.

Strategy 1 — Filter by STEER_UPDATED_AT (fastest)

Every record carries an update timestamp. To find what changed on 2026-04-16, you only need the later snapshot:

import gzip, json
from datetime import datetime, timezone

with gzip.open("invoice_2026-04-16.gzip") as f:
    rows = json.load(f)

cutoff = datetime(2026, 4, 15, tzinfo=timezone.utc)
changed = [r for r in rows if datetime.fromisoformat(r["STEER_UPDATED_AT"]) > cutoff]
print(f"{len(changed):,} invoices changed since {cutoff.date()}")

This catches inserts and updates but not deletes.

Strategy 2 — Diff two snapshots on the primary key (complete)

When you need to detect deletes too, load both days and diff on the primary key:

import gzip, json

def load(path, pk):
    with gzip.open(path) as f:
        return {r[pk]: r for r in json.load(f)}

PK = "STEER_INVOICE_ID"
prev = load("invoice_2026-04-15.gzip", PK)
curr = load("invoice_2026-04-16.gzip", PK)

inserted = curr.keys() - prev.keys()
deleted  = prev.keys() - curr.keys()
updated  = {
    k for k in curr.keys() & prev.keys()
    if curr[k]["STEER_UPDATED_AT"] != prev[k]["STEER_UPDATED_AT"]
}

print(f"+{len(inserted)}  ~{len(updated)}  -{len(deleted)}")

See the primary-key table above for the correct PK column per data type.

Incremental warehousing

If you load these snapshots into Snowflake, BigQuery, or Postgres, a common pattern is: (1) truncate-and-replace for small types (shop, survey, message_text), and (2) MERGE on primary key + STEER_UPDATED_AT for the large ones (invoice, customer, vehicle, appointment).

Operational Notes

• Timezone: the date in the filename reflects the UTC date of export.
• Publish time: files typically land by ~12:00 UTC. If a given day’s file is missing, re-check later or fall back to the previous day’s snapshot — no partial files are published.
• No notifications: Steer does not currently emit S3 event notifications. Poll the prefix or schedule downloads against expected publish time.
• Versioning: object versions are preserved (s3:GetObjectVersion is granted), but in normal operation a given {data_type}_{date}.gzip key is written once.
• Missed-day recovery: if a daily file is missing past the publish window, contact Steer support — files can be regenerated within the retention window.

Troubleshooting

Symptom	Likely cause	Fix
AccessDenied on aws s3 ls	Wrong tenant_id in the path, or listing the bucket root	List exactly s3://s3-steer-dwh-prod-export/{your_tenant_id}/ (case-sensitive)
NoSuchKey for today’s file	Export hasn’t landed yet (before ~12:00 UTC)	Retry after the publish window, or use yesterday’s snapshot
InvalidAccessKeyId / SignatureDoesNotMatch	Credentials mistyped or rotated	Re-check env vars; request rotation via Steer support if needed
Python MemoryError parsing invoice_*.gzip	Loading the full file into memory	Use gzip.GzipFile + streaming parser (ijson)

Need help? Contact Steer support with your tenant_id and the S3 key you were trying to access.