Delivery and formats

Specter datasets are bulk deliveries of the same data the API serves, written to your own cloud storage on a schedule. They suit backfills, warehouse loads and analysis over the full corpus; the API suits per-record lookups and enrichment inside a product.

​

Destinations

We write to storage you own and control. Every dataset is available on both. If you need a destination that isn’t listed, contact support.

​

Formats

​

Datasets

​

File naming

Every delivery is named for the period it covers, so a run is reproducible from its filename alone. Larger datasets are split across numbered partitions. Read every partition under a period’s prefix as one logical delivery — the partition count changes as the data grows. News Signals is delivered as one file, and revenue signals are part of it — a News Signals delivery already includes them. A customer who takes revenue signals on their own receives one file too, carrying the revenue subset of the same rows. Either way the columns are identical; the filename says which you have.

​

Full and incremental deliveries

A full delivery is the entire dataset as of its period. Load it by replacing what you hold for that dataset. An incremental delivery carries only the records that changed since the previous one. Apply it by upserting on the dataset’s key — company_id for Company DB, and the equivalent identifier on the others. Records are never renumbered, so an identifier that arrived in a full delivery stays valid across every incremental that follows. Datasets marked ”—” above have no incremental stream; take the next full delivery.

​

How long a delivery stays

Every file is removed from your bucket 90 days after it lands. The window is rolling, so at any point you hold roughly the last three months of deliveries and nothing older. Load each delivery into your own storage as it arrives rather than treating the bucket as an archive. A dataset you have not loaded within 90 days has to be re-cut and re-sent, which is not immediate.

​

How changes are announced

A change that removes a field, renames one, or alters the shape of an existing one is published in the dataset changelog at least one month before the delivery that carries it, with the date it takes effect. New fields are additive and can arrive in any delivery without notice, so read by column name rather than by position, and ignore columns you do not recognise.

​

Loading a delivery

Nothing here is required — use whatever your stack already runs. These are the paths that need the least work. Nested fields (array<object> and object in the field reference) arrive as native Parquet structs. Most engines read them directly; flatten them with UNNEST in DuckDB, explode in Spark, or the equivalent in your engine.

​

Next