Handling Data We’re Paid to Distrust: Integrity Across the Synthos Pipeline

Synthos’ pipeline has an unusual security posture: the primary input is arbitrary files from strangers, at sizes up to 500GB, in more than thirty formats spanning tabular data, text corpora, images, audio, video, and packed archives. Every one of those files is simultaneously a customer’s crown jewels and untrusted input.

The first principle is that raw bytes never touch our application servers. Uploads go directly to object storage over time-limited signed URLs; the API sees metadata and completion callbacks, never the stream. That single decision removes an entire class of parser-facing attack surface from the serving path.

The bytes do eventually get parsed — that is the product. Parsing happens in the validation workers, isolated from serving infrastructure, on the assumption that a malicious parquet file or a zip-of-doom will eventually arrive. Format allowlists are enforced at both the client and the initiation endpoint, size and emptiness checks run before any signed URL is minted, and archives are treated with particular suspicion.

Integrity is tracked from the moment of upload: ETags on completion, dataset state machines that cannot skip from uploading to ready without passing processing, and an audit trail of who touched what. When a validation later certifies that dataset, the certificate is anchored to the exact artifact we inspected.

Trust also means letting go properly. Deleting a dataset archives its records and removes access immediately, warranty and certificate references are preserved for accountability without retaining the data itself, and our retention behavior is written into the privacy policy rather than folklore. A platform that judges data quality gets judged on exactly these details — as it should be.