Uploading 500GB Without Tears: Signed URLs, Resumable Transfers, and Folder Ingestion

Our upload ceiling is 500GB per file, and real corpora routinely ship as folders of hundreds of shards — per-split CSVs, WebDataset tars, image directories. Getting that reliably from a customer’s laptop or CI runner into object storage is a product problem disguised as plumbing.

The core design is boring on purpose: the API mints a time-limited signed URL, the client PUTs bytes directly to storage, and a completion call with the ETag flips the dataset’s state machine. Application servers never proxy payload bytes, which keeps them small, cheap, and out of the failure path.

At these sizes, a transfer that cannot survive a dropped connection is a toy. Upload state persists client-side so an interrupted transfer resumes instead of restarting; chunked writes retry with exponential backoff and refuse to retry on client errors; and a paused upload is a first-class state in the UI, not an error.

Folder ingestion walks dropped directories recursively, filters to the supported formats — tabular and structured files, text corpora, images, audio, video, embedding arrays, packed archives — skips hidden and empty files with an honest count of what was skipped, and pushes everything through a bounded-concurrency queue with per-file progress. Files can be tagged into a named dataset group at upload, so a sharded corpus stays one logical thing downstream.

Two details matter more than any throughput graph. First, per-file progress is real progress from the byte stream, with speed and remaining-time from observed rates. Second, when something is skipped or capped — unsupported extension, empty file, batch limits — the UI says so with numbers. Silent truncation is how you convert an upload bug into a data-quality incident, which for this company would be a self-own of historic proportions.