One Corpus, Three Hundred Files: Why We Built Dataset Groups

Watch a real ML team prepare a corpus and you will almost never see one file. You will see train/valid/test splits, per-day exports, sharded parquet directories, WebDataset tars — one logical dataset wearing three hundred filenames. For a while, our data model pretended otherwise, and users paid the tax: upload thirty files, get thirty unrelated datasets, validate them one by one.

Dataset groups fix the model instead of the symptom. Drop a folder and the uploader ingests it recursively with per-file progress; name the group once, and every shard carries the membership. The group then appears as a first-class thing — file count, total size, status rollup — with a single validate action.

Group validation currently runs on the group’s largest ready file, with the report presented at group level — and the interface says exactly that, because implying full-group merging before the pipeline does it would be a lie with a UI. As merged-group validation lands in the backend, the same surfaces light up with the stronger semantics; the product contract was designed for that upgrade from day one.

This is a pattern we return to constantly: ship the workflow early, label the current semantics honestly, and let the capability deepen underneath without breaking anyone. Users get value now; nobody gets misled about what the number means.

The feature touched everything — upload initiation carrying group names, list views growing group chips, validation records carrying group identity into reports. Product work on infrastructure is mostly this: a one-sentence user need (“my dataset is a folder”) fanning out into a dozen precise contract decisions. The sentence is the easy part; the contracts are the product.