Eighteen Small Models, One Big Answer: Our Extrapolation Stack

A Synthos validation is, mechanically, an experiment: train 15–18 models of increasing size on carefully drawn samples of the candidate corpus, measure each quality dimension at each scale, and fit the trend. The product is a prediction about a model none of us will train — with error bars we stand behind.

Everything downstream depends on the sampling. Stratified diversity sampling has to preserve exactly the properties we are testing for — tail mass, rare classes, temporal structure — because a sampler that quietly drops outliers would blind the entire cascade to outlier problems. We validate our samplers with the same paranoia customers should apply to their generators.

Per-dimension metrics across the cascade form scale curves, and the shape of each curve carries the signal. Healthy data yields the smooth, decelerating improvements scaling-law work would predict. Collapse-prone data bends early — the curve flattens where it should still be climbing, or dimensions decouple, with loss improving while distributional fidelity stalls.

The fit produces the headline risk score, a collapse probability, and per-dimension trajectories, each with confidence intervals derived from cross-cascade variance. Wide intervals are reported as wide; the warranty program means an overconfident interval is not a cosmetic bug but a financial one, and that constraint has shaped our fitting choices more than any benchmark.

The whole cascade must fit inside the pipeline’s 32-hour training window, which turns proxy training into an optimization discipline of its own: aggressive early stopping once a curve’s contribution has converged, shared preprocessing across scales, and schedules tuned so the marginal proxy adds information rather than ceremony. The interesting constraint is never “can we train it” — it is “does the eighteenth model change the answer.” When it stops doing so, we stop adding models.