Inside the 48-Hour Validation Pipeline

The 48-hour promise is the hardest constraint in our stack. It is not a marketing number we round down to — it is a pipeline budget that every stage has to fit inside, with slack for retries, because customers schedule training runs against it.

The budget splits roughly three ways. Hours 0–8: intelligent analysis — stratified diversity sampling, collapse-signature matching against known failure patterns, and a pre-screening risk assessment that decides how much cascade the corpus needs. Hours 8–40: the multi-scale run itself, training 15–18 proxy models from 1M to 500M parameters. Hours 40–48: extrapolation, report assembly, row-level findings, and certificate issuance.

The proxy trainings are embarrassingly parallel until they are not: bigger proxies want the samplers’ outputs, extrapolation wants every proxy’s curves, and a straggling 500M run can eat the whole budget. The orchestrator treats the cascade like a DAG with per-stage deadlines — small proxies fan out wide and early, results stream into the extrapolation fitter incrementally, and a straggler past its deadline gets rescheduled onto fresh capacity rather than awaited politely.

Go is the right tool for this layer. The orchestrator is mostly goroutines, channels, and context deadlines around GPU jobs; the failure modes are timeouts and partial results, and Go makes both explicit. Compute-heavy stages call into the ML stack; coordination stays in one binary we can reason about.

The dashboard shows a staged pipeline view — queued, sampling, proxy training, extrapolation, report — with a percentage and an ETA. The rule is that this view must never be theater: stages map to real orchestrator states, and the ETA comes from observed stage durations, not a spinner with ambitions. If we would not bet on the number, we do not show the number.