Benchmark RadarOmniOpt
A cross-domain benchmark for comparing optimizers in large-scale model training. It covers 24+ optimizers across two stages: Stage 1 sweeps on C4 with LLaMA-3 architectures (60M to 1B), and Stage 2 transfers to FineWeb-Edu with four architectures (Transformer++, GLA, DeltaNet, Gated DeltaNet) at 340M and 1B scales. Controlled-variable protocol with fixed architecture, data, and schedule settings.
- Released
- 2026-07-04
- Readiness
- Runnable
- Primary field
- General AI
Why it matters
Optimizer selection is a system-level decision impacting compute, memory, and tuning budget. This benchmark provides a unified protocol for comparing methods across multiple scales and architectures, offering reproducible evidence for practitioners to choose optimizers based on measured training objectives and trade-offs.
Motivation
Optimizer selection for large-scale model training has become a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity, yet the landscape of over one hundred methods remains fragmented.
Primary resources
Benchmark Radar records only publicly supported details and links back to primary sources for verification.