RealFeedApp NanoGPT speedrun record improved from 39.9s to 6.7s
Retrieval-augmented n-gram hints enabled the model to reach GPT-2-level loss on an 8xH100 node in 6.7 seconds, surpassing all previous combined improvements since 2025.
- Field
- training language models
- What they did
- Researchers improved the NanoGPT Speedrun world record from 39.9 seconds to 6.7 seconds by using retrieval-augmented n-gram hints.
- Why it matters
- This improvement is larger than all prior improvements combined since 2025, allowing models to be trained much faster.
#nanogpt#n-gram#retrieval-augmented#training speed#gpt-2 loss
