OpenAI's Original Scaling Law Had a Bug That Led the Industry to Build Oversized, Undertrained Models for Two Years
Former OpenAI researcher Diogo Almeida published a detailed breakdown of the bug in OpenAI's original 2020 scaling laws, confirming that a combination of fixed training data, an aggressive learning rate decay, and a misleading independence claim caused the entire AI industry to build oversized, undertrained models until DeepMind's Chinchilla correction in 2022.