Human-like Neural Nets by Catapulting

Gwern's April 2024 speculative proposal: train massively overparameterised neural networks with deliberately high learning rates and strong regularisation to trigger "catapulting" / grokking — the phase transition where networks shift from memorisation to true generalisation. He argues that over-parameterisation, often treated as a wasteful side effect of modern AI, may in fact be the route to human-like generalisation, and that doing so deliberately could resolve outstanding mysteries about the gap between artificial and biological intelligence. Updated repeatedly through 2026.