Musk Praises Kimi Team’s Attention Residuals Paper Co-Authored by 17-Year-Old Student

Large language models commonly use PreNorm residual connections, which add the output of each layer to a running total. As models deepen, hidden states can continue to grow, diluting new information from individual layers. The Kimi team at Chinese AI company Moonshot AI proposed Attention Residuals, or AttnRes, which instead uses softmax to dynamically select representations from earlier layers, improving information flow and computational efficiency in deep models.

The Kimi team published the paper on March 16, 2026, and incorporated AttnRes into Kimi Linear. A model with 48 billion total parameters and 3 billion active parameters was pretrained on 1.4 trillion tokens and outperformed the baseline across all evaluations. Elon Musk wrote “Impressive work” on X the same day. Among the 37 authors, 17-year-old Shenzhen high school student Chen Guangyu was an equal-contribution co-first author alongside Zhang Yu and Su Jianlin.