神经语言模型缩放规律被量化

给定有限的计算预算，钱该花在更大模型、更多数据，还是更长的训练上？

2020 年 Kaplan 等人的工作，标题几乎像一份实验备忘录：Scaling Laws for Neural Language Models。问题并不浪漫。团队在跨越约七个数量级的训练计算上拟合交叉熵损失，观察损失如何随非嵌入参数量 (N)、数据集大小 (D) 与训练计算量 (C) 变化。在作者检验的架构族与数据制度下，损失与这些规模变量之间呈现平滑的幂律趋势：当其他因素尚未成为瓶颈时，曲线可以外推，而不是在某个神秘阈值后突然断裂。

论文据此讨论计算最优的分配——在固定 (C) 下如何平衡 (N) 与训练 token 数——并给出当时制度内的经验公式。细节依赖具体的模型深度宽度比、优化器与数据清洗；它不是物理学常数，而是一张在给定实验网格里画出来的地图。实验覆盖的模型规模从很小的网络到数亿参数量级，训练 token 与计算量同步变化，损失在双对数坐标下近似呈直线。作者也讨论当数据有限时过拟合如何改变曲线，以及嵌入参数与非嵌入参数应分开计数。把这些条件写清，是为了防止把“幂律”误读成与架构、数据质量无关的万能公式。

地图的用处立刻超过了论文附录。研究经理与基础设施团队获得一种共同语言：损失下降可以提前粗算，集群采购与训练日程可以对照斜率讨论，而不是只靠“感觉还该再加卡”。更大的模型与更长的训练因此更容易通过预算审查——缩放从口号变成可写进规划表的假设。与此同时，边界也必须写清：幂律描述的是交叉熵等训练/验证损失趋势，不等于下游任务能力的单调保证；数据质量、对齐目标与推理时计算都不在原始曲线里。

后续工作很快改写了地图的某些等高线。DeepMind 的 Chinchilla 等研究表明，在相近计算下，更小模型配更多 token 可能优于“过度加大参数、数据吃不饱”的配置，修正了业界一度偏向堆参数的读法。Kaplan 原文与这些修正并不互相消灭：前者确立“规模变量可拟合、可外推”的工程习惯；后者调整在特定制度下什么叫算力最优。

对需要提前数月锁定芯片与电力的组织，曲线提供的是谈判语言：在何种参数量与数据量组合下，预期损失落在可接受区间。它不能回答产品是否有用、对齐是否成功、推理时额外计算是否值得。把缩放定律误当成“只需加大一切”的许可证，会在数据重复与评估失真时撞墙；把它当成可证伪的预算假说，则仍是训练规划里少有的定量工具。

把损失对计算量的曲线画在双对数纸上，斜率成为团队共享的直觉装置。研究生用它估计“再加十倍数据大概降多少损失”，经理用它解释为什么要订下一批芯片。误用出现在把交叉熵与用户满意度直接画等号，或在数据质量崩溃后仍盲信外推。缩放定律研究的成熟标志，不是斜率永远不变，而是社区学会标注定律的适用域与失效域。

缩放定律没有承诺无限进步。它承诺的是：在你还没有撞上数据重复、优化失败或评估失真之前，损失往往沿着可以画出来的斜率走。对需要提前数月预定 GPU 的组织来说，一条可争论的斜率，已经比一句“更大更好”有用得多。七个数量级上的平滑趋势，足以支持规划，不足以取消实验。曲线是地图，不是地形本身。

Given a finite compute budget, should you spend on a larger model, more data, or longer training?

Kaplan et al. (2020) titled their work like a lab memo: Scaling Laws for Neural Language Models. The question is unromantic. The team fit cross-entropy loss across roughly seven orders of magnitude of training compute, tracking how loss moved with non-embedding parameter count (N), dataset size (D), and training compute (C). Within the architecture family and data regime they tested, loss followed smooth power-law trends in those scale variables. When other factors were not yet bottlenecks, curves could be extrapolated rather than breaking after a mystical threshold.

From the fits the authors discussed compute-optimal allocation—how to balance (N) against training tokens at fixed (C)—and offered empirical formulas for that regime. Details depend on depth–width choices, optimizers, and data cleaning. These are not physical constants; they are a map drawn on a particular experimental grid. Models ran from very small nets to hundreds of millions of parameters, with tokens and compute moving in step; on log–log axes, loss looked approximately linear. The authors also discussed how limited data and overfitting bend the curves, and why embedding and non-embedding parameters should be counted separately. Writing those conditions down prevents reading “power law” as a universal formula independent of architecture and data quality.

The map’s usefulness outran the appendix. Research managers and infrastructure teams gained a shared language: loss drops could be rough-forecast, cluster purchases and training calendars argued against slopes instead of against a feeling that “we need more cards.” Larger models and longer runs became easier to defend in budget review—scaling moved from slogan toward a planning assumption. Boundaries still need writing down. Power laws describe trends in training or validation loss such as cross-entropy; they do not guarantee monotonic downstream capability. Data quality, alignment objectives, and test-time compute sit outside the original curves.

Later work redrew some contour lines. Results such as DeepMind’s Chinchilla showed that, at similar compute, smaller models with more tokens can beat parameter-heavy, data-starved configurations, correcting an industry reading that over-favored stacking parameters. Kaplan’s paper and these revisions do not annihilate each other. The former established the engineering habit that scale variables can be fit and extrapolated; the latter adjusted what “compute-optimal” means under a given regime.

For organizations that lock chips and power months ahead, the curves supply negotiation language: under which combinations of parameters and data does expected loss fall in an acceptable band. They cannot answer whether a product is useful, whether alignment succeeds, or whether extra test-time compute is worth it. Treating scaling laws as a license to “just scale everything” collides with data repetition and broken evaluation; treating them as a falsifiable budget hypothesis keeps one of the few quantitative tools in training planning.

Log–log plots of loss versus compute made slope a shared intuition device. Students use it to estimate how much loss another tenfold of data might buy; managers use it to explain why the next batch of chips is on order. Misuse appears when cross-entropy is equated with user satisfaction, or when extrapolation continues after data quality collapses. Maturity means labeling domains of validity and failure—not insisting the slope never moves.

Scaling laws do not promise endless progress. They promise that before data repetition, optimization failure, or broken evaluation intervene, loss often follows a slope one can draw. For organizations that reserve GPUs months ahead, a debatable slope is already more useful than “bigger is better.” Seven orders of smooth trend are enough to plan with and not enough to cancel experiment. The curve is a map, not the terrain.