GPT-3 发布

过去，为一个模型准备任务，通常要在训练集和代码里动手。2020 年的 GPT-3 把这项工作的一部分搬进了输入框。

论文描述了一个约 1750 亿参数的自回归 Transformer，在大规模网络文本混合语料上训练，上下文窗口 2048。评估被故意写成三种提示情形。零样本只有任务说明；单样本再给一个完整示例；少样本则把若干输入与答案依次写进上下文。模型读完这些文字后接着生成，不进行梯度更新。这里的「学习」发生在一次推理的上下文里：示例临时规定了答案格式、标签意义，甚至一种从未专门微调过的文字操作。

提示因此不再只是问题本身，也像一份微型数据集。论文报告 LAMBADA 少样本准确率约 86.4%；TriviaQA 从零样本约 64.3% 升到少样本约 71.2%；WebQuestions 从约 14.4% 到约 41.5%。翻译、文法纠正、新造词解释与简单代码补全，也被放进同一套 few-shot 协议里比较。模型规模表从约 1.25 亿排到 1750 亿，用以展示 few-shot 表现如何随参数量上升。

1750 亿让这种现象醒目，却没有让表现整齐。某些两位数算术可靠，位数增加后迅速出错；词义判断 WiC 接近随机；常识与对抗性推理仍明显落后于专门系统。示例顺序与措辞会改变分数——作者把上下文中究竟形成了怎样的内部任务，称为尚未解释的问题。训练语料的巨大范围带来另一层不确定：测试题可能以近似形式出现在网络数据中；污染分析有帮助，却无法为所有基准给出绝对排除。偏差章节留下一句准确而沉重的概括：互联网训练的模型具有互联网尺度的偏差。

OpenAI 没有发布 GPT-3 权重，而是通过 API 提供访问。提示既是操作模型的方法，也是产品接口：开发者不必拥有训练集群，甚至不必训练下游模型，就能用几行例子试出翻译器、分类器或写作工具。同一基础模型支撑大量应用的商业形态，在这里成形。少样本协议还改变了评测文化：同一模型可以用不同提示模板得到相差很大的分数，「提示工程」进入论文与产品讨论。

GPT-3 把任务定义的位置从训练管线移到一段可改写的文本里。这段文本便宜、灵活、脆弱；它让模型看似立刻领会要求，也让错误随着换序、改写或一道稍难的算术显现。论文里忽高忽低的表格，留下的是这种新接口最初的手感，而不是一张全能成绩单。上下文学习把示例预算从标注平台挪到提示词——预算很便宜，方差却很大。一千七百五十亿参数是放大器；放大器后面的接口形状——API 密钥与提示配置——才是产品史真正接手的部分。

Preparing a model for a task used to mean touching training sets and code. GPT-3, in 2020, moved part of that work into the input box.

The paper describes an autoregressive Transformer with about 175 billion parameters, trained on a large mix of web and other text, with a 2,048-token context. Evaluation is deliberately cast as three prompting regimes. Zero-shot gives only a task description; one-shot adds a single full example; few-shot stacks several input–answer pairs into context. The model continues the text without gradient updates. “Learning,” here, happens inside one forward context: examples temporarily define answer format, label meaning, even a textual operation never fine-tuned as such.

A prompt is therefore not only a question but a tiny dataset. The paper reports roughly 86.4% few-shot accuracy on LAMBADA; TriviaQA rising from about 64.3% zero-shot to 71.2% few-shot; WebQuestions from about 14.4% to 41.5%. Translation, grammar correction, invented-word explanation, and simple code completion enter the same few-shot protocol. A model-size table from about 125 million to 175 billion parameters shows few-shot gains rising with scale.

175 billion parameters make the phenomenon visible without making results tidy. Some two-digit arithmetic looks reliable; more digits fail quickly. WiC word-sense judgment sits near chance. Common-sense and adversarial reasoning lag specialized systems. Example order and wording move scores—the authors call what internal task forms in context an open question. The breadth of training text adds another uncertainty: test items may appear in approximate form online; contamination analyses help but cannot absolutely clear every benchmark. The bias discussion leaves a precise, heavy line: models trained on the internet have internet-scale bias.

OpenAI did not release GPT-3 weights. Access came through an API. Prompting became both a control method and a product interface: developers without training clusters—or even without training a downstream model—could trial translators, classifiers, or writing tools from a few examples. The business form in which one base model serves many applications takes shape here. Few-shot protocol also changed evaluation culture: the same model can score very differently under different prompt templates, so “prompt engineering” entered papers and products.

GPT-3 moved task definition from the training pipeline into rewritable text. That text is cheap, flexible, and fragile. It can make a model look immediately compliant and still break on a reordering, a rephrase, or a slightly harder sum. The paper’s uneven tables preserve the feel of a new interface, not a certificate of universal competence. In-context learning moved the example budget from annotation platforms into prompts—cheap budget, high variance. 175 billion parameters are an amplifier; the interface shape behind the amplifier—API keys and prompt configuration—is what product history actually inherited.