GPT-2 展示零样本任务迁移

2019 年 2 月，OpenAI 公布 GPT-2：约 15 亿参数的 Transformer 语言模型，在 WebText 上训练——大约 800 万份从出站链接抓取的网页文档。作者刻意避开以维基百科为主的常见爬取捷径，以减少与某些下游评测的重叠。同一家族还有 117M、345M、762M 等较小变体；最大的 1.5B 版本演示了在不进行任务微调时，仅靠提示完成基础问答、摘要、翻译与续写。

「零样本」在这里有操作定义：不更新权重，不提供梯度步，只把任务说明与输入写进上下文，让模型继续往下生成。效果参差——有的样例流畅得令人不安，有的则胡编乱造——但方向已经清晰：足够大的通用语言建模，可以把一部分原先需要标注集与微调脚本的任务，收成提示。与 GPT-1 相比，重点从「预训练后微调」滑向「预训练后直接尝试」。

与此同时，OpenAI 决定不立即放出最大模型的权重，理由是恶意生成、虚假信息等滥用风险；先开放小模型与论文，再在约九个月后的 2019 年 11 月分阶段放完 1.5B 检查点。技术发布变成公共争论：有人认为是负责任的披露，有人认为是营销式的饥饿管理，也有人指出能力边界仍远未达到「全面伪造现实」。争论的价值不在站队口号，而在它第一次让「强大语言模型的权重是否应立即公开」成为广泛议题。

WebText 的构建本身是方法的一部分：从 Reddit 等高质量出站链接收集文档，过滤后形成约 40GB 文本；上下文长度 1024，使用 BPE 词表。训练数据来自网页，偏差、攻击性内容与事实错误一并进入分布——模型会流畅地错。完整权重最终仍然公开，社区得以复现、蒸馏与继续微调，也为下一阶段准备了对照：开放权重与托管接口，从此是两条可以分叉的路。

15 亿参数很快不再稀奇。稀奇的是同一年里并行发生的两件事：语言建模开始被当作多任务接口来展示，以及权重发布本身变成需要论证的动作。从拒发到放完的九个月，是权重治理讨论的压缩教程——能力展示、风险声明与社区复现必须读成同一事件，而不是三条孤立新闻。

In February 2019 OpenAI presented GPT-2: a Transformer language model with about 1.5 billion parameters, trained on WebText—roughly eight million web documents gathered from outbound links. The authors deliberately avoided a Wikipedia-heavy crawl to reduce overlap with some downstream evaluations. Smaller variants at 117M, 345M, and 762M parameters accompanied the largest model, which demonstrated basic question answering, summarization, translation, and continuation without task-specific fine-tuning.

Zero-shot here has an operational meaning: no weight updates, no gradient steps—only a task description and input in context, with the model continuing the text. Results were uneven. Some samples were fluently unsettling; others fabricated freely. The direction was clear enough: large-scale general language modeling can fold some work that once required labeled sets and fine-tuning scripts into prompting. Relative to GPT-1, emphasis slid from “pretrain then fine-tune” toward “pretrain then try directly.”

OpenAI also chose not to release the largest weights immediately, citing misuse risks such as malicious generation and disinformation. Smaller models and the paper came first; the full 1.5B checkpoint followed in stages, completing around November 2019 about nine months later. The decision turned a technical release into a public argument. Some called it responsible disclosure; others called it marketing by scarcity; still others noted that capability remained far from “total forgery of reality.” The lasting value of the fight is less any slogan than the fact that “should powerful language-model weights ship immediately?” became a broad question for the first time.

WebText’s construction was part of the method: outbound links from high-quality sources, filtered into about 40GB of text, with a 1,024-token context and a BPE vocabulary. Web training data carried bias, abusive content, and factual error into the distribution—the model can be wrong fluently. Full weights did eventually open, enabling reproduction, distillation, and further fine-tuning, and setting up the next contrast: open weights versus hosted APIs as forkable paths.

1.5 billion parameters did not stay rare for long. What was rare was the coincidence of two events in one year: language modeling shown as a multi-task interface, and weight release itself treated as an action that required argument. The nine months from refusal to full release compressed a tutorial in weight governance—capability demos, risk statements, and community reproduction must be read as one event, not three isolated headlines.