InstructGPT 用人类反馈对齐指令

2022 年 1 月，OpenAI 公开 InstructGPT 论文时，语言模型的主流训练目标仍是预测下一个词。续写可以流畅，却不天然知道用户是在提问、要求改写，还是在试探危险内容。

大约四十名标注员接上了另一条生产线。他们先写答案：面对来自 OpenAI API 和标注员自拟的指令，示范一个理想回应。论文列出的监督训练数据包括 11,295 条标注员提示和 1,430 条客户提示；研究者用这些示范对 GPT-3 做监督微调。模型由此先学会一种基本姿态：输入不一定是等待续写的文章，也可能是需要执行的要求。

接着，人类不再写唯一答案，而是比较。系统为同一提示生成四到九个候选，标注员排序；研究者用这些比较训练奖励模型，估计人更偏好哪种输出。最后，PPO 以奖励模型为评分者继续优化语言模型，同时混入预训练目标，减轻某些公共 NLP 任务上的性能退化。这条「监督微调 → 奖励模型 → 强化学习」的三阶段结构，后来被无数对齐与指令微调工作复述或改写；InstructGPT 把它写成了可核对的实验账本。

生产线改变了「更好」的含义。预训练优化下一个 token 是否更可能出现；InstructGPT 优化的是一组人面对完整回答时的选择。结果很醒目：在论文的 API 提示测试集上，1.3B 参数的 InstructGPT 输出比原始 175B GPT-3 更受标注员偏好。175B InstructGPT 与同尺寸 GPT-3 相比，胜率为 85±3%；即使给后者少样本提示，前者仍有 71±4% 的偏好率。在封闭域任务里，编造输入之外信息的比例从 GPT-3 的 41% 降到 21%。

这些比较绑定于论文的提示分布、标注规范与评价者。它们证明偏好可以被训练，不等于模型已经真实可靠。InstructGPT 仍会捏造事实、误解带有错误前提的问题，也会用很长的保留语回答简单问题。毒性有所下降，论文没有观察到偏差指标的显著改善。为了讨好奖励模型而牺牲原有任务能力的现象，被作者称为 alignment tax；混合预训练梯度可以缓解，却不能把所有回退抹掉。

更重要的限定写在讨论部分。研究者说，他们对齐的是特定标注员与研究人员影响下的偏好，不是笼统的「人类价值」。标注规范由 OpenAI 制定，训练数据以英语为绝大多数，标注员之间的一致率约为 73%。换一批人、换一种文化背景、换一套规则，「好回答」的边界可能移动。论文还把 API 用户提示与标注员自拟提示区分开，提醒训练分布并不等于「全人类指令」。

ChatGPT 会把类似方法推到产品级数据飞轮，但飞轮的第一圈齿轮，仍是这大约四十人与三阶段流水线。InstructGPT 把一个新问题带到模型核心：能力之外，回答方式也可以被训练。直接、诚实、谨慎、拒绝哪些要求，都不再只是提示技巧，而成为数据与优化目标的一部分。在 2022 年初的论文里，每一个「更符合人意」背后，仍站着有限的人、有限的规范，以及必须追问的一句——符合谁的意。

When OpenAI published the InstructGPT paper in January 2022, the dominant training objective for language models was still next-token prediction. Completions could be fluent without knowing whether a user was asking a question, requesting a rewrite, or probing for harmful content.

Roughly forty labelers staffed another production line. First they wrote answers. Given prompts from the OpenAI API and prompts the labelers invented, they demonstrated ideal responses; researchers supervised-fine-tuned GPT-3 on those demonstrations. The paper lists 11,295 labeler prompts and 1,430 customer prompts in the supervised data. The model learned a basic stance: input need not be an unfinished document; it can be a request to execute.

Then humans stopped writing a single gold answer and began comparing. The system sampled four to nine candidates for the same prompt; labelers ranked them. Researchers trained a reward model to estimate which outputs people preferred. Finally, PPO optimized the language model against that reward model while mixing in a pretraining objective to limit regression on some public NLP tasks. The three-stage structure—supervised fine-tuning, reward model, reinforcement learning—would be retold and partly rewritten across later alignment work. InstructGPT wrote it as a checkable experimental ledger.

The line changed what “better” meant. Pretraining optimizes whether the next token is likely; InstructGPT optimizes choices a group of people make over full answers. Results were striking: on the paper’s API-prompt test set, a 1.3B InstructGPT was preferred to the original 175B GPT-3. Against same-size GPT-3, 175B InstructGPT won 85±3% of comparisons; even with few-shot prompts for the baseline, it still won 71±4%. On closed-domain tasks, inventing information outside the input fell from 41% for GPT-3 to 21%.

Those comparisons are bound to the paper’s prompt distribution, labeling rules, and raters. They show preference can be trained; they do not show the model has become factually reliable. InstructGPT still fabricated facts, mishandled false premises, and answered simple questions with long hedges. Toxicity fell; bias metrics did not show clear improvement. Authors called the sacrifice of some capabilities to please the reward model an “alignment tax”; mixing pretraining gradients could soften it without erasing every regression.

The sharper limit sits in the discussion. Researchers wrote that they aligned to preferences shaped by particular labelers and research staff—not to “human values” in general. OpenAI wrote the labeling instructions; English dominated the data; inter-annotator agreement was about 73%. Another group, culture, or rulebook could move the border of a “good” answer. The paper also separates API-user prompts from labeler-written prompts, reminding readers that the training distribution is not “all human instructions.”

ChatGPT would later scale similar methods into a product flywheel; the first gear remains roughly forty people and a three-stage line. InstructGPT brought a new question into the model core: beyond capability, manner of answer can be trained. Directness, honesty, caution, and which requests to refuse stop being only prompt craft and become data and objectives. In early 2022, every “more preferred by people” still stood on finite people, finite rules, and a question that cannot be skipped: preferred by whom?