Transformer 架构论文发表

标题写着「注意力就是你所需要的一切」。翻到图 1，画面却挤满别的零件：多头注意力旁边是逐位置前馈网络，外面绕着残差连接与层归一化，底部还垫着位置编码。

2017 年，Vaswani 等人把这套结构叫作 Transformer。它不是一个孤独的注意力算子，而是一套去掉循环与卷积之后，仍能完成机器翻译的完整编码器—解码器。

在此之前，RNN 与 LSTM 必须按时间步往前推。序列一长，训练就很难并行；远处的词要穿过一长串中间状态才能碰头。注意力机制其实早已出现在神经机器翻译里，多半挂在循环编码器上，充当对齐模块。Transformer 把自注意力抬成主干：每个位置用查询、键、值做缩放点积注意力，直接读取序列里其他位置，理论路径长度被压到常数级。多头把投影切成若干子空间并行计算，再拼回原宽度。基础模型用 8 个头、模型维度 512、前馈内层 2048，编码器与解码器各 6 层。

自注意力本身对排列不敏感，顺序必须另补。作者把正弦与余弦位置编码加在词嵌入上，让模型知道「第几个词」，却不必重新引入循环。解码器侧保留掩码，防止看到未来词；编码器—解码器注意力则让目标位置去查源句表示。

验收场景是 WMT 2014 英德与英法翻译。论文报告 base 与 big 配置在英德上分别约 27.3 与 28.4 BLEU，英法 big 模型约 41.8 BLEU；big 模型在 8 块 P100 上大约训练 3.5 天，相对此前许多强系统更省。论文里还有一张对照表：自注意力每层复杂度随序列长度平方增长，但串行步数为常数，远距离依赖的路径长度也是常数。表格比口号更能解释「为什么值得拿掉循环」。标签平滑、学习率预热与 Adam 变体等训练细节写在正文与附录里——「全是注意力」并不等于「没有训练诀窍」。

二次复杂度是同一设计的影子。序列长度为 (n) 时，时间与内存大致随 (n^2) 涨。后来的长文档、长上下文研究，许多是在稀疏、线性近似或分层注意力上还这笔债。架构本身却极好拆：只留编码器，可以做 BERT 式双向理解；只留解码器，可以做 GPT 式自回归生成。2017 年的论文写的是翻译系统；工业界读到的，是一块能在 GPU 与 TPU 上铺开的通用序列积木。

「All You Need」是修辞压缩。真正被需要的，是注意力与前馈、残差、归一化、位置信息一起工作时的可并行训练性质。循环被拿掉之后，序列模型终于和矩阵硬件站在同一侧。翻译分数会过时；并行训练的亲和性，定义了此后十年序列模型与硬件的关系。

The title says attention is all you need. Figure 1 still draws everything else: multi-head attention beside position-wise feed-forward networks, residual connections and layer normalization wrapping both, positional encodings at the bottom.

In 2017, Vaswani and colleagues called this stack the Transformer. It is not a lone attention operator. It is a full encoder-decoder that translates without recurrence or convolution.

RNNs and LSTMs advance one time step after another. Longer sequences are hard to train in parallel, and distant dependencies must travel through long chains of intermediate states. Attention already existed in neural machine translation, usually as an alignment module on top of recurrent encoders. The Transformer promotes self-attention to the backbone: each position reads other positions directly through scaled dot-product attention over queries, keys, and values, reducing theoretical path length to a constant. Multi-head attention splits the projections into several subspaces, computes them in parallel, and concatenates them back. The base model uses eight heads, model dimension 512, feed-forward inner width 2048, and six layers each in the encoder and decoder.

Because self-attention is permutation-invariant on its own, order must be supplied separately. The authors add sinusoidal positional encodings to token embeddings so the model can distinguish position without recurrence. The decoder keeps causal masking so a position cannot attend to future tokens; encoder-decoder attention lets target positions query source representations.

The acceptance tests were WMT 2014 English–German and English–French. The paper reports roughly 27.3 BLEU for the base English–German model and 28.4 for the big configuration, about 41.8 BLEU for the big English–French model, and notes that the big model trained in about 3.5 days on eight P100 GPUs—lower cost than many strong systems of the period. A complexity table compares self-attention with recurrence and convolution on compute, sequential steps, and path length: attention is quadratic in length but constant in sequential depth and constant in path length for distant pairs. That table explains the bet better than the slogan. Optimization details—label smoothing, learning-rate warmup, Adam variants—are written down so “attention is all you need” is never mistaken for “no training craft.”

Quadratic cost is the design’s shadow: for sequence length (n), self-attention time and memory scale roughly as (n^2). Much later long-context work pays that debt with sparsity, linear approximations, or hierarchy. The architecture itself splits cleanly. Encoder-only stacks become BERT-style bidirectional models; decoder-only stacks become GPT-style autoregressive generators. The 2017 paper describes a translation system. Industry read a general sequence block that spreads efficiently across GPUs and TPUs.

“All You Need” is rhetorical compression. What was needed was the training property of attention working together with feed-forwards, residuals, normalization, and positional signals. Once recurrence was gone, sequence models finally stood on the same side as matrix hardware. Translation BLEU ages; parallel training affinity defined the hardware relationship of sequence models for a decade.