TD-Gammon 与时间差分学习

1992 年，IBM 的研究者 Gerald Tesauro 发表了一篇关于双陆棋的论文，实验结果让整个强化学习圈子震动。他用一个神经网络训练了一个下双陆棋的程序，取名叫 TD-Gammon。训练方法特别之处在于：它没有背过任何人类棋谱，没有人类高手指导，唯一的老师是它自己——与自己下棋，从输赢中学习。而这个自学成才的程序，棋力达到了接近人类大师的水平。

双陆棋是个狡猾的选择。棋盘上 15 枚棋子要按骰子点数移动，含有大量随机性，局面评估复杂到无法穷举。正是这种复杂度，让双陆棋成了检验学习算法的绝佳沙盘——如果程序能从这种混沌里自己摸出赢棋的门道，那「从经验学习」就得到了最硬的证明。TD-Gammon 的网络很小，输入是棋盘局面，输出是对当前玩家胜率的估计，核心是 TD(λ)——时间差分学习。

TD(λ) 的思路很特别：它不等到棋局结束才更新，而是每走一步就根据「当前预测」与「下一步预测」的差异来调整网络权重。预测之间的差距，本身就是学习信号。程序从随机权重起步，与自己对弈成千上万局，逐步把「对局面的直觉」雕刻进网络的参数里。没有外部奖励函数的人为设计，只有「预测要对齐未来」这一条朴素原则。

结果令人惊讶。TD-Gammon 不仅学会了基本棋理，还发展出了人类专家认可的高级策略。在 1992 年的版本之后，Tesauro 给网络加了更多隐藏单元、延长训练，棋力进一步提升——有分析认为已经接近世界级水平。它偶尔的失误也被研究者认真分析，因为它暴露了纯自学的盲区。更重要的是，它证明了：在足够复杂的博弈里，纯粹的自我对弈足以逼近顶尖技能，不需要人类知识作拐杖。

这个结论的影响远远超出双陆棋。2016 年 AlphaGo 击败李世石时，DeepMind 团队明确把自我对弈、价值网络、从经验学习列为核心思想——这些思路的谱系里，TD-Gammon 是一个绕不开的名字。后来的 AlphaZero 干脆连人类棋谱都不要了，完全从零开始自我对弈，把 TD-Gammon 的哲学推到了极致。从 1992 年的双陆棋，到 2017 年的国际象棋与围棋，自我对弈这条线贯穿了整个强化学习的高光时刻。

回看 TD-Gammon，它的意义被很多人概括为一句话：机器可以不靠老师，只靠对手是自己，就学到顶尖技能。这句话今天听起来平常，但在 1992 年，它是一个让人怀疑又不得不信的实验结果。那个在 IBM 实验室里与自己下了无数盘棋的小神经网络，用最朴素的方式预告了后来 AI 世界最壮观的景象之一——机器的学习，可以始于一张白纸。

In 1992 IBM researcher Gerald Tesauro published a paper about backgammon whose experimental result shook the reinforcement-learning community. He had trained a neural network to play backgammon and named it TD-Gammon. The remarkable thing about the training: it had studied no human game records and received no coaching from grandmasters. Its only teacher was itself—playing against itself and learning from wins and losses. And this self-taught program played near human master level.

Backgammon was a cunning choice. Fifteen checkers move by dice rolls, randomness is everywhere, and position evaluation is too complex for exhaustive search. That complexity made backgammon a perfect sandbox for testing learning algorithms—if a program could find winning skill in that chaos by itself, then "learning from experience" had gotten its hardest proof. TD-Gammon's network was small: input was the board position, output an estimate of the current player's win probability, and the engine was TD(λ)—temporal-difference learning.

The TD(λ) idea is unusual. Instead of waiting for the game to end, the network updates its weights at every move, using the difference between the current prediction and the next-step prediction. The gap between successive predictions is itself the learning signal. Starting from random weights, the program played thousands and thousands of games against itself, gradually carving "intuition about positions" into its parameters. No hand-designed reward function—just the plain principle that predictions should align with the future.

The results surprised everyone. TD-Gammon not only learned basic strategy; it developed advanced tactics that human experts recognized. After the 1992 version, Tesauro added more hidden units and longer training, pushing skill further—analyses suggested it approached world-class level. Its occasional blunders were studied carefully, because they revealed the blind spots of pure self-learning. More importantly, it proved that in a sufficiently complex game, pure self-play can approach top skill without human knowledge as a crutch.

The influence reached far beyond backgammon. When AlphaGo beat Lee Sedol in 2016, the DeepMind team explicitly named self-play, value networks, and learning from experience as core ideas—within that lineage, TD-Gammon is an unavoidable name. AlphaZero later dropped human games entirely and trained from scratch by self-play, pushing TD-Gammon's philosophy to its extreme. From 1992's backgammon to 2017's chess and Go, self-play runs like a thread through the brightest moments of reinforcement learning.

In hindsight, TD-Gammon's meaning is often compressed into one line: a machine can reach top skill with no teacher other than itself as opponent. That sentence sounds ordinary today, but in 1992 it was a result people doubted until they had to believe it. The small network playing countless games against itself in an IBM lab was, in the plainest way, a preview of one of the most spectacular sights the AI world would later see—machine learning that begins from a blank slate.