AI21 发布 Jamba 混合架构模型

2024 年 3 月，AI21 Labs 发布 Jamba。这是一款在架构上不走寻常路的模型：它是首个将 Mamba 状态空间模型与 Transformer 注意力机制结合的大规模混合架构。对大多数用户来说，Jamba 可能只是又一个开源大模型；但对研究架构的人来说，这是一个重要的信号——「Transformer 之外」的路线正在走向规模化。

Jamba 要解决的问题很具体：成本。Transformer 的自注意力机制随序列长度平方增长，序列越长，推理越贵。状态空间模型（SSM）通过固定大小的隐状态把长序列处理成本压下来，但纯 SSM 在记忆与复杂推理上又有短板。Jamba 的思路是把两者拼接：用 SSM 处理大部分 token，在关键位置引入注意力，既保留效率又兼顾能力。

这个混合思路的效果是可见的。Jamba 在长上下文处理与推理吞吐上展现出效率优势，同时公开了权重，让社区可以直接检验。它不是实验室里的概念验证，而是一个可以跑起来的开源模型。对于需要处理超长文档、代码库的企业开发者来说，这种「长上下文不贵」的能力方向有着天然的吸引力。

Jamba 的影响不在于它本身有多强，而在于它开启了一扇门。它证明状态空间模型与注意力的混合不是学术幻想，而是可以落地的工程方案。此后的时间里，线性注意力、混合架构、乃至 400 万上下文的 MiniMax-01，都在这条「Transformer 之外」的延长线上生长。AI21 虽然不在大模型的第一梯队，却在这个方向上留下了自己的印记。

回看 Jamba，它像是一次清醒的偏航。当整个行业都在为更大的 Transformer、更多的注意力头而内卷时，AI21 选择问一个不同的问题：能不能花更少的钱处理更长的文本？这个问题的答案，最终演变成了一场关于「高效架构」的持久探索——而 Jamba，正是这条探索路径上最早被点亮的路标之一。

In March 2024 AI21 Labs released Jamba—a model that took an unconventional architectural path. It was the first large-scale hybrid combining the Mamba state-space model with Transformer attention. To most users, Jamba was just another open-source model; to architecture researchers, it was an important signal: the "beyond Transformer" route was going large-scale.

The problem Jamba addressed was concrete: cost. Transformer self-attention grows quadratically with sequence length—the longer the sequence, the more expensive the inference. State-space models (SSMs) use a fixed-size hidden state to compress long-sequence processing costs, but pure SSMs fall short on memory and complex reasoning. Jamba's idea was to splice the two: SSM for most tokens, attention at key positions, gaining efficiency while retaining capability.

The effect of this hybrid idea was visible. Jamba showed efficiency advantages in long-context processing and inference throughput, and it released weights so the community could test it directly. It was not a lab proof-of-concept but a runnable open model. For enterprise developers dealing with very long documents and codebases, this "long context without the price" direction held natural appeal.

Jamba's impact lies less in how strong it is and more in the door it opened. It proved that mixing state-space and attention was not academic fantasy but a feasible engineering plan. In the time since, linear attention, hybrid architectures, and even MiniMax-01 with its 4-million context all grew along this "beyond Transformer" extension line. AI21 may not be in the top tier of the model race, but it left its mark on this direction.

Looking back, Jamba reads like a deliberate detour. While the whole industry competed over bigger Transformers and more attention heads, AI21 chose to ask a different question: could longer text be processed for less money? The answer to that question grew into a sustained exploration of "efficient architecture"—and Jamba is one of the earliest signposts lit along that path.