Attention Is All You Need, without the mythology

Attention begins with three learned representations: queries, keys, and values. The model compares queries with keys, scales the dot products by the square root of the key dimension, applies a softmax, and uses the resulting weights to combine values. In compact form, the paper writes this as softmax(QKᵀ / √dₖ)V.

Multi-head attention repeats that operation in different learned projection spaces, then combines the results. The base Transformer used eight heads, a model width of 512, and a position-wise feed-forward inner dimension of 2,048. Because attention itself does not encode order, the paper added sinusoidal positional encodings to the input representations.

A head does not come with a fixed human-readable job. Each head receives learned projections and can develop a different pattern when that helps the training objective. Multiple heads increase the model's opportunity to represent several relationships at once, but inspecting a weight pattern is not automatically a complete explanation of why the model produced an answer.

The 2017 design used a stack of six encoder layers and six decoder layers. Each encoder layer combined multi-head self-attention with a feed-forward network. Each decoder layer added masked self-attention and attention over the encoder output. Residual connections and layer normalization surrounded the sublayers.

Masking in the decoder prevented a position from attending to later output positions during autoregressive generation. That causal constraint remains conceptually important in later decoder-only language models, even though their overall architecture is not identical to the original translation system.

The encoder and decoder had different information roles. The encoder built contextual representations of the entire source sentence. The decoder generated the target one position at a time, using its previous outputs and cross-attention to the encoded source. This is why the original architecture diagram has two towers rather than one repeated stack.

An input token begins as an embedding that represents its identity, combined with positional information. Within a self-attention layer, that position creates a query and compares it with keys from every allowed position. The weighted value vectors form a new representation shaped by the surrounding sequence.

The feed-forward network then transforms each position independently using the same learned function. Attention moves information between positions; the feed-forward sublayer processes the information at each position. Stacking these operations lets later layers work with representations that already contain contextual evidence.

During training, the full target sequence can be processed in parallel because the causal mask blocks forbidden future connections. During generation, however, the system still produces output autoregressively: it predicts a token, appends it to the prefix, and repeats. Parallel training and sequential generation are therefore compatible rather than contradictory.

Recurrent networks process a sequence step by step. Self-attention can connect every position to every other position within a layer and compute positions in parallel during training. The tradeoff is quadratic work in sequence length for dense self-attention, which becomes important as contexts grow.

Maximum path length describes how many computational steps information must traverse between distant positions. In a self-attention layer, any two positions can interact directly, giving a constant-length path within that layer. A recurrent path grows with sequence length because information passes through intermediate states.

On WMT 2014 English-to-German translation, Table 2 reported 28.4 BLEU for the big model. The same table and abstract report 41.8 BLEU on English-to-French, while the prose in Section 6.1 states 41.0. The paper reports 3.5 days on eight NVIDIA P100 GPUs for the big English-to-German model. Keeping those protocol details and the paper's internal discrepancy visible is more accurate than presenting one number without context.

The durable lesson is not that one mechanism solved intelligence. It is that removing sequential computation made large-scale sequence learning more parallelizable, while attention created short paths between distant tokens. The modern AI stack grew from that foundation, then added new training objectives, much larger datasets, alignment methods, retrieval, tools, and product-level controls.

When reading later model papers, separate the inherited core from the surrounding choices. Decoder-only language modeling, larger context windows, new positional methods, sparse or local attention, mixture-of-experts layers, and post-training methods are later design decisions. Calling every one of them simply a Transformer can hide the differences that matter for behavior and cost.