Transformers
Vaswani et al. · NeurIPS 2017
Fetching research paper
Downloading PDF from the archive
Original source not responding
We could not fetch or display this PDF. The host may be down, blocking embedding, or your connection may have dropped.
A button will appear below to pick another paper from the lab.
Continue reading
Choose another paper from the research lab.
These notes are written in plain language for this specific paper—so you can grasp the ideas before you wrestle with the authors’ formal wording. Use the button to open the PDF near the matching section (approximate page; Chromium-style viewers support #page=, otherwise we open a new tab).
Recurrent and convolutional models process text step by step, which limits parallel training on long sequences. The authors propose a model built only on attention—no RNN—so the whole sentence can be processed more in parallel while still capturing long-range dependencies.
Multi-head self-attention lets each position look at all others to build context. Add position encodings (because there’s no recurrence), residuals, and layer norm, and stack layers for an encoder–decoder (e.g., for translation). The famous Figure 1 is the map of data flow.
They train on the WMT English–German and English–French tasks—standard MT benchmarks—so BLEU scores compare directly to prior published systems.
BLEU improves over strong RNN + attention baselines, often faster to train per step because of parallelism. Look at training cost vs. quality tables, not only peak BLEU.
Attention is quadratic in sequence length (every token attends to every token), so very long documents get expensive in memory. Relative position and later sparse/long-context methods address this.
They connect to prior attention papers, CNN/RNN sequence models, and ByteNet. The punchline: attention alone is enough for competitive translation—setting the stage for BERT, GPT, and ViT.
Model size, warmup schedule, and hyperparameters are spelled out; the appendix is detailed. Modern reimplementations are everywhere (e.g., The Annotated Transformer), so students can line up code with the paper line by line.
Eight highlights per paper—why each part matters before you read dense notation and proofs.
RNNs serialize time steps; attention lets every position attend to every other in one layer (modulo depth). That unlocks massive parallel training on accelerators.
Softmax(QKᵀ/√d)V is the workhorse. The scaling prevents softmax saturation as dimension grows—small detail with large stability impact.
Several attention heads in parallel learn different relationship patterns; concatenation and projection mix them. Read it as ensemble of cheap pairwise routers.
Encoder uses full self-attention; decoder masks future tokens so generation stays causal. Confusing the two breaks autoregressive inference.
Attention is permutation-invariant without position info. Sinusoidal or learned embeddings inject order—critical for language and later adapted in vision patches.
Pre/post-norm variants differ, but the theme matches ResNet: stabilize deep stacks. Most modern LLM stacks are variations on this sandwich.
Self-attention is O(n²) in sequence length for full attention. Compare to RNN per-step cost and note why long-context methods (sparse, linear, sliding) exist.
Method and experiments define terms reused in BERT, GPT, ViT, and diffusion transformers. Map Figure 1 to tensor shapes once—it pays off across papers.