Demo 06
Attention And Transformers: Why can tokens directly attend to each other?
Why is Attention better suited than an RNN for modeling long-range dependencies? This chapter places the historical transition after RNNs and before Transformers, showing why direct connections changed NLP.
Teaching interaction
Attention And Transformers: Why can tokens directly attend to each other?
Why is Attention better suited than an RNN for modeling long-range dependencies?
"a model" directly attends to "external knowledge"
With Attention, "a model" can skip intermediate words and connect directly to "external knowledge," which completes the meaning.
Attention lets each token inspect related tokens directly, so long-range relationships do not need to pass through every intermediate position.
- Understand how Attention creates direct connections between tokens instead of passing information only in sequence.
- Observe why relationships between distant tokens weaken more easily in RNN mode.
- Connect Attention to the historical role of Transformers and foundation models.
The attention weights in this demo are preset for teaching and do not come from a real Transformer model. They only illustrate the intuition behind direct connections.
Observation guide
From chain-like transmission to direct connections
The earlier problem was that RNNs passed state sequentially, so relationships between distant tokens could weaken. Attention shortened those information paths, but it did not directly solve training cost, hallucinations, or factual updates. Its follow-on influence includes Transformers, foundation models, and modern multimodal models.
References