Attention in transformers, step-by-step | Deep Learning Chapter 6

What people said

onlyrealcuzzo · 2026-05 · Strong recommendation

Because it doesn't work like how you think at all. You're still thinking it works like Chain of Thought. It doesn't. And the difference is key! It works by introducing probabilistic noise, and exploring N paths fully (each with noise) in parallel (all compressed). It's reasoning at a much, much smaller (probabilistic) level than running everything through the expensive large model (deterministic) and sometimes catching that it said, "I think 1.12 is greater than 1.9 because 12 is bigger than 9, final answer". The easiest way to think about it is: if you understand how hyper words work, it's as if it's searching for different versions of the hyper words that probilisticslly would lead to better outcomes IF it fed them to the LLM before it even does. That's not actually how it works exactly. But I think it is close enough to be helpful to understand where the gain is, a rough idea of what's happening (searching paths), and how it can potentially have huge orders of magnitude improvements (doing so without paying the full price of exploring the paths through the expensive and huge model). And also why it is so much harder to determine what it's "thinking". If you aren't familiar with hyper words, this is an amazing series: [VIDEO] The general idea is that a token is a multidimensional vector to represent a word -> think like "man" is a [noun, singular, English, pronoun, masculine, contemporary, ...]. Each time is sees a new word, it mutates this word to mean some new token (ofte

Read the original HN comment ↗

tomashm · 2024-07 · Strong recommendation

This is good, but what bade me finally understand the transformer architecture [0] and attention [1], are 3Blue1Brown's videos. 0. [LINK] 1. [VIDEO]

Read the original HN comment ↗

wren6991 · 2026-06 · Recommendation

If you want the actual maths instead of handwaving, I recommend: [LINK] For something gentler, 3Blue1Brown: [VIDEO] (this is part of a series)

Read the original HN comment ↗