CS480/680 Lecture 19: Attention and Transformer Networks

What people said

metanonsense · 2023-06 · Recommendation

What made attention mechanism click to me was the video [LINK] by Sebastian Raschka. Another good video on attention that tries to give a bit more intuition on the query/key/value stuff is [VIDEO] In hindsight, what helped me most is understanding the dimensionality at each step of the attention computation (explained well in Raschka's video). Another think I simply did not get from the original Transformer paper is that the learning of self-attention happens in the linear layers. (At least, that's my current understanding. If that's wrong, I appreciate any correction :)

Read the original HN comment ↗

elankart · 2023-05 · Recommendation

This might be dumbing this down very very much but it all boils down to having some sort of a special lookup table and you give it a "query" and a bunch of "keys" and the "value" is the most likely next word. As you input something into the network, a network of these tables are consulted and you are given most likely next word. The novelty in this paper is this "query-key-value" relation that gets learned. A lot of previous work in this area was focused on learning a rough state machine to which you input a set of state transitions and it will give you the most likely next state. This will also work but training such networks is very slow and you also don't have the capability to train the network to "attend" to certain part of the inputs. This lookup based technique lets you do that plus this is also very compute efficient (compared to previous techniques). I'm missing a lot of details but that's basically the intuition behind this. These are very excellent resources: - [LINK] - [VIDEO]

Read the original HN comment ↗

abhgh · 2023-04 · Recommendation

I would also recommend Pascal Poupart's talk [1] on Attention and Transformers. Doesn't seem to be cited often, but its quite well done and seeing him work out some of the details using chalk and board is very reassuring. [1] [VIDEO]

Read the original HN comment ↗