Attention Is All You Need

What people said

bfung · 2023-01 · Strong recommendation

YouTube channel that explains the paper in detail: [VIDEO] And subsequent follow-ups (ROME, editing transformer arch): [LINK] I find the channel amazing explaining super complex topics in simple enough terms for people who have some background in AI.

Read the original HN comment ↗

throwawaymaths · 2023-05 · Recommendation

The Yannic kilcher review is quite good. [VIDEO] I can't ELI5 but I can ELI-junior-dev. Tl;dw: Transformers work by basically being a differentiable lookup/hash table. First your input is tokenized and (N) tokens (this constitutes the attention frame) are encoded both based on token identity and position in the attention frame. Then there is an NxN matrix that is applied to your attention frame "performing the lookup query" over all other tokens in the attention frame, so every token gets a "contextual semantic understanding" that takes in both all the other stuff in the attention frame and it's relative position. Gpt is impressive because the N is really huge and it has many layers. A big N means you can potentially access information farther away. Each layer gives more opportunities to summarize and integrate long range information in a fractal process. Two key takeaways: - differentiable hash tables - encoding relative position using periodic functions NB: the attention frame tokens are actually K-vectors (so the frame is a KxN matrix) and the query matrix is an NxNxK tensor IIRC but it's easier to describe it this way

Read the original HN comment ↗

visarga · 2021-12 · Recommendation

For accessibility I recommend Yannic Kilcher video review of "Attention Is All You Need" [VIDEO] Yannic has been making about 62 other transformer paper reviews since. You can find the usual suspects. [LINK]

Read the original HN comment ↗