HackerTube.dev

Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP

Sasha Rush · 35.4K views · 2023-07-05 · 39:43

Open on YouTube ↗

If playback is restricted in your region, try opening the original video. Use Mark watched to add it to your marked videos.

Overview

Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.

On Hacker News

HN commenters recommend it for building intuition about state space models and the tradeoffs between transformer and SSM layers.

Recommended by 3 HN people across 3 months · HN mentions 2024–2024

Topics

Machine learning & deep learningLLMs & generative AI
Search by tags
ssmlinear rnnmambaattentions4nlp

What people said

a_wild_dandan · 2024-03 · Strong recommendation

To those curious about the tradeoffs between transformer and state space model layers, I highly recommend Sasha Rush's video on it: [VIDEO]

Read the original HN comment ↗
srush · 2024-02 · Recommendation

Hi! Blog author. This was an attempt a couple years ago to understand and write about this paper in a detailed way. Here is a video going through this topic as well: [VIDEO] Section 2 of the blog post is no longer very relevant. A lot of advances (DSS, S4D) simplified that part of the process. Arguably also this all should be updated for Mamba (same authors).

Read the original HN comment ↗
anomaly72 · 2024-01 · Recommendation

Would recommend Sasha Rush's lecture [VIDEO] , it provides the intuition for state space models via linear RNNs

Read the original HN comment ↗

Sources

Video by Sasha Rush, played through YouTube.

Data and selection notes

HN summaries and labels may contain AI errors. HN dates refer to discussions, not video publication.

HN Library: HackerTube export (2026-10-31) and open-index/hacker-news. Upstream summaries and classifications may contain AI errors.

Views, duration and playback status refresh separately from the published catalog. When an update fails, existing data remains visible.

Selection checks
  • SaaS & AI candidates: Matches the configured criteria
  • Editor-approved picks: Editorial review is required

Rule matches identify candidates. They do not guarantee video quality or financial results.

About the library and its sources