HackerTube.dev

Glitch Tokens - Computerphile

Computerphile · 352.8K views · 2023-03-07 · 19:29

Open on YouTube ↗

If playback is restricted in your region, try opening the original video. Use Mark watched to add it to your marked videos.

Overview

Rob Miles explains "glitch" tokens, odd strings in large language model vocabularies that make the model produce gibberish. He covers how they likely arise from tokenizer training data, such as Reddit usernames, that never appeared in the model's own training.

On Hacker News

HN points to it as the accessible explanation of why glitch tokens exist whenever tokenizers or odd GPT behavior come up.

Recommended by 2 HN people across 2 months · HN mentions 2023–2023

Topics

LLMs & generative AIComputer vision, speech & NLP
Search by tags
glitch tokenstokenizersgptai safetybpe

What people said

weinzierl · 2023-06 · Recommendation

Computerphile has an interesting video about Glitch Tokens, but as it goes with Computerphile videos it's also a good introduction to the topic. [VIDEO]

Read the original HN comment ↗
non- · 2023-04 · Recommendation

I found this tool recently when it was linked from this Computerphile video[1] about "glitch tokens". tldw: Certain junk data was thrown out post-tokenization e.g. the /r/counting[2] community data and debug logs from Rocket League some tokens specific to those contexts stuck around, however, and are now like "a color you've never seen before" as far as GPT-X models are concerned giving the model one of these "glitch" tokens causes it to kind of freak out and return gibberish or some completely random response because it has not encountered them during training, because they were removed when the data was cleaned. [1] [VIDEO] [2] [LINK]

Read the original HN comment ↗
veidr · 2023-06 · Reference

That is indeed the funniest (and slightly horrifying) thing. Even if you understand "glitch tokens"[1] (which is BTW the best way I've seen to explain to nontechnicals (hi, Dad!!) and also some subset of technicals (hi various friends of mine!! +_+...) that LLM applied statistics models aren't anything like sentient AGI)... ...regardless, ChatGPT and its ilk nevertheless do often already outperform "random human being coworker or uncle" at recommending just about anything... which is like Ted Chiang said[2], “That’s more a statement about how much bullshit we are required to generate and deal with in our daily lives.” Although he was not referring to inputs from your (dumbass) coworkers/uncles, it still holds true. [1]: [VIDEO] [2]: [LINK]

Read the original HN comment ↗

Sources

Video by Computerphile, played through YouTube.

Data and selection notes

HN summaries and labels may contain AI errors. HN dates refer to discussions, not video publication.

HN Library: HackerTube export (2026-10-31) and open-index/hacker-news. Upstream summaries and classifications may contain AI errors.

Views, duration and playback status refresh separately from the published catalog. When an update fails, existing data remains visible.

Selection checks
  • SaaS & AI candidates: Matches the configured criteria
  • Editor-approved picks: Editorial review is required

Rule matches identify candidates. They do not guarantee video quality or financial results.

About the library and its sources