No Priors Ep. 80 | With Andrej Karpathy from OpenAI and Tesla

What people said

modeless · 2024-10 · Recommendation

Check the video here. [VIDEO] He's not just politely saying he likes Tesla, he's talking for 5 minutes in detail about exactly why he believes Tesla is ahead of Waymo and will beat them to scale.

Read the original HN comment ↗

HarHarVeryFunny · 2025-10 · Reference

Right. In a recent YouTube interview Karpathy claimed that LLMs have a lot more "working memory" than a human: [VIDEO] What I assume he's talking about is internal activations such as stored in KV cache that have same lifetime as tokens in the input, but this really isn't the same as "working memory" since these are tied to the input and don't change. What it seems an LLM would need to do better at these sort of iterative/sequencing tasks would be a real working memory that had more arbitrary task-duration lifetime and could be updated (vs fixed KV cache), and would allow it to track progress or more generally maintain context (english usage - not LLM) over the course of a task. I'm a bit surprised that this type of working memory hasn't been added to the transformer architecture. It seems it could be as simple as a fixed (non shifting) region of the context that the LLM could learn to read/write during training to assist on these types of task. An alternative to having embeddings as working memory is to use an external file of text (cf a TODO list, or working notes) for this purpose which is apparently what Claude Code uses to maintain focus over long periods of time, and I recently saw mentioned that the Claude model itself has been trained to use read/write to this sort of text memory file.

Read the original HN comment ↗

93po · 2024-09 · Reference

Karpathy himself believes that neural networks are perfectly plausible as a key component to AGI. He has said that it doesn't need to be superseded by something better, it's just that everything else around it (especially infrastructure) needs to improve. As one of the most valuable opinions in the entire world on the subject, I tend to trust what he said. source: [VIDEO]

Read the original HN comment ↗