Trends in Deep Learning Hardware: Bill Dally (NVIDIA)

What people said

raphlinus · 2024-01 · Recommendation

In a recent talk[1], Bill Dally of Nvidia suggests using actual logarithms, as they're more consistent in relative error than floats, which is especially relevant at low precision as increasingly used in AI. Multiplications then become simple additions, but then addition becomes more complicated, as in general it needs lookup tables. This is the same mechanism as a slide rule, and also reminds me of the way the Yamaha DX7 works. The lookup tables can be expensive, and the talk contains an idea how to batch them when adding up the results of many many multiplications (as is also often the case for AI workloads). [1]: [VIDEO]

Read the original HN comment ↗

alecco · 2023-12 · Recommendation

It looks like both Pat Gelsinger and Lisa Su have no clue about software and can't even delegate this. They expect "the community" to do the software for their extremely complicated hardware. If you see the presentations by Nvidia's Bill Daly [1] it shows they've been evolving hardware and software together. The CUDA programming model was a massive bet a long time ago. And they deservedly won ML/AI. Unless Intel and AMD do a radical change it's game over for them. They will lose to ARM and Nvidia. [1] [VIDEO]

Read the original HN comment ↗

huijzer · 2023-12 · Recommendation

Depends what you consider as part of chip production. If you consider new instructions and the related software as new chip production, then the performance improvements have continued quite dramatically. Single chip inference has gone 1000x over the last 10 years according to Bill Dally from Nvidia [1] at 10 minutes (probably not original source). Main gains according to the video have been different numeric types (fp16 for example) and more specialised hardware together with more complex instructions. Note that process improvements have only caused a 2.5x from that 1000x. He also mentions that Google‘s TPU has no fundamental benefit over dedicated instructions in GPUs. Quite an interesting video IMO. [1]: [VIDEO]

Read the original HN comment ↗