Training AI Without Writing A Reward Function, with Reward Modelling

What people said

cubefox · 2023-07 · Strong recommendation

Specifically about RLHF, I find this video by Rob Miles still the best presentation of the ingenious original 2017(!) paper: [VIDEO] RLHF is actually older than GPT-1, which came out in 2018. It didn't get applied to language models until 2022 with InstructGPT, an approach which combined supervised instruction fine-tuning with RLHF.

Read the original HN comment ↗

sillysaurusx · 2019-12 · Strong recommendation

I would highly recommend [VIDEO] as a way of framing the problem. It seems like our options for ML are currently: self-play, dataset fitting, or human feedback. It's possible to solve any problem that can fit into one of those molds. (Unfortunately the opposite is true: if a problem doesn't fit into those categories, it seems very hard to solve with any form of ML.) But, now that I think about it a bit more, the sentence "use a trained net as a tool for building a controller with known continuity properties" seems neither obvious nor straightforward to do. I'm decently well-versed in AI techniques at this point, and I can't really think how I'd sit down and write a program that would generate such a system. Anyone have any leads? Basically, the goal is to make a neural net that helps you design a system that can be predictable. One way would be to write a loss function that penalizes unpredictability, and then let it run until it converges on a predictable control system. But that's a bit like saying "Just draw an owl." Not so easy to think of the actual code to do that.

Read the original HN comment ↗

afarrell · 2020-07 · Recommendation

ADHDer here. Please steal some of the motivation-techniques I've learnt over the years: 1. Find a friend who is a software engineer or data analyst and cares about you as a person. Talk about this with them. Humans evolved being able to talk things over with their aunt or best friend. When you are perplexed by something, a rubber duck probably won't cut it -- you should talk with a human. 2. Make sure you're taking care of your body and brain: exercise, good food, good sleep (this is easier if you're getting exercise), meditation, etc. 3. Understand what your "reward model" is. Under what conditions do you recognize "Aha! I've made progress!"? To really understand what I'm getting at here, watch this video: [VIDEO] 4. Find internal things to strengthen your reward model's signals. Go back to when you originally started learning ML and Re-connect to your "Why". Do you still find those reasons compelling? If so, take some time to bring some vividness to your picture. -- Start With Why. 5. Find external punishments to remove. Now that you've spent 8 months working on it, you've noticed the things which are annoying you about your workflow. - After 8 months, it is totally healthy to take a week or two and improve your development environment to sand the edges off of things that give you friction. For a stranger's recommendation: [LINK] has made it dramatically easier for me to write little scripts that take away annoying tasks. - Maybe you feel like you've been "in the weeds" for

Read the original HN comment ↗