John Schulman - Reinforcement Learning from Human Feedback: Progress and Challenges

What people said

KKKKkkkk1 · 2023-05 · Recommendation

There is this idea that the goal of RLHF is to make ChatGPT woke or as you put it to lobotomize it. I suspect that this is a conspiracy theory. There's a very good talk by John Schulman, chief architect of ChatGPT [0], where he explains that if you don't include a RL component in your training, you're essentially doing imitation learning. It's well known that imitation learning fails miserably when presented with conditions that are not in your training set, i.e., answering questions that don't exist on the Internet already. So the goal of RLHF is actually to reduce hallucination. [0] [VIDEO]

Read the original HN comment ↗

mike_hearn · 2023-04 · Recommendation

Sure, here you go: [VIDEO] Summarizing: 1. Obviously you can't completely "solve" truthfulness because people disagree on what is and is not true. But you can go a long way. 2. The models do know what they don't know. Their level of uncertainty is not only expressed in the final token logprobs but also seems to be reified somehow, such that they can express their own level of certainty in words. 3. Many of the problems are introduced by subtle issues introduced during training which effectively teach the AI to guess. One example is by training on QA data sets where the answer is never "I don't know". RLHF introduces its own problems because the human trainers don't know what the model knows, so may reward it for getting a correct answer by guessing. 4. Many failures are caused by lack of access to information. Giving the LLM tools to let it search the web and access other databases can help a lot, and it's relatively easy to do. 5. In some cases you want it to guess. Like, it's much more useful when coding for it to spit out a screenful of code that has one or two minor errors, than to just refuse to try at all because the result might not be perfect.

Read the original HN comment ↗

chongli · 2023-07 · Reference

How is that "as good as an intern?" Interns, for the most part, do not hallucinate and confidently lie on topics they know nothing about. Good interns tend to ask a lot of questions and quickly learn the cultural cues around the office, allowing them to fit in and become productive. LLMs are incapable of any of this. John Schulman from OpenAI recently gave a talk about the hallucination and uncertainty issues [1]. He claimed that models already have information about their uncertainty but that it remains an open problem as to how to express that uncertainty in natural language. One of the big issues with trying to prevent the model from hallucinating is that you can err too much on the side of caution, causing the model to lie about things it actually does know the correct answer to. [1] [VIDEO]

Read the original HN comment ↗