Multi-Agent Hide and Seek

What people said

aeternum · 2019-10 · Recommendation

Specialized AI is unlikely to be a threat. The issue will be with general AI, and simple but not fully thought out utility functions. This short video makes the concern much more real: [VIDEO] Consider what would happen if you took that same AI, make its utility function similar to this: [LINK] , and released it on the internet.

Read the original HN comment ↗

HanClinto · 2024-07 · Reference

True RL is not limited by being tethered to human-annotated data, and it is able to create novel approaches to solve problems. True RL requires a very clear objective function (such as the rules of Go, or Starcraft, or Taboo!) that the model can evaluate itself against. Andrej Karpathy talks about the difference between RLHF and "true" RL here: [LINK] > The other thing is that we're doing reinforcement learning from human feedback (RLHF), but that's like a super weak form of reinforcement learning. I think... what is the equivalent in AlphaGo for RLHF? What is the reward model? What I call it is a "vibe check". Imagine if you wanted to train an AlphaGo RLHF, it would be giving two people two boards and asking: "Which one do you prefer?" -- and then you would take those labels and you would train the model and then you would RL against that. What are the issues with that? It's like, number one -- that's just vibes of the board. That's what you're training against. Number two, if it's a reward model that's a neural net, then it's very easy to overfit to that reward model for the model you're optimizing over, and it's going to find all these spurious ways of hacking that massive model is the problem. > AlphaGo gets around these problems because they have a very clear objective function, and you can RL against it. > So RLHF is nowhere near [true] RL -- it's silly. And the other thing is that imitation is super-silly. RLHF is a nice improvement, but it's still silly, and I think p

Read the original HN comment ↗

PartiallyTyped · 2022-09 · Reference

Here [1]. They called it "emergent" property, but I only see that as inevitable behaviour. [1] [VIDEO]

Read the original HN comment ↗