Get Over Q*, OpenAI takes AGI to the Next Level with PPO

The OpenAI drama ends. The real action begins with the company secretly working on Q* (possibly based on Q-learning), but there is another interesting technique which is OpenAI’s all time favourite — PPO (short for proximal policy optimisation). 

OpenAI’s VP product Peter Welinder recently posted on X “Everyone reading up on Q-learning. Just wait until they hear about PPO.” 

What is PPO?

PPO is a reinforcement learning algorithm used to train artificial intelligence models to make decisions in complex, or simulated environments. 

Interestingly, PPO became the default reinforcement learning algorithm at OpenAI in 2017 because of its ease of use and good performance. 

The "proximal" in PPO's name refers to the constraint applied to the policy updates. This constraint helps prevent significant policy changes, contributing to more stable and reliable learning.

OpenAI employs PPO due to its effectiveness in optimising policies for sequential decision-making tasks. 

Moreover, PPO strikes a balance between exploration and exploitation, crucial in reinforcement learning, by incrementally updating policies while ensuring that the changes are constrained. 

OpenAI adopts PPO in a variety of use cases, ranging from training agents in simulated environments to mastering complex games. 

PPO's versatility allows it to excel in scenarios where an agent must learn a sequence of actions to achieve a specific goal, making it valuable in fields such as robotics, autonomous systems, and algorithmic trading. 

Chances are pretty much that OpenAI is aiming to achieve AGI through gaming and simulated environments with help of PPO. 
Interestingly earlier, this year OpenAI acquired Global Illumination to train agents in a simulated environment.