Reinforcement Learning Comparison
DQN, REINFORCE, and A2C: Three Ways to Approach CartPole
CartPole is useful for comparing reinforcement-learning methods because the environment and goal stay the same while the way the agent learns can change completely. My browser demo is based on a Deep Q-Network, but REINFORCE and Advantage Actor-Critic show two other ways to solve the same control problem.
The CartPole control problem
The agent receives four values: cart position, cart velocity, pole angle, and pole velocity. It then chooses one of two actions, moving the cart left or right. A strong method has to connect each action to both its immediate result and the effect it may have several steps later.
How DQN approaches the problem
DQN is a value-based method. The network predicts a Q-value for each action, and the agent usually chooses the action with the larger predicted value. Experiences are stored in replay memory and sampled later so the network can learn from earlier decisions instead of only the current step.
q_update = reward + GAMMA * np.amax(
self.model.predict(state_next)[0]
)This is the approach represented by the original notebook and the interactive browser project.
How REINFORCE is different
REINFORCE is a policy-gradient method. Instead of predicting the value of each action, it learns a policy that produces a probability distribution over the available actions. The agent completes an episode, calculates the discounted return for each step, and then adjusts the policy so actions connected to stronger returns become more likely.
for each episode:
collect states, actions, and rewards
calculate discounted returns
normalize returns
update the policy using each action and returnThe method makes policy gradients easier to follow because the connection between an episode’s total return and the policy update is direct. Its main drawback is that it waits until the episode ends before learning. It can also produce high-variance updates because a strong outcome may include both good decisions and lucky ones.
How A2C changes the learning process
Advantage Actor-Critic uses two parts. The actor learns which action to choose, while the critic estimates how valuable the current state is. Rather than waiting for a full episode, A2C can update after each step.
advantage = reward + gamma × V(next_state) - V(current_state)
The advantage tells the actor whether an action worked better or worse than expected. The actor uses that information to update its policy, and the critic updates its value estimate. This usually makes training more stable than REINFORCE while keeping the flexibility of direct policy learning.
Comparing the three methods
| Method | What it learns | When it updates | Main tradeoff |
|---|---|---|---|
| DQN | Value of each action | From replayed experiences | Works well with discrete actions but depends on value estimates. |
| REINFORCE | Action policy | After a complete episode | Clear policy-gradient method, but updates can be slow and unstable. |
| A2C | Policy and state value | At each step | More efficient feedback, but requires both actor and critic networks. |
Why the comparison matters
The environment does not determine the learning method. CartPole can be solved by estimating action values, optimizing a policy directly, or combining a policy with a value estimate. Looking at all three approaches makes it easier to understand why reinforcement learning includes several families of algorithms instead of one standard solution.
For this project, DQN is a practical match because CartPole has two discrete actions and the original implementation already uses replay memory and Q-value prediction. REINFORCE and A2C provide useful comparisons without changing what the finished browser demo actually runs.
References
- GeeksforGeeks. (2023). Advantage Actor Critic (A2C) in reinforcement learning.
- Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction (2nd ed.). MIT Press.
- Yoon, D. (2018). REINFORCE: Policy gradient with TensorFlow 2 and PyTorch.