Technical Article

How a Deep Q-Network Learns to Balance

CartPole is useful because the goal is easy to see while the learning process is not. The agent only moves left or right, but it has to connect four changing measurements to the long-term effect of every action.

What the agent receives

The environment describes each moment with four numbers:

• Cart position tells the agent where the cart is on the track.
• Cart velocity shows its current direction and speed.
• Pole angle measures how far the pole has moved from upright.
• Pole velocity shows how quickly that angle is changing.

Those values form the state. The network receives that state and produces two Q-values: one for pushing left and one for pushing right. The action with the stronger predicted value becomes the agent’s choice unless exploration selects a random action.

What a Q-value means

A Q-value is not a direct measurement of balance. It is the network’s estimate of how useful an action will be from the current state when future rewards are considered.

q_update = reward + GAMMA * np.amax(
    self.model.predict(state_next)[0]
)

The first part is the immediate reward. The second part uses the best predicted value from the next state and discounts it with GAMMA. With gamma set to 0.95, the agent considers future stability without treating a prediction about the future exactly like a reward it has already received.

Why the agent starts with random actions

At the beginning, the network has not learned which output deserves more confidence. Always choosing its current “best” action would lock it into decisions based on nearly meaningless predictions. An exploration rate of 1.0 gives the agent a broad range of early experiences.

After training begins, the exploration rate is multiplied by 0.995 until it reaches 0.01. The balance gradually shifts from discovering possibilities to using what the network has learned, while a small amount of randomness remains.

Why replay memory matters

Every step creates a five-part experience: the current state, selected action, reward, next state, and whether the episode ended. The agent saves those experiences instead of learning only from the most recent step.

Once memory contains at least 20 entries, the solver samples a batch and trains on those earlier transitions. Random sampling reduces the influence of consecutive, highly related states and gives the network repeated opportunities to learn from both successful and unsuccessful decisions.

How the network learns from a batch

For each sampled experience, the model predicts the current Q-values. The selected action’s target is replaced with either the terminal reward or the calculated immediate-plus-future value. Training then adjusts the network so its output moves closer to that target.

The network uses two hidden layers with 24 ReLU units each and a two-value linear output layer. Repeating this update across many batches gradually changes which actions the model values in different parts of the state space.

How progress is measured

An episode ends when the pole angle or cart position leaves the permitted range, or when the run reaches its maximum length. The score represents how many steps the agent kept the system within those limits. The rolling average matters more than a single high score because it shows whether the behavior is becoming consistently reliable.

What the simulation makes easier to understand

Watching the four state values beside the moving cart connects the math to visible behavior. The exploration rate explains why the agent may still make an unexpected move. Replay-memory growth shows that training is building from more than the current episode. The chart then shows whether those decisions are producing stronger performance over time.

Continue with another CartPole approach

The original project uses DQN, but CartPole can also be solved with policy-gradient and actor-critic methods. My companion article compares the DQN process shown here with REINFORCE and Advantage Actor-Critic.