Inside the CartPole Build

Workspace and dependencies

The notebook imports Gym for the environment, NumPy for state and value operations, Keras for the dense network, and a separate score logger for run history.

01import random
02import gym
03import numpy as np
04from collections import deque
05from keras.models import Sequential
06from keras.layers import Dense
07from keras.optimizers import Adam
08
09from scores.score_logger import ScoreLogger

Training configuration

These are the baseline values stored in the notebook and represented by the model profile on the interactive page.

ENV_NAME = "CartPole-v1"

GAMMA = 0.95
LEARNING_RATE = 0.001

MEMORY_SIZE = 1000000
BATCH_SIZE = 20

EXPLORATION_MAX = 1.0
EXPLORATION_MIN = 0.01
EXPLORATION_DECAY = 0.995

Network architecture

The solver uses two 24-unit ReLU layers and a two-value linear output layer. Those outputs represent the predicted value of moving left or right.

class DQNSolver:
    def __init__(self, observation_space, action_space):
        self.exploration_rate = EXPLORATION_MAX
        self.action_space = action_space
        self.memory = deque(maxlen=MEMORY_SIZE)

        self.model = Sequential()
        self.model.add(Dense(24, input_shape=(observation_space,), activation="relu"))
        self.model.add(Dense(24, activation="relu"))
        self.model.add(Dense(self.action_space, activation="linear"))
        self.model.compile(loss="mse", optimizer=Adam(lr=LEARNING_RATE))

Action selection and replay memory

The agent sometimes explores with a random action. Otherwise, it predicts both Q-values and selects the larger one. Stored transitions are sampled in batches so the network can learn from earlier decisions.

def act(self, state):
    if np.random.rand() < self.exploration_rate:
        return random.randrange(self.action_space)

    q_values = self.model.predict(state)
    return np.argmax(q_values[0])

def remember(self, state, action, reward, next_state, done):
    self.memory.append((
        state,
        action,
        reward,
        next_state,
        done
    ))

Q-value update

For non-terminal states, the target combines the immediate reward with the strongest predicted value available from the next state.

if len(self.memory) < BATCH_SIZE:
    return

batch = random.sample(self.memory, BATCH_SIZE)
for state, action, reward, state_next, terminal in batch:
    q_update = reward
    if not terminal:
        q_update = reward + GAMMA * np.amax(
            self.model.predict(state_next)[0]
        )

    q_values = self.model.predict(state)
    q_values[0][action] = q_update
    self.model.fit(state, q_values, verbose=0)

Parameter tests

I tested a larger gamma value to give future rewards more weight and separately increased the learning rate to compare the effect of a much larger update step.

# Increased from the 0.95 baseline
GAMMA = 0.995

# Expected: stronger long-term decisions,
# with the possibility of slower convergence.

# Increased from the 0.001 baseline
LEARNING_RATE = 0.1

Minimum-exploration experiment

I also tested whether raising the minimum exploration rate would prevent the agent from settling too early on a limited strategy. For this experiment, I increased EXPLORATION_MIN from 0.01 to 0.02 and kept the other baseline values unchanged.

EXPLORATION_MAX = 1.0
EXPLORATION_MIN = 0.02
EXPLORATION_DECAY = 0.995
LEARNING_RATE = 0.001
GAMMA = 0.95

Episodes completed: 440
Solve point: run 340
Maximum score: 500
Late-stage average: over 190
Minimum observed score: 8

The agent learned more gradually, became stable above 180 at approximately run 150, and maintained strong averages through the final 100 or more runs. Occasional low episodes still occurred, but the higher exploration floor helped the model recover without lowering its overall late-stage performance.

Recorded training output

The retained results show three solved runs. The model reached the required 195 rolling average and recorded the maximum CartPole-v1 score of 500.

In [2]: cartpole()
Run: 189, exploration: 0.01
Scores: min 16, avg 195.38
SOLVED in 89 runs, 189 total runs.

Average: 197.25
Maximum score: 500
SOLVED in 99 runs, 199 total runs.

Average: 199.43
Maximum score: 500
SOLVED in 39 runs, 139 total runs.