Modeling the World

Introduction

What does Machine Learning (ML) learn? It took me a while to answer this question but the answer is actually straightforward. Machine learning always learns a function: $y=f(x)$ .

In supervised learning, you show it examples of inputs $x$ with the right answers $y$ , and it learns to predict $y$ from $x$ .
In unsupervised learning, you only show it inputs, and it finds structure in them: clusters, directions, a compressed description.

But Reinforcement learning (RL) learns something else: how to act. An agent looks at its situation, takes an action, sees what happens, and does it again, step after step. Nobody tells it which action was right. It only receives a number, a reward, after each step, which can be: points in a game, a penalty for falling, a bonus for reaching the goal. Its job is to find a way of acting, a policy, that collects as much reward as possible in the long run.

This setting covers a lot of ground: an agent learning to play Atari games from the pixels and the score, a simulated body learning to walk without falling, AlphaGo beating Lee Sedol at Go, AlphaZero mastering chess and shogi by self-play, language models fine-tuned from human preferences. Each time, a sequence of decisions, each one changing what the next one will face, and a score that says how well it went.

Two questions run through the whole subject:

What is the best way to act? It needs a precise answer: best for which score, over how long, and when the world is random.
How do we find it? When the rules of the world are known, we can compute the best behaviour without ever acting: this is planning. When they are not, the agent must learn it from experience, it is called learning and is the true purpose of RL.

This page is about the first case: planning with a model. This is the beginning of the story of Reinforcement Learning. It builds the framework that all of RL rests on, we’ll see:

RL framework:

Sequential decisions and the return, the score of a whole trajectory.
Markov decision processes, the model of the world.

Evaluating a decision:

Value functions and the Bellman equations: how good a state is, and how we can link them by a one-step recursion.

Taking better decisions:

Optimal values and policies: the best one can do, and the Bellman optimality equation that characterizes it.

Then Part 2, Planning with a Model turns these equations into algorithms that solve them exactly: policy evaluation, policy iteration and value iteration. These are required steps before switching to Part 3, Learning from experience, which removes the model and uses Monte Carlo, temporal-difference learning, SARSA and Q-learning to learn the same values from sampled interactions only. Everything they do is an approximation of what Parts 1 and 2 compute exactly.

A word on notation, which follows Sutton and Barto’s book.

$\doteq$ means “is defined as”.
True values are lowercase, $v_\pi$ , $q_\pi$ , $v_*$ , $q_*$ , and so are the successive approximations of dynamic programming, $v_0, v_1, v_2, \dots$ ; the estimates learned from experience in Part 3 are uppercase, $V$ and $Q$ .
A star $^*$ means optimal.

1. Sequential decisions

Taking good decisions starts with being able to evaluate the impact of a decision in a given situation. This section describes the framework that is used to describe: 1) the impact of a decision and 2) the quality of this decision.

1.1 Decisions in stages

In sequential decision making, a decision maker takes successive decisions, each one based on what it observes of the effect of the previous ones. Picture a decision tree: from the current situation, each possible action leads to a new situation, from which new actions branch out, and so on. A good decision maker finds a good path through this tree.

Decisions in stages

Each action leads to a new situation, from which new actions branch out. A trajectory is one path through the tree.

The framework is called sequential because time is discrete: there is a first decision, a second one, a third one. But the framework says nothing about how long a step lasts: a millisecond for a robot’s motor, a day for a stock portfolio. And only time is discrete: the actions themselves can be a choice among a few options (left, right) or a real number (the torque of a joint).

Think about it for a second, this can cover a large portion of real-world problems.

In RL, a decision maker, the agent, interacts with its environment, everything distinct from itself. They talk in a loop. At each step $t$ , the agent observes the state $S_t$ and picks an action $A_t$ ; the environment answers with a reward $R_{t+1}$ and a new state $S_{t+1}$ . One interaction produces a trajectory:

S_0,\ A_0,\ R_1,\ S_1,\ A_1,\ R_2,\ S_2,\ A_2,\ R_3,\ \dots

Note the index of the reward: $R_{t+1}$ is the reward that follows action $A_t$ , received together with $S_{t+1}$ .

The agent–environment loop

Press Step to run the loop: the agent picks an action A_t, the environment answers with a reward R_t+1 and a new state S_t+1, which the agent observes at the next step. Each event adds its symbols to the trajectory.

This is a sequential decision making framework, with a reward describing how good an action, from a given state, is.

1.2 Optimal means optimal for a score

What does being optimal mean?

Being optimal is always being optimal with regard to a cost or a score. To illustrate this concept, here is the running example of this page, a tiny map with four places at different altitudes:


S (start) · 0 m	A · 10 m
B · 40 m	G (goal) · 5 m

A hiker starts at $S$ and wants to reach $G$ with as little effort as possible, the effort being the total altitude change along the way. Two routes:

right, then down: $0 \to 10 \to 5$ , an effort of $|10 - 0| + |5 - 10| = 15$ m;
down, then right: $0 \to 40 \to 5$ , an effort of $|40 - 0| + |5 - 40| = 75$ m.

The altitude map

The two routes from S to G, on the map and as a graph of states. Each move is an arrow labelled with its action and its effort, the altitude change; G is terminal. The effort of a route is the sum of the efforts of its moves: the route through B climbs 40 m only to come down again.

Both reach $G$ , but the first route is five times cheaper. “Best” only means something with respect to a score computed along the whole trajectory. Control theory usually minimizes a cost $\sum_t c(S_t, A_t)$ ; RL maximizes a sum of rewards $\sum_t R_t$ . They are the same thing with $R = -c$ : on our map, the reward of a move will be minus the absolute altitude change.

In RL, the reward is a step signal: a single number handed to the agent after each step, not a grade computed once at the end. The agent does not see the reward structure in advance; it discovers it one step at a time. Sutton and Barto state the bet behind the whole field as the reward hypothesis: all goals can be described as the maximization of the expected cumulative reward.

Ideally, the reward would be:

$+1$ for success;
$0$ otherwise.

In practice, such a sparse signal is often too hard to learn from, and designers add intermediate rewards to guide the agent, at the risk of rewarding something other than what they meant.

1.3 The return and the discount factor

The quantity the agent actually maximizes is not the reward directly but rather the return, the sum of the discounted rewards to come, from step $t$ on:

G_t \doteq R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}

The number $\gamma \in (0, 1]$ is the discount factor: a reward received $k$ steps from now counts $\gamma^k$ times as much as the same reward received now. Like money, cash now is worth more than potential money later. With $\gamma$ close to $0$ , the agent is said to be myopic: only the next rewards matter. With $\gamma$ close to $1$ , it is said to be far-sighted.

The discount also keeps the sum finite. When the task never ends (a continuing task) and the rewards are bounded, $|R_t| \le R_{\max}$ , the geometric series gives

|G_t| \le \sum_{k=0}^{\infty} \gamma^k R_{\max} = \frac{R_{\max}}{1 - \gamma} \qquad \text{for } \gamma < 1

When the task stops after a finite number of steps (an episodic task, like a game or our hike), the sum is finite anyway, and $\gamma = 1$ is allowed.

On our map, the route right-then-down gives $R_1 = -10$ and $R_2 = -5$ , then the episode ends. With $\gamma = 1$ , $G_0 = -15$ ; with $\gamma = 0.9$ , $G_0 = -10 + 0.9 \times (-5) = -14.5$ : the second step is discounted, the first never is.

For a task that never ends, with a reward of $+1$ at every step, the return is a geometric series: $G_0 = 1 + \gamma + \gamma^2 + \cdots = \frac{1}{1 - \gamma}$ . That is $10$ with $\gamma = 0.9$ , $100$ with $\gamma = 0.99$ , and no finite value with $\gamma = 1$ : roughly, the agent looks $\frac{1}{1 - \gamma}$ steps ahead.

Play with $\gamma$ on a few reward sequences:

The discount factor weighs the future

Light bars: the rewards R; solid bars: their contribution γ^kR to the return G₀. Lower γ and the future fades. The route S → A → G ends after two steps; the task with +1 forever never ends, and its return stays finite only because γ < 1.

One identity will matter more than any other on this page. Take the first reward out of the sum, factor $\gamma$ out of the rest, and shift the index with $j = k - 1$ :

\begin{aligned} G_t &= \sum_{k=0}^{\infty} \gamma^k R_{t+k+1} = R_{t+1} + \sum_{k=1}^{\infty} \gamma^k R_{t+k+1} \\ &= R_{t+1} + \gamma \sum_{j=0}^{\infty} \gamma^j R_{(t+1)+j+1}. \end{aligned}

What remains is the return of the next step:

G_t = R_{t+1} + \gamma\, G_{t+1}

Stated differently, the return is the next reward plus the discounted return from the next step on.

An episode. With $\gamma = 0.5$ , an episode gives the rewards $R_1 = -1$ , $R_2 = 2$ , $R_3 = 6$ , $R_4 = 3$ , $R_5 = 2$ , then ends at $T = 5$ (so $G_5 = 0$ ). Compute $G_5, G_4, \dots, G_0$ .
A task that never ends. With $\gamma = 0.9$ , the rewards are $R_1 = 2$ , followed by an infinite sequence of $7$ s: $R_2 = R_3 = \cdots = 7$ . Compute $G_1$ , then $G_0$ .
The sequence of question 2 never ends. Why is $G_1$ still a finite number?

Hint

Work backwards, from $G_5 = 0$ , with $G_t = R_{t+1} + \gamma\, G_{t+1}$ : each return is one reward plus the discounted return you just computed.

Hint

For question 2, $G_1 = 7 + 7\gamma + 7\gamma^2 + \cdots$ is a geometric series: $\sum_{k \ge 0} \gamma^k = \frac{1}{1 - \gamma}$ for $\gamma < 1$ .

Checkpoint what you should get

In question 1, $G_0 = 2$ ; in question 2, $G_1 = 70$ and $G_0 = 65$ .

Solution

Backwards, with $G_t = R_{t+1} + 0.5\, G_{t+1}$ :
- $G_5 = 0$
- $G_4 = 2 + 0.5 \times 0 = 2$
- $G_3 = 3 + 0.5 \times 2 = 4$
- $G_2 = 6 + 0.5 \times 4 = 8$
- $G_1 = 2 + 0.5 \times 8 = 6$
- $G_0 = -1 + 0.5 \times 6 = 2$
Forwards gives the same result, $G_0 = -1 + 0.5 \times 2 + 0.25 \times 6 + 0.125 \times 3 + 0.0625 \times 2 = 2$ , with more work.
From $t = 1$ on, the reward is always $7$ : $G_1 = \frac{7}{1 - 0.9} = 70$ , and $G_0 = R_1 + \gamma\, G_1 = 2 + 0.9 \times 70 = 65$ .
Because $\gamma < 1$ : the weights $\gamma^k$ form a convergent geometric series, and with bounded rewards $|G_t| \le \frac{R_{\max}}{1 - \gamma}$ . Here the bound $\frac{7}{0.1} = 70$ is reached exactly. With $\gamma = 1$ , the sum would diverge.

What to notice

The backward recursion computes all the returns of an episode in one pass, each from the next one. This is how Monte Carlo methods compute returns in Part 3, and the same split, one reward plus the rest, is the seed of the Bellman equations of section 3.

2. Markov decision processes

The previous section showed what taking decisions looks like. But before applying this to real problems, we need another brick: how can we model the world as a decision tree?

2.1 The Markov property

Do we need to know all the past to take a good decision?

The past is the history, everything the agent has seen so far: observations, actions, rewards. The state is what we keep of it to decide what comes next. Keeping everything works, but it grows without end; the goal is to use the smallest summary that loses nothing useful for predicting the future.

A state $S_t$ is said to be Markov when it has the following property: knowing $S_t$ is enough to predict $S_{t+1}$ , and the rest of the history adds nothing,

\mathbb{P}[S_{t+1} \mid S_t] = \mathbb{P}[S_{t+1} \mid S_1, \dots, S_t]

In words: the future is independent of the past, given the present. For a point mass pushed by a force, position and velocity form a Markov state: the next position and velocity follow from them and from the force applied, whatever path led there. Position alone would not be Markov.

Is the state Markov?

A point mass in the plane. Two different pasts (dashed) reach the same position x_t, and the same force gives the same acceleration a_t. Left: they also share the velocity v_t, and one next state follows. Right: different velocities, two different next positions. Faint dotted: where the mass would go without the action; the thin curve is the path the action bends.

We often get to choose the state, and a common trick is to put a bit of history in it to make it Markov. In the Atari game, DeepMind used the last four frames of images as the state.

2.2 The model of the world

A Markov decision process (MDP) is the mathematical model of a sequential decision problem. It is made of:

a set of states $\mathcal{S}$ , and for each state $s$ a set of actions $\mathcal{A}(s)$ ;
the dynamics $p$ , which says where an action leads and what it rewards;
a discount factor $\gamma \in [0, 1]$ .

The dynamics is a single function describing the joint probability of the next state and of the reward, given the current state and action:

p(s', r \mid s, a) \doteq \mathbb{P}[S_t = s',\ R_t = r \mid S_{t-1} = s,\ A_{t-1} = a]

For each pair $(s, a)$ , it is a probability distribution over the pairs $(s', r)$ : after taking $a$ in $s$ , the agent lands somewhere and collects some reward, so $\sum_{s', r} p(s', r \mid s, a) = 1$ .

Other functions follow from $p$ , for instance the probability of landing in $s'$ and the expected reward:

\begin{aligned} p(s' \mid s, a) &= \sum_{r} p(s', r \mid s, a), \\ r(s, a) &\doteq \mathbb{E}[R_t \mid S_{t-1} = s, A_{t-1} = a] = \sum_{s', r} r\ p(s', r \mid s, a) \end{aligned}

A function $s' = f(s, a)$ would only describe a world where the same action always has the same effect. Real environments are often random: a wheel slips, a die is rolled, another player moves. A probability covers all these cases at once, and a deterministic world is just the case where one outcome has probability $1$ . The reward is in the same distribution because it can be random too, and it often depends on where the agent lands.

The dynamics of one action

The distribution p(s′, r | A, down) on our map. Deterministic: the map of this page. Stochastic: the same move on a slippery path, which can also leave the hiker at A or send her back to S. The thickness of an arrow is its probability; in both cases the probabilities sum to 1. The hiker takes the action again and again: on the slippery path, the outcome changes from one try to the next, and the counts below approach the probabilities.

On our 2×2 map example:

Here is our map framed as an MDP with some design choices:

The states are $S$ , $A$ , $B$ and $G$ .
The reward of a move is minus the altitude change, $r = -|\text{alt}(s') - \text{alt}(s)|$ .
Nothing forces the hiker onward: from $A$ and $B$ , they can also go back to $S$ , or stay where they are, at a small cost of $-1$ .
$G$ is terminal: no action is taken there, the episode ends, and its value is $0$ by convention.

Each cell gives the next state $s'$ and the reward $r$ of an action $a$ taken in a state $s$ ; a dash means the action is not available there.

$s$	right	down	left	up	stay
$S$	$A$ , $-10$	$B$ , $-40$	–	–	–
$A$	–	$G$ , $-5$	$S$ , $-10$	–	$A$ , $-1$
$B$	$G$ , $-35$	–	–	$S$ , $-40$	$B$ , $-1$

This environment is deterministic: for each pair $(s, a)$ , a single pair $(s', r)$ has probability $1$ , for instance $p(A, -10 \mid S, \text{right}) = 1$ , and every other pair has probability $0$ . The sum over $(s', r)$ is still $1$ , not because the world is deterministic, but because $p(\cdot, \cdot \mid s, a)$ is a probability distribution.

Vocabulary:

The process is Markov because $p$ conditions on the last state and action only. It is stationary if $p$ does not change over time, which we assume. And it is finite when there are finitely many states, actions and rewards: $p$ is then a table of numbers, the case of this whole page.

2.3 Policies

Last, how do we describe a behaviour compared to another?

A policy is the agent’s decision rule: it describes which action to take in which state. In general it is a probability law, a probability distribution over the actions for each state:

\pi(a \mid s) \doteq \mathbb{P}[A_t = a \mid S_t = s], \qquad \sum_{a \in \mathcal{A}(s)} \pi(a \mid s) = 1 \ \text{ for every } s

A deterministic policy, $a = \pi(s)$ , is the special case where all the probability sits on one action.

Two policies on our map will come back in the exercises:

$\pi_A$ , deterministic: right at $S$ , down at $A$ , right at $B$ . In the general form, $\pi_A(\text{right} \mid S) = 1$ and $\pi_A(\text{down} \mid S) = 0$ .
$\pi_B$ , stochastic at $S$ : $\pi_B(\text{right} \mid S) = 0.3$ and $\pi_B(\text{down} \mid S) = 0.7$ ; the same as $\pi_A$ at $A$ and $B$ .

Two policies on the map

A hiker walks one episode under π_A, then one under π_B, in a loop. Arrows: the actions of the current policy, their thickness proportional to π(a | s). Beside the map, the two policies as decision tables. π_B draws its action at S at random: run it a few times and both routes appear.

Both sum to $1$ at $S$ , which is what makes them policies. The normalization holds state by state.

Once a problem is written as an MDP, there is a well-defined notion of best behaviour: an optimal policy $\pi_*$ is the decision rule that maximizes the expected return.

The MDP is the model of the problem; the policy is its solution. The next section makes “maximizes the expected return” precise.

3. Value functions and the Bellman equation

Now that we know how to define our environment and what taking decisions means, we will study how to compare policies and find the best actions.

3.1 How good is a state, how good is an action

To compare policies, and to improve them, we need to know how good each situation is. The state-value function of a policy $\pi$ is the expected return when starting in $s$ and following $\pi$ :

v_\pi(s) \doteq \mathbb{E}_\pi[G_t \mid S_t = s]

The action-value function is the expected return when starting in $s$ , taking action $a$ first, then following $\pi$ :

q_\pi(s, a) \doteq \mathbb{E}_\pi[G_t \mid S_t = s,\ A_t = a]

$\mathbb{E}_\pi[\cdot]$ is the expectation of the return, under a certain policy. Since our environment and our policies might be stochastic, the return has to be evaluated on average, to compare between policies.

The expectation is over everything random: the actions drawn from $\pi$ and the outcomes drawn from $p$ . Both functions score the same return; $q_\pi$ just conditions on one more piece of information, the first action. $v_\pi$ ranks states; $q_\pi$ ranks the moves available in a state. In discrete cases, these expectations are expressed as sums directly.

On our 2×2 example, take the policy that flips a coin at $S$ , right or down with probability $\frac12$ each, then heads straight for $G$ , with $\gamma = 1$ .

From $A$ the return is $-5$ , from $B$ it is $-35$ , and from $S$ it is $-15$ or $-75$ depending on the coin ( $-45$ on average), leading to:

$s$	$S$	$A$	$B$	$G$
$v_\pi(s)$	$-45$	$-5$	$-35$	$0$

At $S$ , the first two moves give $q_\pi(S, \text{right}) = -15$ and $q_\pi(S, \text{down}) = -75$ : already, $q_\pi$ says that right is the better move. The whole table of $q_\pi$ , one row per state, one column per action: each entry is the reward of the first move plus the value $v_\pi$ of where it lands.

$q_\pi(s, a)$	right	down	left	up	stay
$S$	$-15$	$-75$	–	–	–
$A$	–	$-5$	$-55$	–	$-6$
$B$	$-35$	–	–	$-85$	$-36$

And averaging the action values over the policy’s own choices gives back the state value (weighted by the probability of each action):

v_\pi(s) = \sum_{a} \pi(a \mid s)\, q_\pi(s, a)

For instance on $S$ :

\qquad v_\pi(S) = \tfrac12 (-15) + \tfrac12 (-75) = -45

3.2 The Bellman expectation equation

Computing $v_\pi(s)$ from its definition means averaging the return over every trajectory that starts in $s$ . Their number explodes with the length of the episode, and it is infinite as soon as the process can run forever, as on our map, where the hiker can walk back and forth.

Every trajectory from S

All the ways the hiker can walk from S on our map, over three steps. A branch ends at G; from S, A or B, two moves keep walking, so the unfinished trajectories double at every step and never run out.

Bellman’s idea is to never enumerate trajectories. Any trajectory from $s$ is one step, followed by a trajectory from the next state, and the value of “the rest” is $v_\pi$ again, one step later.

Think of the hike. You don’t need to walk every route from $S$ to know what $S$ is worth: if you already know that $A$ is worth $-5$ , the move $S \to A$ is worth $-10 + (-5)$ , and nothing beyond $A$ matters. The value of a state is written with the values of its neighbours, and theirs with their neighbours’, and this is a consistency that the table should verify everywhere:

Using $G_t = R_{t+1} + \gamma G_{t+1}$ and the expectation taken given $S_t = s$ , we obtain:

v_\pi(s) = \mathbb{E}_\pi[R_{t+1} + \gamma\, v_\pi(S_{t+1}) \mid S_t = s]

Written with the policy and the dynamics, first the choice of the action, then the outcome:

v_\pi(s) = \sum_{a} \pi(a \mid s) \sum_{s', r} p(s', r \mid s, a)\, \big[ r + \gamma\, v_\pi(s') \big]

From the inside out: $r + \gamma\, v_\pi(s')$ is what one outcome is worth; the inner sum averages it over what the environment may do; the outer sum averages over what the policy may choose.

This is the Bellman expectation equation. Read it as a picture, a backup diagram: from the state $s$ , the policy branches over the actions; from each action, the environment branches over the outcomes $(s', r)$ ; each leaf is worth its reward plus the discounted value of where it lands. The same reasoning for $q_\pi$ puts the outcome first and the next action second:

q_\pi(s, a) = \sum_{s', r} p(s', r \mid s, a)\, \Big[ r + \gamma \sum_{a'} \pi(a' \mid s')\, q_\pi(s', a') \Big]

Backup diagrams

Open circles are states, filled dots are actions. Left, v_π(s): the policy chooses the action, then the environment chooses the outcome. Right, q_π(s, a): the action is given, the environment answers first, then the policy chooses the next action. The values of the leaves are carried up to the root, one step back: a backup.

The equation turns an infinite sum over trajectories into a one-step, self-referential relation. In a finite MDP, it is one linear equation per state, so $|\mathcal{S}|$ equations for the $|\mathcal{S}|$ unknowns $v_\pi(s)$ : a linear system, which can be solved directly. For $\pi_B$ on our map, with $\gamma = 1$ , one equation per state:

\left\{ \begin{aligned} v_{\pi_B}(S) &= 0.3\, \big[ -10 + v_{\pi_B}(A) \big] + 0.7\, \big[ -40 + v_{\pi_B}(B) \big] \\ v_{\pi_B}(A) &= -5 + v_{\pi_B}(G) \\ v_{\pi_B}(B) &= -35 + v_{\pi_B}(G) \\ v_{\pi_B}(G) &= 0 \end{aligned} \right.

Four equations, four unknowns. Here the system is even triangular: $G$ gives $A$ and $B$ , which give $S$ .

Another way to see it: $v_\pi$ is a fixed point. Replace every value by the one-step look-ahead of its neighbours, the right-hand side of the equation, and the table does not change.

Optional Where the Markov property is used: deriving the Bellman equation

The equation is not as straightforward as it might look: it takes two lines of conditioning, and one assumption.

From $v_\pi$ to $q_\pi$ . Split the expectation along the first action, whose probability given $S_t = s$ is $\pi(a \mid s)$ (law of total expectation): $v_\pi(s) = \sum_a \pi(a \mid s)\, q_\pi(s, a)$ .

From $q_\pi$ to $v_\pi$ . Write $G_t = R_{t+1} + \gamma G_{t+1}$ and split along the outcome $(S_{t+1}, R_{t+1}) = (s', r)$ , whose probability is $p(s', r \mid s, a)$ :

\begin{aligned} q_\pi(s, a) = \sum_{s', r} p(s', r \mid s, a)\, \Big[ r + \gamma\, &\mathbb{E}_\pi\big[G_{t+1} \mid S_t = s, A_t = a, \\ &\phantom{\mathbb{E}_\pi\big[} S_{t+1} = s', R_{t+1} = r\big] \Big] \end{aligned}

The last expectation is where the Markov property enters: given the present state $S_{t+1} = s'$ , the future does not depend on how the agent got there, so this expectation is just $v_\pi(s')$ . Substituting one relation into the other gives both Bellman equations.

Without the Markov property, the value of $s'$ would depend on the path that led to it, “the value of a state” would not be defined, and there would be no Bellman equation: no dynamic programming, and none of the learning methods of Part 3.

Take the altitude map of section 2.2, with $\gamma = 1$ , and the two policies of section 2.3.

By backward substitution from $G$ , where $v_{\pi_A}(G) = 0$ , compute $v_{\pi_A}(A)$ , then $v_{\pi_A}(S)$ , then $v_{\pi_A}(B)$ .
Compute $q_{\pi_A}(S, \text{right})$ , $q_{\pi_A}(S, \text{down})$ , $q_{\pi_A}(A, \text{down})$ and $q_{\pi_A}(A, \text{left})$ .
Compare $q_{\pi_A}(S, \cdot)$ with the values $-15$ and $-75$ of the coin-flipping policy of section 3.1. Are they equal? Why, since the two policies differ?
Compute $v_{\pi_B}(S)$ as an average of action values, $v_\pi(s) = \sum_a \pi(a \mid s)\, q_\pi(s, a)$ . Is it closer to $q(S, \text{right})$ or to $q(S, \text{down})$ ?
Check your result with the Bellman expectation equation of $\pi_B$ at $S$ .

Hint

The map is deterministic and $\gamma = 1$ , so each value is one reward plus the value of where you land: $v_{\pi_A}(A) = -5 + v_{\pi_A}(G)$ . For $q$ , the first action is the one you are given, whatever the policy says.

Hint

$q_\pi(S, a)$ fixes the action at $S$ , then follows $\pi$ from the next state on. Which states does it look at, and what do the policies do there?

Checkpoint what you should get

$v_{\pi_A}(S) = -15$ , $q_{\pi_A}(A, \text{left}) = -25$ , and $v_{\pi_B}(S) = -57$ .

Solution

$v_{\pi_A}(A) = -5 + 0 = -5$ , then $v_{\pi_A}(S) = -10 + v_{\pi_A}(A) = -15$ . $\pi_A$ goes right at $B$ , so $v_{\pi_A}(B) = -35 + 0 = -35$ .
$q_{\pi_A}(S, \text{right}) = -10 + v_{\pi_A}(A) = -15$ and $q_{\pi_A}(S, \text{down}) = -40 + v_{\pi_A}(B) = -75$ . Then $q_{\pi_A}(A, \text{down}) = -5 + 0 = -5$ and $q_{\pi_A}(A, \text{left}) = -10 + v_{\pi_A}(S) = -25$ : going back pays the climb from $S$ to $A$ a second time.
They are equal. By definition, $q_\pi(S, a)$ fixes the action at $S$ : it ignores $\pi(\cdot \mid S)$ and depends on the policy only through what it does at the next states, $A$ and $B$ . There, the two policies agree (down at $A$ , right at $B$ ), so they have the same $q(S, \cdot)$ .
For the same reason, $q_{\pi_B}(S, \cdot) = q_{\pi_A}(S, \cdot)$ , since $\pi_B$ differs from $\pi_A$ only at $S$ . So $v_{\pi_B}(S) = 0.3 \times (-15) + 0.7 \times (-75) = -4.5 - 52.5 = -57$ , much closer to $-75$ : $\pi_B$ puts 70% of its probability on the expensive move.
$v_{\pi_B}(S) = 0.3\, [-10 + v_{\pi_B}(A)] + 0.7\, [-40 + v_{\pi_B}(B)] = 0.3\, [-10 - 5] + 0.7\, [-40 - 35] = -57$ . The same computation as in question 4, expanded one step further, from $q$ into $r + \gamma v(s')$ .

What to notice

$v_\pi(s)$ is an average of the $q_\pi(s, a)$ , so it always lies between the smallest and the largest of them, here between $-75$ and $-15$ . And $q$ compares actions without caring what the policy does in the current state: this is what makes the improvement step of Part 2 (section 2) possible.

4. Optimality: the best policy

Wrapping everything up, how do we define the best strategy? What properties does this strategy satisfy?

4.1 Optimal values and optimal policies

The best achievable values, over all policies, are the optimal value functions:

v_*(s) \doteq \max_\pi v_\pi(s), \qquad q_*(s, a) \doteq \max_\pi q_\pi(s, a)

A policy $\pi_*$ is optimal if $v_{\pi_*}(s) = v_*(s)$ in every state.

That a single policy can be best everywhere at once is not obvious; it is a theorem. For a finite MDP with bounded rewards (and $\gamma < 1$ , or episodes that end), the theorem states that:

there always exists an optimal policy, and at least one of them is deterministic;
it can be read from $q_*$ : $\pi_*(s) = \arg\max_a q_*(s, a)$ ;
$v_*$ and $q_*$ are unique, but optimal policies may not be: when two actions tie for the max, both are optimal.

So once $q_*$ is known, acting optimally is a table lookup, and there is no need to search among stochastic policies.

Acting from $v_*$ is also possible, $\pi_*(s) = \arg\max_a \sum_{s', r} p(s', r \mid s, a)\,[r + \gamma v_*(s')]$ , but it needs the model $p$ ; acting from $q_*$ does not. This is why the model-free methods of Part 3 will learn $q$ rather than $v$ .

The optimal values satisfy their own Bellman equations, the Bellman optimality equations. An optimal agent puts all its probability on a best action, so the average over the actions becomes a max:

v_*(s) = \max_{a} \sum_{s', r} p(s', r \mid s, a)\, \big[ r + \gamma\, v_*(s') \big]

q_*(s, a) = \sum_{s', r} p(s', r \mid s, a)\, \Big[ r + \gamma \max_{a'} q_*(s', a') \Big]

No policy is left in these equations: the max is the optimal policy.

Note what changed and what did not: max over what the agent controls (the action), expectation over what it does not (the outcome).

On our 2×2 map, with $\gamma = 1$ , one equation per state again, each value tied to those of its neighbours:

\left\{ \begin{aligned} v_*(S) &= \max\big( -10 + v_*(A),\ -40 + v_*(B) \big) \\ v_*(A) &= \max\big( -5 + v_*(G),\ -10 + v_*(S),\ -1 + v_*(A) \big) \\ v_*(B) &= \max\big( -35 + v_*(G),\ -40 + v_*(S),\ -1 + v_*(B) \big) \\ v_*(G) &= 0 \end{aligned} \right.

Because of the max, the system is no longer linear, and there is no closed-form solution in general. Part 2 (sections 1 to 3) solves it by iteration; on this small map, Exercise 4.1a solves it by hand.

The optimal policy also depends on $\gamma$ . Take a task that never ends, where the agent only decides in one state, $s_0$ : left pays $+1$ now and $0$ on the way back, right pays $0$ now and $+2$ on the way back. Which loop is better?

The discount factor decides the optimal policy

Left: the two loops from s₀, the optimal one highlighted. Right: the value of s₀ under each policy, as a function of γ. Move γ: a short-sighted agent takes the +1 now, a patient one waits for the +2.

On the altitude map, with $\gamma = 1$ :

At $A$ , three actions are available: down, left and stay. Explain why stay and left are never optimal, and deduce $v_*(A)$ . Do the same at $B$ and deduce $v_*(B)$ .
Deduce $v_*(S)$ , by comparing right and down from $S$ .
Plug the three values back into the Bellman optimality equations at $S$ , $A$ and $B$ , and check that they hold.
Deduce $\pi_*$ . Compare it with $\pi_A$ and $\pi_B$ .

Hint

Start from the equation of $A$ in the system above, $v_*(A) = \max\big( -5 + v_*(G),\ -10 + v_*(S),\ -1 + v_*(A) \big)$ . Can the last term ever be the max, for a finite $v_*(A)$ ?

Hint

For left: from $S$ , going down costs $-40$ and every later reward is $\le 0$ , so it is worth at most $-40$ ; going right is worth $-10 + v_*(A)$ , which is at least $-15$ (go down to $G$ from $A$ ). So $v_*(S) = -10 + v_*(A)$ . What does that give for $-10 + v_*(S)$ ?

Checkpoint what you should get

$v_*(S) = -15$ , $v_*(A) = -5$ , $v_*(B) = -35$ .

Solution

Stay gives $-1 + v_*(A)$ , which is strictly less than $v_*(A)$ : it can never be the max. (Staying forever would return $-1 - 1 - \cdots = -\infty$ .) Left gives $-10 + v_*(S) \le -10 + (-10 + v_*(A)) = -20 + v_*(A) < v_*(A)$ : going back pays the climb between $S$ and $A$ twice, for nothing. Only down is left: $v_*(A) = -5 + 0 = -5$ . At $B$ , stay loses for the same reason, and up gives $-40 + v_*(S) \le -55$ ; right gives $-35$ , so $v_*(B) = -35$ .
$v_*(S) = \max(-10 + v_*(A),\ -40 + v_*(B)) = \max(-15, -75) = -15$ , reached by right.
With $v_*(S) = -15$ :
- $v_*(A) = \max(-5 + 0,\ -10 - 15,\ -1 - 5) = \max(-5, -25, -6) = -5$ ;
- $v_*(B) = \max(-35 + 0,\ -40 - 15,\ -1 - 35) = \max(-35, -55, -36) = -35$ ;
- $v_*(S) = \max(-10 - 5,\ -40 - 35) = -15$ .
$v_*$ is a fixed point of the optimality equations: one more update changes nothing.
$\pi_*(S) = \text{right}$ , $\pi_*(A) = \text{down}$ , $\pi_*(B) = \text{right}$ : this is $\pi_A$ , and indeed $v_{\pi_A}(S) = -15 = v_*(S)$ (Exercise 3.2). It is deterministic, as the theorem promises, and unique here because no two actions tie. The stochastic $\pi_B$ only gets $-57$ .

What to notice

Mixing in a bad action never helps once you know $q_*$ : randomness is useless for acting in a known MDP. It becomes useful for learning, when the agent must try actions to find out what they are worth (Part 3).

A continuing task: from the state $s_0$ , the agent chooses left, which pays $+1$ and leads to $s_\text{left}$ , or right, which pays $0$ and leads to $s_\text{right}$ . From $s_\text{left}$ , the only action returns to $s_0$ with reward $0$ ; from $s_\text{right}$ , the only action returns to $s_0$ with reward $+2$ . The agent decides only in $s_0$ : there are two deterministic policies, $\pi_\text{left}$ and $\pi_\text{right}$ .

Write the infinite reward sequence seen from $s_0$ under each policy.
Show that $v_{\pi_\text{left}}(s_0) = \frac{1}{1 - \gamma^2}$ and $v_{\pi_\text{right}}(s_0) = \frac{2\gamma}{1 - \gamma^2}$ .
Which policy is optimal for $\gamma = 0$ ? For $\gamma = 0.9$ ? For $\gamma = 0.5$ ?
For which $\gamma$ are the two policies equally good?

Hint

Each policy cycles through $s_0$ every two steps. Group the rewards by pairs: the sum is a geometric series of ratio $\gamma^2$ .

Hint

Or, with the Bellman equation over one cycle: under $\pi_\text{left}$ , $v(s_0) = 1 + \gamma\,(0 + \gamma\, v(s_0))$ .

Checkpoint what you should get

Left is optimal at $\gamma = 0$ , right at $\gamma = 0.9$ ( $\frac{180}{19}$ against $\frac{100}{19}$ ), and they tie at $\gamma = \frac12$ , both worth $\frac43$ .

Solution

$\pi_\text{left}$ : $1, 0, 1, 0, \dots$ and $\pi_\text{right}$ : $0, 2, 0, 2, \dots$
Under $\pi_\text{left}$ , the reward is $1$ at the even steps and $0$ at the odd ones: $v_{\pi_\text{left}}(s_0) = \sum_{k \ge 0} \gamma^{2k} = \frac{1}{1 - \gamma^2}$ . Under $\pi_\text{right}$ , $2$ at the odd steps: $v_{\pi_\text{right}}(s_0) = \sum_{k \ge 0} 2\gamma^{2k + 1} = \frac{2\gamma}{1 - \gamma^2}$ . With the Bellman equation: $v = 1 + \gamma^2 v$ and $v = 2\gamma + \gamma^2 v$ give the same results.
Since $1 - \gamma^2 > 0$ $1 - γ^{2} > 0$ , compare the numerators, $1$ $1$ and $2\gamma$ $2 γ$ :
- $\gamma = 0$ : $1$ against $0$ , left is optimal (only the next reward counts);
- $\gamma = 0.9$ : $\frac{100}{19} \approx 5.26$ against $\frac{180}{19} \approx 9.47$ , right is optimal;
- $\gamma = 0.5$ : both are worth $\frac43$ , an exact tie.
$\frac{1}{1 - \gamma^2} = \frac{2\gamma}{1 - \gamma^2} \iff 1 = 2\gamma \iff \gamma = \frac12$ .

What to notice

In section 1.3, $\gamma$ only reweighted the rewards of a fixed trajectory. Here it changes the argmax: which policy is optimal. $\gamma$ is not a tuning knob, it is part of the problem statement, like the rewards. Check your answers on the figure above.

4.2 How to choose a reward? The rewards are part of the problem

All the rewards of the altitude map are negative. Is that an arbitrary choice? Why not add a constant to make them positive, and maximize a “score” instead of minimizing an effort? The answer differs between continuing and episodic tasks.

Fix a constant $c$ and shift every reward of an MDP: $R'_t = R_t + c$ . Write $v'_\pi$ for the values in the shifted MDP.

Continuing task, $\gamma < 1$ . Show that $v'_\pi(s) = v_\pi(s) + v_c$ , with a constant $v_c$ that depends neither on $s$ nor on $\pi$ . Give $v_c$ . Does the shift change which policy is optimal?
Episodic task, $\gamma = 1$ . Let $T$ be the (random) time at which the episode ends. Show that $v'_\pi(s) = v_\pi(s) + c\, \mathbb{E}_\pi[T \mid S_0 = s]$ . Why is this different from question 1?
On the altitude map ( $\gamma = 1$ ), shift every reward by $c = +11$ . Compare the return from $S$ of $\pi_A$ (two steps to $G$ ) with the return of the policy that goes to $A$ , plays stay $k$ times, then goes to $G$ . What is $v'_*(S)$ ?
For which values of $c$ does the shifted map keep the same optimal behaviour, reaching $G$ along the cheapest route?

Hint

Question 1: $G'_t = \sum_k \gamma^k (R_{t+k+1} + c)$ ; split the sum in two.

Hint

Question 3: with the shift, a step to $A$ pays $-10 + 11 = +1$ , the last step $-5 + 11 = +6$ , and each stay $-1 + 11 = +10$ .

Hint

Question 4: look at what one more stay adds to the return, $-1 + c$ , and what the last step costs, $-5 + c$ . Be careful with the policy that stays at $A$ forever.

Checkpoint what you should get

$v_c = \frac{c}{1 - \gamma}$ ; with $c = 11$ , $\pi_A$ returns $7$ and dawdling $k$ times returns $7 + 10k$ , so $v'_*(S) = +\infty$ ; the optimal behaviour survives only for $c < 1$ .

Solution

$G'_t = \sum_{k \ge 0} \gamma^k R_{t+k+1} + c \sum_{k \ge 0} \gamma^k = G_t + \frac{c}{1 - \gamma}$ , on every trajectory. Taking the expectation, $v'_\pi(s) = v_\pi(s) + v_c$ with $v_c = \frac{c}{1 - \gamma}$ , the same for every state and every policy, because every trajectory is infinitely long. All policies move by the same amount: their ranking, and $\pi_*$ , do not change. In a continuing task, only the differences between rewards matter, not their sign.
Now $G'_0 = \sum_{t=1}^{T} (R_t + c) = G_0 + c\,T$ , so $v'_\pi(s) = v_\pi(s) + c\, \mathbb{E}_\pi[T \mid S_0 = s]$ (for policies that end the episode). The shift depends on the policy, through the expected length of the episodes it produces: adding $c$ to every reward puts a price on time, $c$ per step, and policies differ in how much time they take.
$\pi_A$ : $G'_0 = 1 + 6 = 7$ (check: $-15 + 2 \times 11 = 7$ ). Staying $k$ times at $A$ : $G'_0 = 1 + 10k + 6 = 7 + 10k$ (check: $-15 - k + 11(k + 2) = 7 + 10k$ ). Each extra stay now pays $+10$ : the longer the agent dawdles, the better, without limit, so $v'_*(S) = +\infty$ and no policy is optimal. The least-effort problem has become “stay out as long as possible”.
Each extra stay adds $-1 + c$ :
- $c < 1$ : staying still costs something, the reasoning of Exercise 4.1a still holds, and $\pi_*$ is unchanged, with $v'_*(S) = -15 + 2c$ ;
- $c = 1$ : staying is free. Dawdling a finite number of times ties with $\pi_A$ , at $-13$ , but parking at $A$ forever returns $-9$ and never pays the $-5 + 1 = -4$ of the last step: the optimal agent never reaches $G$ ;
- $c > 1$ : staying pays, and $v'_*(S) = +\infty$ .
The map stays a least-effort problem only for $c < 1$ . This is why stay costs $-1$ : a free stay is exactly the case $c = 1$ .

What to notice

In an episodic task, the level of the rewards is part of the specification: a reward of $0$ per step says “time is free”, a negative one says “hurry”, a positive one says “take your time”. A robot rewarded $+1$ at the goal and $0$ elsewhere, with $\gamma = 1$ , scores the same whether it arrives fast or slowly.

4.3 Some useful vocabulary

Solving the Bellman optimality equation is the whole problem: from $v_*$ or $q_*$ , the optimal policy is found from it. We might solve it from different approaches in various cases.

Each one answers a question with one of two opposite terms:

The vocabulary to understand an algorithm

What does it compute?

Prediction: evaluate a given policy, find v_π

Control: find the best policy, v_* and π_*

Does it use the model p?

Model-based: has p, or learns it

Model-free: does without p

What does it learn?

Value-based: v or q, then acts greedily

Policy-based: π directly. Actor-critic methods learn both.

Whose data does it learn from?

On-policy: the policy it evaluates and improves

Off-policy: another policy, the one that behaves

When does it update?

Online: at every step, while interacting

Offline: afterwards, at the end of an episode or from stored data

How does a learning agent pick its action?

Exploit: take the best action it knows, and collect reward now

Explore: try another one, and learn something for later

Every algorithm of the course can be placed on each of the two axes.

Let’s focus a bit on the last line. Every agent faces a trade-off that a planner does not: explore to find out more about the environment, or exploit what it already knows to collect reward. It is a trade-off, not a choice made once: an agent that only exploits can settle on a mediocre action it never questions, and one that only explores never uses what it has learned. A good learner explores a lot at first, and less as it learns.

Explore or exploit

The second miner leaves his own hole to exploit the first one, and never finds the diamond just below where he stopped. Reward now weighs against knowledge for later: too much of either loses.

4.4 Wrap-up: the 2×2 map, solved

Everything on this page fits on our four-state map. Pick a policy, or build your own one state at a time, and read its values: each one is the Bellman expectation equation of its state, numbers included.

Click on a state to see the values or change its policy:

The map, solved

The value of each state under the chosen policy, computed from the Bellman expectation equations, and the optimal value v_* below it. Select a state to see its backup with numbers and to change its action. The line below the map says whether the policy is optimal, and if not, which single change would improve it.

What to look at in the figure above

π_A reaches $v_*$ everywhere: it is optimal, as in Exercise 4.1a. π_B loses $42$ at $S$ by going down 70% of the time.
Stay when you can, with $\gamma = 1$ : $A$ and $B$ are worth $-\infty$ , the hiker never reaches $G$ . Lower $\gamma$ and the values become finite: discounting makes an endless task finite, as in section 1.3.
The optimal policy depends on $\gamma$ . Staying forever at $A$ is worth $-\frac{1}{1-\gamma}$ , against $-5$ for going down to $G$ : below $\gamma = 0.8$ , the optimal hiker parks at $A$ instead of finishing. At $B$ , where finishing costs $35$ , this happens below $\gamma \approx 0.97$ . A short-sighted agent finds the end of the hike too expensive.

An eye to what’s next 👀

On four states, everything can be solved by hand. But in real problems, solving the Bellman equations exactly is rarely possible. It needs the dynamics, which robots, cars or robots do not hand out; the computing power to handle every state, while Go has about $10^{170}$ legal positions; and the Markov property.

The rest of the course is about approximate solutions, learned from interaction. We’ll first study problems with a model (model-based) before switching to problems with no access to a model (model-free).

Part 2, Planning with a Model, does find approximate solutions on any finite MDP, with iterative algorithms: evaluate a policy, improve it toward the optimal value function, and repeat, until the policy is optimal.

Extra exercises

Four one-line relations, used constantly from Part 2 on. Give each, and justify it in a sentence.

$v_*$ in terms of $q_*$ .
$q_*$ in terms of $v_*$ and the dynamics $p(s', r \mid s, a)$ .
$\pi_*$ in terms of $q_*$ .
$\pi_*$ in terms of $v_*$ and $p$ .
Compare 3 and 4: to act optimally from $q_*$ , what must you know about the environment? And from $v_*$ ?

Hint

An optimal agent puts all its probability on a best action: the average $\sum_a \pi(a \mid s)\, q_*(s, a)$ of section 3.1 collapses to a max.

Hint

For 2, the agent chooses $a$ , but not the outcome: the sum over $(s', r)$ stays an expectation.

Checkpoint what you should get

$v_*(s) = \max_a q_*(s, a)$ , and $\pi_*(s) \in \arg\max_a q_*(s, a)$ : acting from $q_*$ needs no model.

Solution

$v_*(s) = \max_{a} q_*(s, a)$ : an optimal agent puts all its probability on a maximizing action, so the average over the actions becomes a max.
$q_*(s, a) = \sum_{s', r} p(s', r \mid s, a)\, \big[ r + \gamma\, v_*(s') \big]$ : take $a$ , then act optimally. The sum over $(s', r)$ stays an expectation, since the agent does not choose the outcome.
$\pi_*(s) \in \arg\max_{a} q_*(s, a)$ . Membership, not equality: when several actions tie, any choice among them, or any mixture of them, is optimal.
Substituting 2 into 3: $\pi_*(s) \in \arg\max_{a} \sum_{s', r} p(s', r \mid s, a)\, \big[ r + \gamma\, v_*(s') \big]$ .
From $q_*$ , nothing: read the row $q_*(s, \cdot)$ , take the largest entry, act. From $v_*$ , the model $p$ , to look one step ahead for each candidate action before comparing them.

What to notice

$q$ is what you can act on without a model. Dynamic programming (Part 2) may work with $v$ , because the model is given; Monte Carlo, SARSA and Q-learning (Part 3) never know $p$ , so they estimate $q$ .

Much of the literature does not use $p(s', r \mid s, a)$ , but the two functions of section 2.2, $p(s' \mid s, a)$ and $r(s, a)$ .

Show that, for any function $v$ of the states, $\sum_{s', r} p(s', r \mid s, a)\, \big[ r + \gamma\, v(s') \big] = r(s, a) + \gamma \sum_{s'} p(s' \mid s, a)\, v(s')$ Which property of the bracket makes this work?
Deduce the four Bellman equations, for $v_\pi$ , $q_\pi$ , $v_*$ and $q_*$ , written with $p(s' \mid s, a)$ and $r(s, a)$ only.
In one sentence: what is lost, and what is gained, in this notation?

Hint

Split the sum in two. In the second term, $v(s')$ does not depend on $r$ : it comes out of the sum over $r$ .

Checkpoint what you should get

The bracket is affine in $r$ and in $v(s')$ . Only the mean of the reward survives: its distribution is lost.

Solution

Split the sum: $\sum_{s', r} p(s', r \mid s, a)\, r + \gamma \sum_{s'} \Big( \sum_{r} p(s', r \mid s, a) \Big) v(s') = r(s, a) + \gamma \sum_{s'} p(s' \mid s, a)\, v(s')$ The first sum is $r(s, a)$ by definition; in the second, $v(s')$ does not depend on $r$ , and summing $r$ out of $p(s', r \mid s, a)$ gives $p(s' \mid s, a)$ . It works because the bracket is affine in $r$ and in $v(s')$ .
Applying 1 to each equation: $\begin{aligned} v_\pi(s) &= \sum_{a} \pi(a \mid s)\, \Big[ r(s, a) + \gamma \sum_{s'} p(s' \mid s, a)\, v_\pi(s') \Big] \\ q_\pi(s, a) &= r(s, a) + \gamma \sum_{s'} p(s' \mid s, a) \sum_{a'} \pi(a' \mid s')\, q_\pi(s', a') \\ v_*(s) &= \max_{a} \Big[ r(s, a) + \gamma \sum_{s'} p(s' \mid s, a)\, v_*(s') \Big] \\ q_*(s, a) &= r(s, a) + \gamma \sum_{s'} p(s' \mid s, a)\, \max_{a'} q_*(s', a') \end{aligned}$
Gained: compactness, sums over the states only, and $r(s, a)$ is often what a simulator hands out. Lost: everything about the distribution of the reward, of which only the mean survives.

What to notice

Losing the distribution is harmless here: the Bellman equations depend on the reward only through its expectation. It is fatal as soon as you care about risk or about the spread of the return, which is why risk-sensitive and distributional RL keep the four-argument form.

The dynamics of section 2.2 satisfy $\sum_{s', r} p(s', r \mid s, a) = 1$ . In an episodic task, which set must $s'$ range over for this to hold? Where does the agent land on the last step of an episode?
Pole-balancing, as an episodic task with discounting: every reward is $0$ , except $-1$ when the pole falls, which ends the episode. Let $K$ be the number of steps from $t$ until the fall. Give $G_t$ .
The same task as a continuing one: the pole is put back up after each fall, and the task never ends. The falls happen $K_1 < K_2 < \cdots$ steps after $t$ . Give $G_t$ , and say how it differs from question 2.
Check in both cases that maximizing the return means “balance the pole as long as possible”.

Hint

Question 2: a single reward is nonzero, $R_{t+K} = -1$ , and it carries the weight $\gamma^{K-1}$ .

Checkpoint what you should get

Episodic: $G_t = -\gamma^{K-1}$ . Continuing: $G_t = -\sum_{i \ge 1} \gamma^{K_i - 1}$ .

Solution

Over the states and the terminal state. On the last step, the agent lands in the terminal state; leaving it out of the sum would lose that probability, and the total would fall short of $1$ . The condition still only concerns the non-terminal $s$ , since no action is taken from a terminal state: the agent acts from the non-terminal states and lands anywhere, terminal included.
Only the fall contributes: $G_t = -\gamma^{K-1}$ (a fall at the very next step, $K = 1$ , gives $-1$ ).
Every future fall contributes, each discounted by how far away it is: $G_t = -\sum_{i \ge 1} \gamma^{K_i - 1}$ . The episodic return only sees the next fall, what comes after belongs to another episode; the continuing return adds up the whole infinite future.
In both cases the return is a sum of terms $-\gamma^{K-1}$ , which increase with $K$ when $\gamma < 1$ : pushing every fall further away increases the return, and an agent that never lets the pole fall reaches the best possible return, $0$ .

What to notice

Here the rewards are $0$ or negative and $\gamma < 1$ , so “maximize the return” and “last as long as possible” coincide. Exercise 4.2 showed how easily this breaks when the level of the rewards changes.

No computation, and no single answer.

Is the MDP framework enough to represent every goal-directed learning task? Propose at least two counterexamples, and say in each case what exactly breaks.
Consider driving a car. The actions could be the accelerator, the steering wheel and the brakes; or one level below, the driver’s muscles; or one level above, the choice of a route. Where should the boundary between agent and environment be drawn? Is there a single right answer? What rule of thumb would you propose?

Solution

Some directions, not a complete answer.

The state may not be observable: a poker player does not see the other hands, and the observation is not Markov (section 2.1). The goal may not fit in one number: several objectives that cannot be traded against each other, or constraints that must never be violated. Other agents may learn at the same time, so the dynamics are no longer stationary. And the reward may simply not be available, or be too sparse to learn from (section 1.2).
There is no single answer; Sutton and Barto’s rule is that the boundary separates what the agent can change arbitrarily from what it cannot. Anything it cannot control directly belongs to the environment, even its own body. In practice, the boundary is chosen at the level where the decisions of interest are taken: the route for navigation, the pedals and the wheel for driving, the muscles for motor control.

Modeling the World

Author

Affiliations

Published

Introduction

1. Sequential decisions

1.1 Decisions in stages

1.2 Optimal means optimal for a score

1.3 The return and the discount factor

2. Markov decision processes

2.1 The Markov property

2.2 The model of the world

2.3 Policies

3. Value functions and the Bellman equation

3.1 How good is a state, how good is an action

3.2 The Bellman expectation equation

4. Optimality: the best policy

4.1 Optimal values and optimal policies

4.2 How to choose a reward? The rewards are part of the problem

4.3 Some useful vocabulary

4.4 Wrap-up: the 2×2 map, solved

Extra exercises