Welcome to discuss about this book here!

[MathFoundationRL]

Hi there,

If you have any feedback about the book, you can leave a comment here. Thanks!

[etbox]

There excesses a right parentheses in the equation.

[MathFoundationRL]

Noted and thanks a lot!

[eccstartup]

In slides 9, page 22. I guess this is `exercise`.

[MathFoundationRL]

Yes you are right. Thanks for letting me know.

Shiyu

[L3Y1Q2]

Great book and course! Helped me reorganize some of the points of RL.

In the process of reading this book (ver 2022.8), I met some little confusion, probably clerical errors.

Thanks!

[MathFoundationRL]

Thanks for your detailed feedback.

• page 29: you are right.
• page 64: you are right
• page 91: you are right
• page 110 and more: could you specify more specifically?
• page111: you are right
• page128: you are right
• page133 and more: you are right

Overall, it's my pleasure that you read very carefully. Thanks for your feedback.

Shiyu

[L3Y1Q2]

Page 110:

Page 112:

[MathFoundationRL]

noted and thanks!

[businiaon]

if there should be a1, while a2 represents rightwards and a1 represents upwards.

[MathFoundationRL]

You are right. They should be a1. Thanks for the feedback!

[businiaon]

page 27, from Figure 2.3 if there should be p(r = 0|s1, a3) = 1 and p(r<> 0|s1, a3) = 0, not p(r = −1|s1, a3) = 1 and p(r <> −1|s1, a3) = 0 ?

[MathFoundationRL]

Yes, here r should be or not be equal to 0 (not -1)

[eccstartup]

第6章，page 20，这个好像不收敛的。

w = 0
g = lambda w: w**3 - 5

import random
for i in range(100):
    print(i, w)
    w = w - 1/(i+10) * (g(w) + random.gauss(0, 1))

[eccstartup]

a=1/k总是溢出

[ZiqianXie]

g的梯度是无界的, 收敛性用RM定理没法保证

[MathFoundationRL]

如果各位英文写作没问题的话，还请用英文，这样可以维护一个大家都能看的讨论环境，本书也有很多国外的读者，谢谢。

[MathFoundationRL]

要想在全局收敛，需要g的梯度有上界。

[ZiqianXie]

Dvoretzky 定理的证明只包括了 $\alpha_k, \beta_k$ 都由 $\mathcal{H}_k$ 决定的情况, 最好额外说明一下.

[MathFoundationRL]

谢谢，我会做相应修改

[MathFoundationRL]

如果各位英文写作没问题的话，还请用英文，这样可以维护一个大家都能看的讨论环境，本书也有很多国外的读者，谢谢。

[fanshJ]

6.2 Page 107
式子左边的 w_∞ − w_1 是不是应该改为 w_1 - w_∞，或者右边加负号？

[MathFoundationRL]

是的，谢谢

[MathFoundationRL]

如果各位英文写作没问题的话，还请用英文，这样可以维护一个大家都能看的讨论环境，本书也有很多国外的读者，谢谢。

[ritsuki1227]

Thank you very much for writing such a clear and helpful book!
Could you explain why the inequality shown on the image (bottom of the page 48) implies the contraction property of the Bellman Optimality Equation?
Maybe I don't understand some properties of the sup-norm.

[MathFoundationRL]

This inequality is the same as the definition of a contraction mapping. It shows that the function f(v) is contractive, (not the Bellman optimality equation is contractive).

[ritsuki1227]

Thank you for your comment. My understanding is that the definition states that $f$ is contractive iff

$$||f(v_1) - f(v_2)||\leq \gamma ||v_1 - v_2|| \quad \exists \gamma \in (0, 1) \forall v_1, v_2 \in \mathbb{R}^d,$$

which is not obviously the same as the inequality in the image above for me ($||.||$ denotes vector norm whereas $||.||_\infty$ denotes supremum norm). I guess I'm missing some basic concepts of sup-norm, but is it true that the implication below holds?

$$||f(v_1) - f(v_2)||_\infty \leq \gamma ||v_1 - v_2||_\infty \implies ||f(v_1) - f(v_2)|| \leq \gamma ||v_1 - v_2|| .$$

Also, I've found a minor typo on the section 3.3.3 on page 44. "for any $x_1, x_2 \in \mathbb{R}$" should be "for any $x_1, x_2 \in \mathbb{R}^d$".

[MathFoundationRL]

Now I understand. First, in the definition of contraction mapping, $||\cdot||$ refers to a general vector norm. That is, if there exists any type of vector norm so that this inequality is valid, then the function is contractive. Second, in the proof it is a special $L_\infty$ vector norm (it is still a vector norm). Third, in Theorem 3.2, the norm should also be the $L_\infty$ norm (but I wrote it as a general norm). Hope this answers your question. I will also revise to make it clearer.

You are right about the typo. Thanks!

[ritsuki1227]

Thank you very much. It is clear to me now!

[ZiqianXie]

Some notation inconsistency:

Page 196, the matrix-vector form should have $\nabla_\theta v_\pi\in\mathbb{R}^{m|S|}$ instead of $n|S|$ since $m$ is the dimension of $\theta$.

And later in 9.12, $n$ is being used to denote $|S|$

[MathFoundationRL]

You are right. Thanks!

By the way, I didn't expect but am happy to see that some readers reads the proof details carefully.

[bayncat]

Hello. Is there a missing "αk" before the summation symbol?

[MathFoundationRL]

Yes, you are right. Thanks!

[SakuraToErii]

I think this "not" may need to be deleted.

[SakuraToErii]

Sorry, I forgot to say it is on page 88.

[MathFoundationRL]

Yes, you are right. It will be corrected in the next version. Thanks

[Rino-Li]

In page. 85, the second paragraph of the subsection A comprehensive example: Episode length and sparse reward, "See, for example, Figure 5.3(h)" should be "5.3(a)", because you mentioned that the episode length is 1.

[Rino-Li]

It is a perfect course to study, thanks to teacher Zhao.

[MathFoundationRL]

Thanks. What I meant is actually correct, but it may be confusing. I will revise it in the future.

[SakuraToErii]

On page 214, this "z" may need to be modified as "x".

[MathFoundationRL]

Thanks. This is a new typo that has not been noticed so far. I will correct it.

[SakuraToErii]

On page 157, I think the "column" may need to be modified as "row".

[MathFoundationRL]

Thanks for the feedback. But this statement is correct. The ith row is the probabilities of reaching state i when starting every state.

[SakuraToErii]

Thank you and I have another question: All of this means that [P_π]ij is from state i to state j, in other words, i is the row and j is the column.......

[MathFoundationRL]

Now I understand. It is correct to say "column", but the meaning of the entries in the ith column is the probabilities reaching state i from different starting states. Thanks for point out the problem! You read very carefully!

[SakuraToErii]

Thank you for your patience. I read your book carefully due to its high quality and clear context. I am very grateful that I have been learning with such a cool book in RL.

[MathFoundationRL]

Thank you! I am finalizing the manuscript of the book. The final version will be better than the first draft. I will it will be cooler:) Please stay tuned.

[renjieDLUT]

"the action value of s_t"我理解应该是state value吧

[MathFoundationRL]

Yes, it should be state value. Thanks.

[xuxiaolin0926]

163页 $s_{2}$ 与 $s_{3}$ 的typo。

感谢您的书。

[MathFoundationRL]

Yes, s2 should be s3. Thanks.

[Annihillusion]

Thank you so much for writing such a helpful book!
Could you further explain why this equation hold? It may not seen so trivial to me. Thank you in advance!

What's more, in Page 205, it might be a little mistake.

[MathFoundationRL]

Hi there, for the first underlined equation: d^TP=d^T, this is due the property of the stationary distribution d. You can check more about d in the book.

About the second circled equation: \sum d(s)=1. This is why it holds.

[Annihillusion]

Thank you so much for your prompt and patient reply! I understand these two equations now.

[yapeg]

Page 172. Chapter 8. Algorithm 8.1.
'A function $\hat{v}(s,w)$ that is $\cancel{a}$ differentiable in $w$.'

[MathFoundationRL]

Thanks, it is indeed a typo

[HaomingFu]

Page 53, Section 3.3.3, Box 3.1. I think there is a typo.
In the sentence "We prove that the consequence $\{x_k\}_{k=1}^\infty$ with $x_k = f(x_{k=1})$ is convergent", the "consequence" should be "sequence".

[MathFoundationRL]

Thank you. It is indeed a typo.

[wyanyue]

I think there's a symbol \gamma missing here. There's the same typo in slides.

[MathFoundationRL]

Thank you. It is indeed a typo!

[lhz99]

print error
on the right graph, I think it is a “SGD(m=1)" not “SG(m=1)"

[L-lorish]

it is actually Pr(s| s') because in following proof, the author has change s to s' and s' to s, equivalently.

[lhz99]

But why need to change here? I'am confused

[MathFoundationRL]

It is mainly for the notational convenience.

[lhz99]

row 2 and 3 in the derivation of tr[var(X)] is repeated

[MathFoundationRL]

Many thanks for the feedback!

[L-lorish]

I am confused of the result of derivative.if adding the explanation is much better!

[L-lorish]

The derivative formula may be derived or inspired by the result.lol

[MathFoundationRL]

(10.17) is a result of a standard derivative operation. If you have any specific question, you can ask.

[L-lorish]

I have delved further into the matter and discovered that this derivative is indeed correct. However, upon my first look, I found myself puzzled by the derivative of $q_{\mu}$. Later, I came to understand that this confusion arose because initially, $v_{\mu}(s)$ had not been expanded.

$q_{\mu}(s,\mu(s)) = \sum_{r}P(r|s,\mu(s))r + \gamma \sum_{s^{\prime}}P(s^{\prime}|s,\mu(s))v_{\mu}(s^{\prime})$

Combined with $v_{\mu}(s) = q_{\mu}(s,\mu(s))$, it clearly indicates that $q_{\mu}(s)$ is a function of both $q_{\mu}$ and $\mu(s)$

[lhz99]

There is total derivative, right?

[Abner-fu]

[MathFoundationRL]

Could you provide more derivation details so that I can help to check it quickly?

[Abner-fu]

Here are my derivation details. If I have any problems, please tell me. Thank you.

[MathFoundationRL]

I think you got the same result as mine, is it? Also, did you change your question? Now your answer in your question is the same as my result

[Abner-fu]

I figured it out. I didn't notice this formula before. $\sum_{a \in A} \sum_{r \in R} \pi(a \mid s) P(r \mid s, a) \beta=\beta $.
It's my problem. Thank you very much.

[Eternity-Wang]

Page 207. Chapter 9. Here I think the π(a|s,θ) should take values in the interval (0,1) due to the softmax function, not [0,1].

[MathFoundationRL]

It is indeed more accurate to use the open interval. Thanks.

[FBwenoll]

Prof. Zhao, when will this book be published?

[MathFoundationRL]

Thanks for asking. It has been sent to the printing factory. Hopefully it can be officially published in July 2024 by Tsinghua University Press (for mainland of China). The international version will be printed by Springer probably in the first half of 2025.

[HaomingFu]

In Section 3.6, Page 64, book ver. March 2024, it says:
"In particular, a policy is optimal if its state values are greater than or equal to those of any other policy."
Based on Theorem 3.4, can I make a stronger statement like:
"In particular, a policy is optimal if and only if its state values are greater than or equal to those of any other policy?"

[MathFoundationRL]

It is actually a definition instead of a conclusion or statement. It may be better to use "when" rather than "if", but it is a convention to use "if" in definitions as well.

[HaomingFu]

Let me explain my stance a bit more. My question may seem a bit trivial, but it is not uncommon that an optimal policy may not exist at all because it is possible that for any policy $\pi$ it achieves the best state values for some states, while there does not exist a policy that achieves the best for all states. After learning Theorem 3.3 and 3.4, I know this possibility can be ruled out and that an optimal policy indeed exists. I want to confirm that my understanding is correct. So that is why I am so obsessed with the problem that I asked about.

Anyways, thanks to you Prof. Zhao. You have already answered my question and dispelled my doubts.

[co-gy]

maybe it's $q_t(s_t, a_t)$ here? (the subscript t is missing)

[MathFoundationRL]

Thanks. There should be a subscript here.

[L-lorish]

Prof. Zhao, can u add more algorithm such as TRPO, PPO, SAC... into this book, though u have mentioned part of that basic mathematical knowledge in some chapters. VERY THANKS.

[MathFoundationRL]

Hi there,

Thanks. Quite a few readers sent me suggestions like that. At this moment, I do not have the plan to do that due to my very limited time and there are quite many materials online:)

[qiufengyuyi]

Prof. Zhao，i find the lecture video of A2C algo is different from the algo code on the e-book.
the lecture ppt shows the critic before the actor ,whereas the e-book shows the actor before the critic.
i thihk the actor should be conducted before the critic ,because the advange will change after the critic, which will lead to unstable update in one training step. please correct me if i'm wrong.
the lecture ppt slide:

the e-book :

[MathFoundationRL]

Hi there. If you see any difference, the book and the updated ppt slides are the most appropriate. For this algo, since the actor and critic are independent, updating whichever does not matter. The advantage does not change. Hope it helps.

[qiufengyuyi]

my bad! i thought after the w was updated ,the TD error will be calculated again. actually ,the TD error will just cal once in one step

[unlikezy]

Great book and video! Truly grateful that we have such excellent Professor and learning materials in China.

I'd like to comment on the setup of grid world used in this book.
I think readers would most likely assume that in this game, the agent should avoid forbidden grids and find the shortest path to the target. However, this assumption is wrong.

As shown in the screenshot below, let's focus on the bottom-left corner. If we use this state-value, and try to do one step of policy improvement, we would find that the optimal action is to turn right, directly step into forbidden grid and get to target as soon as possible.

related video: https://www.bilibili.com/video/BV1Le411K7qY?t=652.0

Calculation:

q(s1, ↑) =  0+0.9*2.5 = 2.25
q(s1, →) = -1+0.9*9.0 = 7.1
q(s1, ↓) = -1+0.9*2.3 = 1.07
q(s1, ←) = -1+0.9*2.3 = 1.07
q(s1, o) = 0+0.9*2.3 = 2.07

The reason is that the game is

1. a continuing task, i.e. not episodic.
2. the agent will receive a reward indefinitely once it reaches the target and stays there.

So the punishment for stepping into forbidden grid would be highly compensated if agent could get to the target quickly.

Though everything is still correct if the reader understands the setup, I feel that this might mislead some readers' intuitions. Because readers subconsciously might always expect the optimal policy is the shortest path avoiding forbidden grid.

I guess that Professor may have done this on purpose in order to illustrate core idea more simply, as a compromise. But it's still great to hear and confirm that from Professor.

PS:
I wrote a program to calculate state-values for episodic and continuing setups respectively, code being pasted at the bottom of this post. The calculation result is as follows:

In the episodic setup, still focusing on the bottom-left corner, the agent no longer chooses to turn right, since:

q(s1, ↑) =  0+0.9*0.25 = 0.225
q(s1, →) = -1+0.9*0.9 = -0.19
q(s1, ↓) = -1+0.9*0.22 = -0.8
q(s1, ←) = -1+0.9*0.22 = -0.8
q(s1, o) = 0+0.9*0.22 = 0.198

code

import numpy as np

# Constants
gamma = 0.9  # Discount factor
reward_target = 1  # Reward for reaching the target
reward_boundary = -1  # Reward for hitting the boundary
reward_forbidden = -1  # Reward for forbidden states
n_rows = 5  # Number of rows in the grid
n_cols = 5  # Number of columns in the grid

actions = ['^', '>', 'v', '<', 'o']  # List of possible actions

#Grid world
grid_world = np.array([
	['S', 'S', 'S', 'S', 'S'],
	['S', 'F', 'F', 'S', 'S'],
    ['S', 'S', 'F', 'S', 'S'],
	['S', 'F', 'T', 'F', 'S'],
	['S', 'F', 'S', 'S', 'S'],
])

policy = np.array([
	['>', '>', '>', 'v', 'v'],
	['^', '^', '>', 'v', 'v'],
	['^', '<', 'v', '>', 'v'],
	['^', '>', 'o', '<', 'v'],
	['^', '>', '^', '<', '<'],
])

#return next_state, reward, done
def P(state, action, is_episodic=False):
    row, col = state
    target_row, target_col = row, col
    if action == '^':
        if row == 0:
            return (row, col), reward_boundary, False
        else:
            target_row = row - 1
    if action == '>':
        if col == n_cols - 1:
            return (row, col), reward_boundary, False
        else:
            target_col = col + 1
    if action == 'v':
        if row == n_rows - 1:
            return (row, col), reward_boundary, False
        else:
            target_row = row + 1
    if action == '<':
        if col == 0:
            return (row, col), reward_boundary, False
        else:
            target_col = col - 1

    if grid_world[target_row, target_col] == 'F':
        return (target_row, target_col), reward_forbidden, False
    if grid_world[target_row, target_col] == 'T':
        if is_episodic:
            return (target_row, target_col), reward_target, True
        else:
            return (target_row, target_col), reward_target, False
    if grid_world[target_row, target_col] == 'S':
        return (target_row, target_col), 0, False

# Calculate the state value of a policy
def calc_state_value_of_policy(policy, is_episodic=False):
    state_value = np.zeros((n_rows, n_cols))
    while True:
        new_state_value = np.zeros((n_rows, n_cols))
        for row in range(n_rows):
            for col in range(n_cols):
                action = policy[row, col]
                (next_row, next_col), reward, done = P((row, col), action, is_episodic)
                new_state_value[row, col] = reward + gamma * state_value[next_row, next_col]*(1-done)
        if np.sum(np.abs(new_state_value - state_value)) < 1e-4:
            break
        state_value = new_state_value
    return state_value

if __name__ == '__main__':
    print("State Value of continuing setup:")
    state_value = calc_state_value_of_policy(policy, is_episodic=False)
    print(state_value)
    print("State Value of episodic setup:")
    state_value = calc_state_value_of_policy(policy, is_episodic=True)
    print(state_value)

[MathFoundationRL]

Hi there, very detailed analysis.

But the policy here is given to demonstrate how to calculate its state value. It is intuitively "good" but may not be optimal. At this point, the concept of "optimal policies" have not been introduced.

You can double check if the state value is correct here using your algo. Please let me know if you see any errors.

[unlikezy]

Thanks for replying, Prof. Zhao!
Yes, I checked the state value calculation in the slide and it is correct, and the result is as follows:

[MathFoundationRL]

Great!

[fgc346]

Hi，Prof Zhao.
I read page 241，in the proof process， should the symbol a be β？

[MathFoundationRL]

Wow. It is a typo. You read very carefully. Thanks.