Welcome to discuss about this book here!

[MathFoundationRL]

Hi there,

If you have any feedback about the book, you can leave a comment here. Thanks!

[etbox]

There excesses a right parentheses in the equation.

[MathFoundationRL]

Noted and thanks a lot!

[eccstartup]

In slides 9, page 22. I guess this is `exercise`.

[MathFoundationRL]

Yes you are right. Thanks for letting me know.

Shiyu

[L3Y1Q2]

Great book and course! Helped me reorganize some of the points of RL.

In the process of reading this book (ver 2022.8), I met some little confusion, probably clerical errors.

Thanks!

[MathFoundationRL]

Thanks for your detailed feedback.

• page 29: you are right.
• page 64: you are right
• page 91: you are right
• page 110 and more: could you specify more specifically?
• page111: you are right
• page128: you are right
• page133 and more: you are right

Overall, it's my pleasure that you read very carefully. Thanks for your feedback.

Shiyu

[L3Y1Q2]

Page 110:

Page 112:

[MathFoundationRL]

noted and thanks!

[businiaon]

if there should be a1, while a2 represents rightwards and a1 represents upwards.

[MathFoundationRL]

You are right. They should be a1. Thanks for the feedback!

[businiaon]

page 27, from Figure 2.3 if there should be p(r = 0|s1, a3) = 1 and p(r<> 0|s1, a3) = 0, not p(r = −1|s1, a3) = 1 and p(r <> −1|s1, a3) = 0 ?

[MathFoundationRL]

Yes, here r should be or not be equal to 0 (not -1)

[eccstartup]

第6章，page 20，这个好像不收敛的。

w = 0
g = lambda w: w**3 - 5

import random
for i in range(100):
    print(i, w)
    w = w - 1/(i+10) * (g(w) + random.gauss(0, 1))

[eccstartup]

a=1/k总是溢出

[ZiqianXie]

g的梯度是无界的, 收敛性用RM定理没法保证

[MathFoundationRL]

如果各位英文写作没问题的话，还请用英文，这样可以维护一个大家都能看的讨论环境，本书也有很多国外的读者，谢谢。

[MathFoundationRL]

要想在全局收敛，需要g的梯度有上界。

[ZiqianXie]

Dvoretzky 定理的证明只包括了 $\alpha_k, \beta_k$ 都由 $\mathcal{H}_k$ 决定的情况, 最好额外说明一下.

[MathFoundationRL]

谢谢，我会做相应修改

[MathFoundationRL]

如果各位英文写作没问题的话，还请用英文，这样可以维护一个大家都能看的讨论环境，本书也有很多国外的读者，谢谢。

[fanshJ]

6.2 Page 107
式子左边的 w_∞ − w_1 是不是应该改为 w_1 - w_∞，或者右边加负号？

[MathFoundationRL]

是的，谢谢

[MathFoundationRL]

如果各位英文写作没问题的话，还请用英文，这样可以维护一个大家都能看的讨论环境，本书也有很多国外的读者，谢谢。

[ritsuki1227]

Thank you very much for writing such a clear and helpful book!
Could you explain why the inequality shown on the image (bottom of the page 48) implies the contraction property of the Bellman Optimality Equation?
Maybe I don't understand some properties of the sup-norm.

[MathFoundationRL]

This inequality is the same as the definition of a contraction mapping. It shows that the function f(v) is contractive, (not the Bellman optimality equation is contractive).

[ritsuki1227]

Thank you for your comment. My understanding is that the definition states that $f$ is contractive iff

$$||f(v_1) - f(v_2)||\leq \gamma ||v_1 - v_2|| \quad \exists \gamma \in (0, 1) \forall v_1, v_2 \in \mathbb{R}^d,$$

which is not obviously the same as the inequality in the image above for me ($||.||$ denotes vector norm whereas $||.||_\infty$ denotes supremum norm). I guess I'm missing some basic concepts of sup-norm, but is it true that the implication below holds?

$$||f(v_1) - f(v_2)||_\infty \leq \gamma ||v_1 - v_2||_\infty \implies ||f(v_1) - f(v_2)|| \leq \gamma ||v_1 - v_2|| .$$

Also, I've found a minor typo on the section 3.3.3 on page 44. "for any $x_1, x_2 \in \mathbb{R}$" should be "for any $x_1, x_2 \in \mathbb{R}^d$".

[MathFoundationRL]

Now I understand. First, in the definition of contraction mapping, $||\cdot||$ refers to a general vector norm. That is, if there exists any type of vector norm so that this inequality is valid, then the function is contractive. Second, in the proof it is a special $L_\infty$ vector norm (it is still a vector norm). Third, in Theorem 3.2, the norm should also be the $L_\infty$ norm (but I wrote it as a general norm). Hope this answers your question. I will also revise to make it clearer.

You are right about the typo. Thanks!

[ritsuki1227]

Thank you very much. It is clear to me now!

[ZiqianXie]

Some notation inconsistency:

Page 196, the matrix-vector form should have $\nabla_\theta v_\pi\in\mathbb{R}^{m|S|}$ instead of $n|S|$ since $m$ is the dimension of $\theta$.

And later in 9.12, $n$ is being used to denote $|S|$

[MathFoundationRL]

You are right. Thanks!

By the way, I didn't expect but am happy to see that some readers reads the proof details carefully.

[bayncat]

Hello. Is there a missing "αk" before the summation symbol?

[MathFoundationRL]

Yes, you are right. Thanks!

[SakuraToErii]

I think this "not" may need to be deleted.

[SakuraToErii]

Sorry, I forgot to say it is on page 88.

[MathFoundationRL]

Yes, you are right. It will be corrected in the next version. Thanks

[Rino-Li]

In page. 85, the second paragraph of the subsection A comprehensive example: Episode length and sparse reward, "See, for example, Figure 5.3(h)" should be "5.3(a)", because you mentioned that the episode length is 1.

[Rino-Li]

It is a perfect course to study, thanks to teacher Zhao.

[MathFoundationRL]

Thanks. What I meant is actually correct, but it may be confusing. I will revise it in the future.

[SakuraToErii]

On page 214, this "z" may need to be modified as "x".

[MathFoundationRL]

Thanks. This is a new typo that has not been noticed so far. I will correct it.

[SakuraToErii]

On page 157, I think the "column" may need to be modified as "row".

[MathFoundationRL]

Thanks for the feedback. But this statement is correct. The ith row is the probabilities of reaching state i when starting every state.

[SakuraToErii]

Thank you and I have another question: All of this means that [P_π]ij is from state i to state j, in other words, i is the row and j is the column.......

[MathFoundationRL]

Now I understand. It is correct to say "column", but the meaning of the entries in the ith column is the probabilities reaching state i from different starting states. Thanks for point out the problem! You read very carefully!

[SakuraToErii]

Thank you for your patience. I read your book carefully due to its high quality and clear context. I am very grateful that I have been learning with such a cool book in RL.

[MathFoundationRL]

Thank you! I am finalizing the manuscript of the book. The final version will be better than the first draft. I will it will be cooler:) Please stay tuned.

[renjieDLUT]

"the action value of s_t"我理解应该是state value吧

[MathFoundationRL]

Yes, it should be state value. Thanks.

[SakuraToErii]

Hello dear Professor Zhao, I think this "+" symbol may should be a "=", this equation has "tortured" me for one month, hahahaha.

[MathFoundationRL]

is incorrect, because s is not a function of theta. We should calculate the derivative of q with respect to theta

[MathFoundationRL]

For the equations I gave about, it is better to calculate \partial f(x,y)/\partial \theta. Then, it is clearer.

[Atsan1006]

[Atsan1006]

[MathFoundationRL]

I already mentioned: "For the equations I gave about, it is better to calculate \partial f(x,y)/\partial \theta. Then, it is clearer." I don't see any inaccuracy in my derivation. Sorry that I cannot reply more.

[AIexanderDicke]

Thanks for the nice write-up!

There is a small (but important) typo in Algorithm 4.1:

While $v_k$ has not converged in the sense that $\|v_k − v_{k-1}\|$ is greater than a predefined
small threshold, for the $k$th iteration, do

should read

While $v_k$ has not converged in the sense that $\|v_k − v_{k-1}\|$ is smaller than a predefined
small threshold, for the $k$th iteration, do

[MathFoundationRL]

Thanks for letting me know. What I am referring to is that "not converged" means "xx is greater than xxx". I understand your meaning. Let me think if there is a better way

[xuxiaolin0926]

163页 $s_{2}$ 与 $s_{3}$ 的typo。

感谢您的书。

[MathFoundationRL]

Yes, s2 should be s3. Thanks.

[Annihillusion]

Thank you so much for writing such a helpful book!
Could you further explain why this equation hold? It may not seen so trivial to me. Thank you in advance!

What's more, in Page 205, it might be a little mistake.

[MathFoundationRL]

Hi there, for the first underlined equation: d^TP=d^T, this is due the property of the stationary distribution d. You can check more about d in the book.

About the second circled equation: \sum d(s)=1. This is why it holds.

[Annihillusion]

Thank you so much for your prompt and patient reply! I understand these two equations now.

[yapeg]

Page 172. Chapter 8. Algorithm 8.1.
'A function $\hat{v}(s,w)$ that is $\cancel{a}$ differentiable in $w$.'

[MathFoundationRL]

Thanks, it is indeed a typo

[HaomingFu]

Page 53, Section 3.3.3, Box 3.1. I think there is a typo.
In the sentence "We prove that the consequence $\{x_k\}_{k=1}^\infty$ with $x_k = f(x_{k=1})$ is convergent", the "consequence" should be "sequence".

[MathFoundationRL]

Thank you. It is indeed a typo.

[wyanyue]

I think there's a symbol \gamma missing here. There's the same typo in slides.

[MathFoundationRL]

Thank you. It is indeed a typo!

[lhz99]

print error
on the right graph, I think it is a “SGD(m=1)" not “SG(m=1)"

[L-lorish]

it is actually Pr(s| s') because in following proof, the author has change s to s' and s' to s, equivalently.

[lhz99]

But why need to change here? I'am confused

[MathFoundationRL]

It is mainly for the notational convenience.

[lhz99]

row 2 and 3 in the derivation of tr[var(X)] is repeated

[MathFoundationRL]

Many thanks for the feedback!

[L-lorish]

I am confused of the result of derivative.if adding the explanation is much better!

[L-lorish]

The derivative formula may be derived or inspired by the result.lol

[MathFoundationRL]

(10.17) is a result of a standard derivative operation. If you have any specific question, you can ask.

[L-lorish]

I have delved further into the matter and discovered that this derivative is indeed correct. However, upon my first look, I found myself puzzled by the derivative of $q_{\mu}$. Later, I came to understand that this confusion arose because initially, $v_{\mu}(s)$ had not been expanded.

$q_{\mu}(s,\mu(s)) = \sum_{r}P(r|s,\mu(s))r + \gamma \sum_{s^{\prime}}P(s^{\prime}|s,\mu(s))v_{\mu}(s^{\prime})$

Combined with $v_{\mu}(s) = q_{\mu}(s,\mu(s))$, it clearly indicates that $q_{\mu}(s)$ is a function of both $q_{\mu}$ and $\mu(s)$

[lhz99]

There is total derivative, right?

[Abner-fu]

[MathFoundationRL]

Could you provide more derivation details so that I can help to check it quickly?

[Abner-fu]

Here are my derivation details. If I have any problems, please tell me. Thank you.

[MathFoundationRL]

I think you got the same result as mine, is it? Also, did you change your question? Now your answer in your question is the same as my result

[Abner-fu]

I figured it out. I didn't notice this formula before. $\sum_{a \in A} \sum_{r \in R} \pi(a \mid s) P(r \mid s, a) \beta=\beta $.
It's my problem. Thank you very much.

[Eternity-Wang]

Page 207. Chapter 9. Here I think the π(a|s,θ) should take values in the interval (0,1) due to the softmax function, not [0,1].

[MathFoundationRL]

It is indeed more accurate to use the open interval. Thanks.

[FBwenoll]

Prof. Zhao, when will this book be published?

[MathFoundationRL]

Thanks for asking. It has been sent to the printing factory. Hopefully it can be officially published in July 2024 by Tsinghua University Press (for mainland of China). The international version will be printed by Springer probably in the first half of 2025.

[HaomingFu]

In Section 3.6, Page 64, book ver. March 2024, it says:
"In particular, a policy is optimal if its state values are greater than or equal to those of any other policy."
Based on Theorem 3.4, can I make a stronger statement like:
"In particular, a policy is optimal if and only if its state values are greater than or equal to those of any other policy?"

[MathFoundationRL]

It is actually a definition instead of a conclusion or statement. It may be better to use "when" rather than "if", but it is a convention to use "if" in definitions as well.

[HaomingFu]

Let me explain my stance a bit more. My question may seem a bit trivial, but it is not uncommon that an optimal policy may not exist at all because it is possible that for any policy $\pi$ it achieves the best state values for some states, while there does not exist a policy that achieves the best for all states. After learning Theorem 3.3 and 3.4, I know this possibility can be ruled out and that an optimal policy indeed exists. I want to confirm that my understanding is correct. So that is why I am so obsessed with the problem that I asked about.

Anyways, thanks to you Prof. Zhao. You have already answered my question and dispelled my doubts.

[co-gy]

maybe it's $q_t(s_t, a_t)$ here? (the subscript t is missing)

[MathFoundationRL]

Thanks. There should be a subscript here.

[L-lorish]

Prof. Zhao, can u add more algorithm such as TRPO, PPO, SAC... into this book, though u have mentioned part of that basic mathematical knowledge in some chapters. VERY THANKS.

[MathFoundationRL]

Hi there,

Thanks. Quite a few readers sent me suggestions like that. At this moment, I do not have the plan to do that due to my very limited time and there are quite many materials online:)