Reward of CryptoTradingEnv is not single step

[QuantHao]

Hi, @praveen-palanisamy

In CryptoTradingEnv, reward is computed based on ALL action steps, instead of current (single) action

Tensorflow-2-Reinforcement-Learning-Cookbook/Chapter04/crypto_trading_env.py

Line 92 in 31f8376

reward = self.account_value - self.opening_account_balance # Profit (loss)

Do you think compute reward by self.account_value - previous_account_value might be more appropriate?

[QuantHao]

Same issue applies to Chapter 5.

[praveen-palanisamy]

The reward is intentionally set to be the profit/loss (current_balance - opening_balance) on the account controlled by the Agent so that the Agent learns a policy that maximizes the total return (profit).
The Agent is expected to learn the relationship between the series of actions (trades) it executes in an episode and the resulting reward (profit/loss).

Why do you think this is an issue (other than not being strictly Markovian)?
(Converting this to a discussion)

[QuantHao]

Hi, @praveen-palanisamy

I think there are two options to design rewards:

1. assign a reward to EACH action per time step, similar to CartPool, thus this reward (eg. self.account_value - previous_account_value) corresponds to action taken at time t ONLY.

2. assign a reward to ALL actions token along the episode at the end of episode time T, similar to AlphaGO, while assign ZERO reward to intermediate time steps.

Your current setting current_balance - opening_balance seems belongs to the second classification. If yes, NON-ZERO reward should be assigned only when done is True. Am I correct?

[praveen-palanisamy]

The current reward function used in the recipe rewards the agent at every time step based on the current state (profit/loss) of the account controlled by the agent.

There are many ways to design the reward function and you have listed two. My comments w.r.t each of them are below:

1. This is the closest to the existing reward function in the recipe with the difference that you pointed out in your first post (reward is not just based on the trade executed at the current time step). This is a good option to try and compare with the existing reward function and it will be interesting to see if one helps the agent learn faster/better policy than the other.

2. You are correct. If we wanted to reward the agent only when the episode ends, we would calculate and return a non-zero reward only when done is True. But, will make the reward signal very sparse and therefore the agent will find it hard to learn a good policy and will likely require 10s to 100s of millions if not billions of interactions with the environment.

[QuantHao]

I see. Appreciate your comments.