fix(pu): fix noise layer's usage based on the original paper #866
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PaParaZz1
requested changes
Apr 18, 2025
ding/policy/dqn.py
Outdated
| @@ -248,6 +248,8 @@ def _forward_learn(self, data: List[Dict[str, Any]]) -> Dict[str, Any]: | |||
| .. note:: | |||
| For more detailed examples, please refer to our unittest for DQNPolicy: ``ding.policy.tests.test_dqn``. | |||
| """ | |||
| set_noise_mode(self._learn_model, True) | |||
There was a problem hiding this comment.
use noisy_net to control this line
Another question: how to deal with target_model in noisy net
puyuan1996
changed the title
Open
PaParaZz1
approved these changes
Jun 3, 2025
| @@ -201,6 +202,11 @@ def _forward_learn(self, data: dict) -> Dict[str, Any]: | |||
| # ==================== | |||
| self._learn_model.train() | |||
| self._target_model.train() | |||
|
|
|||
| # Set noise mode for NoisyNet for exploration in learning if enabled in config | |||
| set_noise_mode(self._learn_model, True) | |||
There was a problem hiding this comment.
why not use self._cfg.noisy_net to control this logic
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This pull request fixes the usage of Noisy Net in accordance with the original Noisy Net paper.
The key modifications are as follows:
Add
set_noise_modeFunction:A new helper function,
set_noise_mode, is introduced to control whether the noise is enabled (enable_noise). This function is used to update noise settings in the network.Add
_reset_noiseMethod in DQN:A new
_reset_noisemethod has been added to the DQN implementation. During each training step, the noise is reset and the corresponding noise is applied.Model Weight and Noise Settings:
Experimental Result:
After fixing the Noisy Net implementation to be consistent with the paper's description, experimental results indicate that there is no significant performance difference whether Noisy Net is used or not.
Related Issue
Check List