Adding support to PPO for continuous actions spaces

[kuds]

#77

Design

Created a new class called ContinuousProximalPolicyOptimization to handle continuous action spaces for PPO. Updated README with new tutorial, added new unit tests, and general comments clean-up

[rodrigodesalvobraz]

Thank you! I am currently reviewing the PR.

[rodrigodesalvobraz]

Thanks, I've reviewed it and have two requests:

• You duplicated quite a bit of code from ppo.py to continuous_ppo.py. It seems that almost all changes are in _actor_loss. Would you be able to define a base class BasePPO containing most of the common code now in PPO, and subclass it into two subclasses PPO and ContinuousPPO, each of them defining the appropriate _actor_loss?
• Can you replace tuple[Tensor, Normal] by Tuple[Tensor, Normal]? The former does not work with Python 3.8, which we still currently support in pyproject.toml.

[kuds]

Sure, it makes sense to consolidate the core PPO logic into a base class and have the discrete and continuous versions override where needed (e.g., the _actor_loss method). I will work on incorporating both requests into my pull request and should have those done sometime next week. Thanks for the feedback!

[kuds]

Sorry for the delay on this. I have the base class created for PPO with the discrete and continuous versions extending it. I will get my changes checked in by end of this week.

[rodrigodesalvobraz]

Sorry for the delay on this. I have the base class created for PPO with the discrete and continuous versions extending it. I will get my changes checked in by end of this week.

No problem, thanks for the update! Looking forward to it.

[kuds]

I appreciate your patience on this! I have finished creating the PPO base class with the discrete and continuous versions, inheriting from it and overriding it where needed. Let me know your thoughts and if I can help with any other issues or development priorities!

[rodrigodesalvobraz]

It's looking good! I was running the Lunar Lander tutorial (nice!) and encountered an error. I am curious if you are seeing the same thing by any chance? Here's the notebook with the error.

[kuds]

So the issue you are running into is known (Issue #1142) with Gymnasium 0.29.1 and Numpy Verison 2+. You have to downgrade Numpy to version 1+. There is a comment in the Lunar Lander tutorial about this to help during this transition.

The Farama Foundation just released version 1.0.0 of Gymnasium about a week ago, which should resolve this issue with Numpy 2+, but I have not tried it with Pearl. The library upgrade for Gymansium and Numpy should probably be its effort/issue/pull request if you are ok with that.

Let me know if you have any other issues or additional questions!

[kuds]

@rodrigodesalvobraz

It looks like the Pearl repo has undergone some significant changes in the last couple of days, like around the replay buffer. Would you like me to rework my pull request to handle these latest updates?

[rodrigodesalvobraz]

@kuds, that would be greatly appreciated, thank you!
Thank you also for diagnosing that last issue!

[kuds]

I have merged in the latest round of changes from the main pearl branch and assimilated the renamed replay buffers into the PPO code base. I am still running some tests, but I wanted to get this up for an initial review.

[rodrigodesalvobraz]

I have merged in the latest round of changes from the main pearl branch and assimilated the renamed replay buffers into the PPO code base. I am still running some tests, but I wanted to get this up for an initial review.

Sounds good, thanks. I am checking things around, but I see you are still making changes, so let me know when you're ready.

[rodrigodesalvobraz]

Hi @kuds ,
I checked out your PR today and a few unit tests failed:

Errors were:

pearl\policy_learners\sequential_decision_making\ppo.py", line 68, in __init__:
TypeError: ProximalPolicyOptimizationBase.__init__() got an unexpected keyword argument 'history_summarization_learning_rate'

pearl\policy_learners\sequential_decision_making\ppo_continuous.py", line 101, in __init__:
AttributeError: 'ContinuousProximalPolicyOptimization' object has no attribute 'is_action_continuous'

[kuds]

@rodrigodesalvobraz

Sorry for the delay in responding. I have time this week and next to take a look this.

Thank you for your patience!

[kuds]

@rodrigodesalvobraz

Quick update: I am working on migrating my code base to the latest version of Pearl. I should have this pull request re-opened by the weekend.

[rodrigodesalvobraz]

@rodrigodesalvobraz

Quick update: I am working on migrating my code base to the latest version of Pearl. I should have this pull request re-opened by the weekend.

Thanks. I really appreciate you updating so many times. Next time you submit I will make a point of integrating it right away so we don't have this problem anymore!