Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BREAKING][refact]: move actor/critic/hybrid_engine/reward_model/rollout/workers out of ppo directory for reuse #52

Merged
merged 6 commits into from
Dec 18, 2024

Conversation

PeterSH6
Copy link
Collaborator

@PeterSH6 PeterSH6 commented Dec 16, 2024

  • A BREAKING PR, need to fix the import
  • In current architecture:
    • verl.workers: store all the files related to SPMD distributed computation
    • verl.trainer: store different RL algorithms, SFT, and generation scripts

@PeterSH6 PeterSH6 changed the title ['[refact] [BREAKING][refact]: move actor/critic/hybrid_engine/reward_model/rollout/workers out of ppo directory for reuse Dec 16, 2024
@PeterSH6 PeterSH6 changed the title [BREAKING][refact]: move actor/critic/hybrid_engine/reward_model/rollout/workers out of ppo directory for reuse [WIP][BREAKING][refact]: move actor/critic/hybrid_engine/reward_model/rollout/workers out of ppo directory for reuse Dec 16, 2024
@PeterSH6 PeterSH6 changed the title [WIP][BREAKING][refact]: move actor/critic/hybrid_engine/reward_model/rollout/workers out of ppo directory for reuse [BREAKING][refact]: move actor/critic/hybrid_engine/reward_model/rollout/workers out of ppo directory for reuse Dec 16, 2024
@@ -38,7 +38,7 @@
from verl.utils.megatron.pipeline_parallel import (compute_transformers_input_shapes, make_batch_generator)
from verl import DataProto
from verl.trainer.ppo import core_algos
from verl.trainer.ppo.actor import BasePPOActor
from verl.trainer.workers.actor import BasePPOActor
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may also need to rename

  • BasePPOActor -> BaseActor
  • DataParallelPPOActor -> DataParallelActor

Other classes are similar

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can simply delete base class

Copy link
Collaborator Author

@PeterSH6 PeterSH6 Dec 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from verl.trainer.workers.critic import DataParallelPPOCritic
from verl.trainer.workers.actor.megatron_actor import MegatronPPOActor

DDP & Megatron class have the same issue, I think we still need to delete PPO. Shall we just make the PPO implementation a base class? So that other algorithms can simply override the functions.

@PeterSH6 PeterSH6 merged commit f535977 into main Dec 18, 2024
2 checks passed
@PeterSH6 PeterSH6 deleted the gm/refact branch December 18, 2024 06:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants