Which Parameter is used for Stage 3 ZeRO? #3605

Unanswered

fahadh4ilyas asked this question in Q&A

fahadh4ilyas
May 24, 2023

I've been reading DeepSpeed Integration from Transformers and there is something that interesting for me. Quoted from here

If you’re migrating from ZeRO-2 configuration note that allgather_partitions, allgather_bucket_size and reduce_scatter configuration parameters are not used in ZeRO-3. If you keep these in the config file they will just be ignored.

But, I never found this fact in here. Is this really true? If it's true, which parameters are used for stage 3 ZeRO?

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment