How to specify the process groups in 3D parallelism? #4063
Unanswered
Muggle666
asked this question in
Community | Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I am using 3D training parallelism, but how do I specify the process groups in different parallelism? Like I want to use tensor parallelism only within one node but use other parallelisms across nodes. Does colossal ai support this feature and how can I implement this? By the way, how does the colossal ai handle this by default?
Beta Was this translation helpful? Give feedback.
All reactions