请问ColossalAI支持offload的操作吗 #207
-
|
请问显存不足的情况可以怎么处理？ColossalAI支持如 cpu offload的之类的操作吗？ |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 9 replies
-
|
Hi，we support offload. You can read the documentation at https://colossalai.org/tutorials/features/zero_redundancy_and_zero_offload |
Beta Was this translation helpful? Give feedback.
-
|
你好，我们目前在写tensor parallel部分的tutorial，但是由于并行种类较多，所以仍在进行中。我们写完之后会更新到tutorial文档的features目录下。我们的例子库里提供了GPT的代码，你可以试一下用1D/2D模式跑一下看看。paper里的应该是基于ViT的结果，所以会有出入。 |
Beta Was this translation helpful? Give feedback.
-
|
请问ColossalAI有提供像megatron对使用xxx.idx xxx.bin这种已处理后的数据集的方法吗？查看tutorials和docs后，似乎没有找到 |
Beta Was this translation helpful? Give feedback.
-
|
Hi，抱歉回复晚了，目前并没有提供这样的功能，不过应该替换一下dataset类就可以？ |
Beta Was this translation helpful? Give feedback.
-
|
https://github.com/hpcaitech/ColossalAI-Examples/tree/main/language/gpt |
Beta Was this translation helpful? Give feedback.
-
|
十分感谢 ，在源代码基础上修改了一些，已经解决了，十分感谢 |
Beta Was this translation helpful? Give feedback.
-
|
您好，上面得页面找不到了，请问现在哪个优化器支持offloading呢 |
Beta Was this translation helpful? Give feedback.
Hi，we support offload. You can read the documentation at https://colossalai.org/tutorials/features/zero_redundancy_and_zero_offload