Support for Swin + AlBert

[nostalgicimp]

Does Colossal already support or plan to support Swin + AlBert model? How much effort does it take to write an Swin, AlBert model in Colossal?
For various and endless new models, how does Colossal support that? e.g. tons of models from huggingface.

Thanks.

[binmakeswell]

Hi @nostalgicimp Thanks for your comments.
We currently do not directly provide Swin and AlBert examples, but we provide examples of MoE and BERT.
https://github.com/hpcaitech/ColossalAI-Examples/tree/main/image/moe
https://github.com/hpcaitech/ColossalAI-Examples/tree/main/language/bert

For various and endless new models, Colossal-AI seamlessly accelerates large models at with Hugging Face, you can refer to the example and blog of OPT.
https://github.com/hpcaitech/ColossalAI-Examples/tree/main/language/opt
https://medium.com/@yangyou_berkeley/colossal-ai-seamlessly-accelerates-large-models-at-low-costs-with-hugging-face-4d1a887e500d

We are also working to further improve automation, such as automatic parallelism, to reduce the workload of adapting models to more easily support new models that are emerging.

[kkkkkong]

你好，已收到您的邮件！