V0.1.0 Released Today!

[FrankLeeeee]

Overview

We are happy to release the version v0.1.0 today. Compared to the previous version, we have a brand new zero module and updated many aspects of our system for better perforamance and usability. The latest version can be installed by pip install colossalai now. We will update our examples and documentation in the next few days accordingly.

Highlights:

Note:
a. Only the major base commits are chosen to display. Successive commits which enhance/update the base commit are not shown.
b. Some commits do not have associated pull request ID for some unknown reasons.
c. The list is ordered by time.

Features

• add moe context, moe utilities and refactor gradient handler (add moe context, moe utilities and refactor gradient handler #455 )By @1SAA
• [zero] Update initialize for ZeRO ([zero] Update initialize for ZeRO #458 ) By @ver217
• [zero] hybrid cpu adam ([zero] hybrid cpu adam #445 ) By @feifeibear
• added Multiply Jitter and capacity factor eval for MOE (added Multiply Jitter and capacity factor eval for MOE #434 ) By @1SAA
• [fp16] refactored fp16 optimizer (refactored fp16 optimizer #392 ) By @FrankLeeeee
• [zero] memtracer to record cuda memory usage of model data and overall system ([zero] memtracer to record cuda memory usage of model data and overall system #395 ) By @feifeibear
• Added tensor detector (Added tensor detector #393 ) By @Gy-Lu
• Added activation offload (Added activation offload #331 ) By @Gy-Lu
• [zero] zero init context collect numel of model ([zero] zero init context collect numel of model #375 ) By @feifeibear
• Added PCIE profiler to dectect data transmission (Added PCIE profiler to dectect data transmission #373 ) By @1SAA
• Added Profiler Context to manage all profilers (Added Profiler Context to manage all profilers #340 ) By @1SAA
• set criterion as optional in colossalai initialize ([fix] set criterion as optional in colossalai initialize #336 ) By @FrankLeeeee
• [zero] Update sharded model v2 using sharded param v2 ([zero] Update sharded model v2 using sharded param v2 #323 ) By @ver217
• [zero] zero init context ([zero] zero init context #321 ) By @feifeibear
• Added profiler communication operations By @1SAA
• added buffer sync to naive amp model wrapper (added buffer sync to naive amp model wrapper #291 ) By @FrankLeeeee
• [zero] cpu adam kernel ([zero] cpu adam kernel #288 ) By @Gy-Lu
• Feature/zero (Feature/zero #279 ) By @feifeibear @FrankLeeeee @ver217
• impl shard optim v2 and add unit test By @ver217
• [profiler] primary memory tracer By @raejaf
• add sharded adam By @ver217

Unit Testing

• [test] fixed amp convergence comparison test ([test] fixed amp convergence comparison test #454 ) By @FrankLeeeee
• [test] optimized zero data parallel test ([test] optimized zero data parallel test #452 ) By @FrankLeeeee
• [test] make zero engine test really work ([test] make zero engine test really work #447 ) By @feifeibear
• optimized context test time consumption ([unit test] optimized context test time consumption #446 ) By @FrankLeeeee
• [unitest] polish zero config in unittest ([unitest] polish zero config in unittest #438 ) By @feifeibear
• added testing module ([feature] added testing module #435 ) By @FrankLeeeee
• [zero] polish ShardedOptimV2 unittest ([zero] polish ShardedOptimV2 unittest #385 ) By @feifeibear
• [unit test] Refactored test cases with component func ([unit test] Refactored test cases with component func #339 ) By @FrankLeeeee

Documentation

• [doc] Update docstring for ZeRO ([doc] Update docstring for ZeRO #459 ) By @ver217
• update README and images path (update README and images path #384 ) By @binmakeswell
• add badge and contributor list By @FrankLeeeee
• add community group and update issue template (add community group #271 ) By @binmakeswell
• update experimental visualization (Feature/docs/zh hans #253 ) By @Sze-qq
• add Chinese README By @binmakeswell

CI/CD

• update github CI with the current workflow ([devops] update github CI with the current git workflow #441 ) By @FrankLeeeee
• update unit testing CI rules By @FrankLeeeee
• added compatibility CI and options for release ci By @FrankLeeeee
• added pypi publication CI and remove formatting CI By @FrankLeeeee

Bug Fix

• fix gpt attention mask ([model_zoo] fix gpt attention mask #461 ) By @ver217
• [bug] Fixed device placement bug in memory monitor thread ([bug] Fixed device placement bug in memory monitor thread #433 ) By @FrankLeeeee
• fixed fp16 optimizer none grad bug (fixed fp16 optimizer none grad bug #432 ) By @FrankLeeeee
• fixed gpt attention mask in pipeline (fixed gpt attention mask in pipeline #430 ) By @FrankLeeeee
• [hotfix] fixed bugs in ShardStrategy and PcieProfiler (fixed bugs in ShardStrategy and PcieProfiler #394 ) By @1SAA
• fixed bug in activation checkpointing test (fixed bug in activation checkpointing test #387 ) By @FrankLeeeee
• [profiler] Fixed bugs in CommProfiler and PcieProfiler (Fixed bugs in CommProfiler and PcieProfiler #377 ) By @1SAA
• fixed CI dataset directory; fixed import error of 2.5d accuracy (Hotfix/minor bug fixes #255 ) By @kurisusnowdeng
• fixed padding index issue for vocab parallel embedding layers; updated 3D linear to be compatible with examples in the tutorial By @kurisusnowdeng

Miscellaneous

• [log] better logging display with rich ([log] better logging display with rich #426 ) By @feifeibear

---

This discussion was created from the release V0.1.0 Released Today!.

[xxw11]

Congratulations~🎉