V0.1.2 Released Today!

[ver217]

Overview

Here are the main improvements of this release:

1. MOE and BERT models can be trained with ZeRO.
2. Provide a uniform checkpoint for all kinds of parallelism.
3. Optimize ZeRO-offload, and improve model scaling.
4. Design a uniform model memory tracer.
5. Implement an efficient hybrid Adam (CPU and CUDA kernels).
6. Improve activation offloading.
7. Profiler TensorBoard plugin of Beta version.
8. Refactor pipeline module for closer integration with engine.
9. Chinese tutorials, WeChat and Slack user groups.

What's Changed

Features

• [zero] get memory usage for sharded param by @feifeibear in [zero] get memory usage for sharded param #536
• [zero] improve the accuracy of get_memory_usage of sharded param by @feifeibear in [zero] improve the accuracy of get_memory_usage of sharded param #538
• [zero] refactor model data tracing by @feifeibear in [zero] refactor model data tracing #537
• [zero] get memory usage of sharded optim v2. by @feifeibear in [zero] get memory usage of sharded optim v2. #542
• [zero] polish ZeroInitContext by @ver217 in [zero] polish ZeroInitContext #540
• [zero] optimize grad offload by @ver217 in [zero] optimize grad offload #539
• [zero] non model data tracing by @feifeibear in [zero] non model data tracing #545
• [zero] add zero config to neutralize zero context init by @1SAA in [zero] add zero config to neutralize zero context init #546
• [zero] dump memory stats for sharded model by @feifeibear in [zero] dump memory stats for sharded model #548
• [zero] add stateful tensor by @feifeibear in [zero] add stateful tensor #549
• [zero] label state for param fp16 and grad by @feifeibear in [zero] label state for param fp16 and grad #551
• [zero] hijack p.grad in sharded model by @ver217 in [zero] hijack p.grad in sharded model #554
• [utils] update colo tensor moving APIs by @feifeibear in [utils] update colo tensor moving APIs #553
• [polish] rename col_attr -> colo_attr by @feifeibear in [polish] rename col_attr -> colo_attr #558
• [zero] trace states of fp16/32 grad and fp32 param by @ver217 in [zero] trace states of fp16/32 grad and fp32 param #571
• [zero] adapt zero for unsharded parameters by @1SAA in [zero] adapt zero for unsharded parameters #561
• [refactor] memory utils by @feifeibear in [refactor] memory utils #577
• Feature/checkpoint gloo by @kurisusnowdeng in Feature/checkpoint gloo #589
• [zero] add sampling time for memstats collector by @Gy-Lu in [zero] add sampling time for memstats collector #610
• [model checkpoint] checkpoint utils by @kurisusnowdeng in [model checkpoint] checkpoint utils #592
• [model checkpoint][hotfix] unified layers for save&load by @kurisusnowdeng in [model checkpoint][hotfix] unified layers for save&load #593
• Feature/checkpoint 2D by @kurisusnowdeng in Feature/checkpoint 2D #595
• Feature/checkpoint 1D by @kurisusnowdeng in Feature/checkpoint 1D #594
• [model checkpoint] CPU communication ops by @kurisusnowdeng in [model checkpoint] CPU communication ops #590
• Feature/checkpoint 2.5D by @kurisusnowdeng in Feature/checkpoint 2.5D #596
• Feature/Checkpoint 3D by @kurisusnowdeng in Feature/Checkpoint 3D #597
• [model checkpoint] checkpoint hook by @kurisusnowdeng in [model checkpoint] checkpoint hook #598
• Feature/Checkpoint tests by @kurisusnowdeng in Feature/Checkpoint tests #599
• [zero] adapt zero for unsharded parameters (Optimizer part) by @1SAA in [zero] adapt zero for unsharded parameters (Optimizer part) #601
• [zero] polish init context by @feifeibear in [zero] polish init context #645
• refactor pipeline---put runtime schedule into engine. by @YuliangLiu0306 in refactor pipeline---put runtime schedule into engine. #627

Bug Fix

• [Zero] process no-leaf-module in Zero by @1SAA in [Zero] process no-leaf-module in Zero #535
• Add gather_out arg to Linear by @Wesley-Jzy in Add gather_out arg to Linear #541
• [hoxfix] fix parallel_input flag for Linear1D_Col gather_output by @Wesley-Jzy in [hoxfix] fix parallel_input flag for Linear1D_Col gather_output #579
• [hotfix] add hybrid adam to init by @ver217 in [hotfix] add hybrid adam to __init__ #584
• Hotfix/path check util by @kurisusnowdeng in Hotfix/path check util #591
• [hotfix] fix sharded optim zero grad by @ver217 in [hotfix] fix sharded optim zero grad #604
• Add tensor parallel input check by @Wesley-Jzy in Add tensor parallel input check #621
• [hotfix] Raise messages for indivisible batch sizes with tensor parallelism by @number1roy in [hotfix] Raise messages for indivisible batch sizes with tensor parallelism #622
• [zero] fixed the activation offload by @Gy-Lu in [zero] fixed the activation offload #647
• fixed bugs in CPU adam by @1SAA in fixed bugs in CPU adam #633
• Revert "[zero] polish init context" by @feifeibear in Revert "[zero] polish init context" #657
• [hotfix] fix a bug in model data stats tracing by @feifeibear in [hotfix] fix a bug in model data stats tracing #655
• fix bugs for unsharded parameters when restore data by @1SAA in fix bugs for unsharded parameters when restore data #664

Unit Testing

• [zero] test zero tensor utils by @FredHuang99 in [zero] test zero tensor utils #609
• remove hybrid adam in test_moe_zero_optim by @1SAA in remove hybrid adam in test_moe_zero_optim #659

Documentation

• Refactored docstring to google style by @number1roy in Refactored docstring to google style #532
• [docs] updatad docs of hybrid adam and cpu adam by @Gy-Lu in [docs] updatad docs of hybrid adam and cpu adam #552
• html refactor by @number1roy in html refactor #555
• [doc] polish docstring of zero by @ver217 in [doc] polish docstring of zero #612
• [doc] update rst by @ver217 in [doc] update rst #615
• [doc] polish amp docstring by @ver217 in [doc] polish amp docstring #616
• [doc] polish moe docsrting by @ver217 in [doc] polish moe docsrting #618
• [doc] polish optimizer docstring by @ver217 in [doc] polish optimizer docstring #619
• [doc] polish utils docstring by @ver217 in [doc] polish utils docstring #620
• [NFC] polish colossalai/kernel/cuda_native/csrc/kernels/cuda_util.cu … by @GaryGky in [NFC] polish colossalai/kernel/cuda_native/csrc/kernels/cuda_util.cu … #625
• [doc] polish checkpoint docstring by @ver217 in [doc] polish checkpoint docstring #637
• update GPT-2 experiment result by @Sze-qq in update GPT-2 experiment result  #666
• [NFC] polish code by @binmakeswell in [NFC] polish code #646

Model Zoo

• [model zoo] add activation offload for gpt model by @Gy-Lu in [model zoo] add activation offload for gpt model #582

Miscellaneous

• [logging] polish logger format by @feifeibear in [logging] polish logger format #543
• [profiler] add MemProfiler by @raejaf in [profiler] add MemProfiler #356
• [Bot] Synchronize Submodule References by @github-actions in [Bot] Synchronize Submodule References #501
• [tool] create .clang-format for pre-commit by @BoxiangW in [tool] create .clang-format for pre-commit #578
• [GitHub] Add prefix and label in issue template by @binmakeswell in [GitHub] Add prefix and label in issue template #652

Full Changelog: v0.1.1...v0.1.2

---

This discussion was created from the release V0.1.2 Released Today! .