Code

[zero bubble]support zbv all#6081

Code

Open

flybird11111 wants to merge 167 commits intohpcaitech:feature/zerobubblehpcaitech/ColossalAI:feature/zerobubblefrom flybird11111:support-zbv-allflybird11111/ColossalAI:support-zbv-allCopy head branch name to clipboard

+298-54

Commits

Commits on Jul 1, 2024

• fp8 operators for compressed communication

  Show description for f5a52e1
  BurkeHulk

  authored
  f5a52e1

Commits on Jul 4, 2024

• Merge branch 'hpcaitech:main' into feature/fp8_comm

  BurkeHulk

  authored
  6991819

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for e17f835
  pre-commit-ci[bot]

  committed
  e17f835

Commits on Jul 10, 2024

• fix typo

  GuangyaoZhang

  committed
  dbfa7d3

Commits on Jul 12, 2024

• fix scaling algorithm in FP8 casting

  BurkeHulk

  committed
  1e19594

• support fp8 communication in pipeline parallelism

  BurkeHulk

  committed
  e881901

• add fp8_communication flag in the script

  BurkeHulk

  committed
  6601874

• Merge remote-tracking branch 'origin/feature/fp8_comm' into feature/fp8_comm

  Show description for 1f1b856
  BurkeHulk

  committed
  1f1b856

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 51f916b
  pre-commit-ci[bot]

  committed
  51f916b

Commits on Jul 16, 2024

• Merge pull request hpcaitech#5885 from BurkeHulk/feature/fp8_comm

  Show description for 9470701
  BurkeHulk

  authored
  9470701

• shardformer fp8

  GuangyaoZhang

  committed
  457a0de

Commits on Jul 17, 2024

• fix rebase

  GuangyaoZhang

  committed
  5a310b9

• remove all to all

  GuangyaoZhang

  committed
  6a20f07

Commits on Jul 18, 2024

• Merge pull request hpcaitech#5899 from BurkeHulk/SP_fp8

  Show description for d0bdb51
  GuangyaoZhang

  authored
  d0bdb51

• fix shardformer fp8 communication training degradation

  GuangyaoZhang

  committed
  5b969fd

• Merge pull request hpcaitech#5921 from BurkeHulk/fp8_fix

  Show description for 62661cd
  GuangyaoZhang

  authored
  62661cd

Commits on Jul 24, 2024

• [fp8] support all-gather flat tensor (hpcaitech#5932)

  ver217

  authored
  5fd0592

Commits on Aug 2, 2024

• [fp8] add fp8 comm for low level zero

  ver217

  committed
  ae486ce

• [test] add zero fp8 test case

  ver217

  committed
  91e596d

• Merge pull request hpcaitech#5961 from ver217/feature/zeor-fp8

  Show description for c297e21
  BurkeHulk

  authored
  c297e21

Commits on Aug 5, 2024

• [Feature] llama shardformer fp8 support (hpcaitech#5938)

  Show description for 53cb960
  GuangyaoZhang

  authored
  53cb960

Commits on Aug 6, 2024

• [FP8] rebase main (hpcaitech#5963)

  Show description for 0c10afd
  

  18 peopleauthored
  0c10afd

• [fp8]support all2all fp8 (hpcaitech#5953)

  Show description for afb26de
  

  flybird11111

  and

  pre-commit-ci[bot]

  authored
  afb26de

Commits on Aug 7, 2024

• [fp8] add fp8 linear (hpcaitech#5967)

  Show description for 76ea164
  ver217

  authored
  76ea164

• [fp8] support fp8 amp for hybrid parallel plugin (hpcaitech#5975)

  Show description for ccabcf6
  ver217

  authored
  ccabcf6

• fix (hpcaitech#5976)

  flybird11111

  authored
  7739629

Commits on Aug 8, 2024

• [Feature]: support FP8 communication in DDP, FSDP, Gemini (hpcaitech#5928)

  Show description for b480eec
  

  BurkeHulk

  and

  pre-commit-ci[bot]

  authored
  b480eec

• [test ci]Feature/fp8 comm (hpcaitech#5981)

  Show description for 4b9bec8
  flybird11111

  authored
  4b9bec8

Commits on Aug 9, 2024

• [fp8] support gemini plugin (hpcaitech#5978)

  Show description for 8241c0c
  ver217

  authored
  8241c0c

• [fp8] use torch compile (torch >= 2.3.0) (hpcaitech#5979)

  Show description for e4aadee
  botbw

  authored
  e4aadee

• [fp8]Moe support fp8 communication (hpcaitech#5977)

  Show description for f1a3a32
  

  flybird11111

  and

  pre-commit-ci[bot]

  authored
  f1a3a32

Commits on Aug 12, 2024

• [fp8] support hybrid parallel plugin (hpcaitech#5982)

  Show description for b2483c8
  wangbluo

  authored
  b2483c8

Commits on Aug 13, 2024

• [fp8] refactor fp8 linear with compile (hpcaitech#5993)

  Show description for 0978080
  ver217

  authored
  0978080

Commits on Aug 14, 2024

• [fp8] support asynchronous FP8 communication (hpcaitech#5997)

  Show description for 597b206
  

  flybird11111

  and

  pre-commit-ci[bot]

  authored
  597b206

Commits on Aug 15, 2024

• [fp8] update torch.compile for linear_fp8 to >= 2.4.0 (hpcaitech#6004)

  botbw

  authored
  88fa096

• [fp8] linear perf enhancement

  botbw

  committed
  1a2e90d

• [fp8]update reduce-scatter test (hpcaitech#6002)

  Show description for 20722a8
  flybird11111

  authored
  20722a8

Commits on Aug 16, 2024

• [fp8] add use_fp8 option for MoeHybridParallelPlugin (hpcaitech#6009)

  wangbluo

  authored
  3f09a61

• [fp8] zero support fp8 linear. (hpcaitech#6006)

  Show description for 0a51319
  flybird11111

  authored
  0a51319

Commits on Aug 17, 2024

• merge

  wangbluo

  committed
  4cf79fa

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 81272e9
  pre-commit-ci[bot]

  committed
  81272e9

Commits on Aug 19, 2024

• fix the merge

  wangbluo

  committed
  02636c5

• Merge branch 'fp8_merge' of https://github.com/wangbluo/ColossalAI into fp8_merge

  wangbluo

  committed
  52289e4

• fix the merge

  wangbluo

  committed
  1a5847e

• fix the merge

  wangbluo

  committed
  3353042

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 64aad96
  pre-commit-ci[bot]

  committed
  64aad96

• fix the merge

  wangbluo

  committed
  4c82bfc

• Merge branch 'fp8_merge' of https://github.com/wangbluo/ColossalAI into fp8_merge

  wangbluo

  committed
  0d8e82a

• fix

  wangbluo

  committed
  12b4401

• fix

  wangbluo

  committed
  2eb3683

• fix the merge

  wangbluo

  committed
  88b3f06

• fix

  wangbluo

  committed
  1f703e0

Commits on Aug 20, 2024

• fix

  wangbluo

  committed
  5382311

• fix

  wangbluo

  committed
  f7acfa1

• fix

  wangbluo

  committed
  2ee6235

• fix

  wangbluo

  committed
  2e4cbe3

• fix merge

  wangbluo

  committed
  2d362ac

Commits on Aug 21, 2024

• fix the merge

  wangbluo

  committed
  eb5ba40

• fix

  wangbluo

  committed
  193030f

• fix

  wangbluo

  committed
  6aface9

• fix

  wangbluo

  committed
  698c8b9

• fix

  wangbluo

  committed
  8b8e282

Commits on Aug 22, 2024

• [fp8] Merge feature/fp8_comm to main branch of Colossalai (hpcaitech#6016)

  Show description for eea37da
  

  17 peopleauthored
  eea37da

• Merge pull request hpcaitech#6023 from wangbluo/fp8_merge

  Show description for d77e66a
  wangbluo

  authored
  d77e66a

• fix

  wangbluo

  committed
  971b16a

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for a292554
  pre-commit-ci[bot]

  committed
  a292554

• Merge pull request hpcaitech#6024 from wangbluo/fix_merge

  Show description for afe845f
  wangbluo

  authored
  afe845f

• Merge branch 'main' into feature/fp8_comm

  ver217

  authored
  caab4a3

Commits on Aug 23, 2024

• Update train_dpo.py

  flybird11111

  authored
  0bc9a87

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 3b0df30
  pre-commit-ci[bot]

  committed
  3b0df30

• Update low_level_zero_plugin.py

  flybird11111

  authored
  9e76764

• Merge pull request hpcaitech#6029 from hpcaitech/flybird11111-patch-1

  Show description for 0bf46c5
  wangbluo

  authored
  0bf46c5

Commits on Aug 26, 2024

• fix

  wangbluo

  committed
  dae3999

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 80d24ae
  pre-commit-ci[bot]

  committed
  80d24ae

• Merge pull request hpcaitech#6033 from wangbluo/fix

  Show description for 4a6f31e
  wangbluo

  authored
  4a6f31e

Commits on Aug 27, 2024

• Merge pull request hpcaitech#6012 from hpcaitech/feature/fp8_comm

  Show description for 17904cb
  ver217

  authored
  17904cb

• [CI] Remove triton version for compatibility bug; update req torch >=2.2 (hpcaitech#6018)

  Show description for d383449
  Edenzzzz

  authored
  d383449

Commits on Aug 28, 2024

• [plugin] hotfix zero plugin (hpcaitech#6036)

  Show description for cc1b0ef
  ver217

  authored
  cc1b0ef

• [Colossal-LLaMA] Refactor latest APIs (hpcaitech#6030)

  Show description for 4a68efb
  

  TongLi3701

  and

  pre-commit-ci[bot]

  authored
  4a68efb

• add fused norm (hpcaitech#6038)

  TongLi3701

  authored
  0d3a85d

Commits on Aug 29, 2024

• [FP8] unsqueeze scale to make it compatible with torch.compile (hpcaitech#6040)

  GuangyaoZhang

  authored
  e96a076

Commits on Sep 2, 2024

• [colossalai/checkpoint_io/...] fix bug in load_state_dict_into_model; format error msg (hpcaitech#6020)

  Show description for e9032fb
  

  flymin

  and

  pre-commit-ci[bot]

  authored
  e9032fb

Commits on Sep 3, 2024

• [Hotfix] Remove deprecated install (hpcaitech#6042)

  Show description for c650a90
  TongLi3701

  authored
  c650a90

• [fp8] optimize all-gather (hpcaitech#6043)

  Show description for c3b5caf
  ver217

  authored
  c3b5caf

• [fp8] fix linear hook (hpcaitech#6046)

  ver217

  authored
  26e5539

Commits on Sep 9, 2024

• [fp8] disable all_to_all_fp8 in intranode (hpcaitech#6045)

  Show description for 5ce6dd7
  

  BurkeHulk

  and

  pre-commit-ci[bot]

  authored
  5ce6dd7

Commits on Sep 10, 2024

• [release] update version (hpcaitech#6041)

  Show description for b3db105
  ver217

  authored
  b3db105

• [Feature] Split cross-entropy computation in SP (hpcaitech#5959)

  Show description for 8fd25d6
  

  Edenzzzz

  and

  pre-commit-ci[bot]

  authored
  8fd25d6

• [hotfix] moe hybrid parallelism benchmark & follow-up fix (hpcaitech#6048)

  Show description for c54c4fc
  botbw

  authored
  c54c4fc

Commits on Sep 11, 2024

• [fp8] hotfix backward hook (hpcaitech#6053)

  Show description for 13946c4
  ver217

  authored
  13946c4

• [doc] update sp doc (hpcaitech#6055)

  Show description for a35a078
  

  flybird11111

  and

  pre-commit-ci[bot]

  authored
  a35a078

Commits on Sep 13, 2024

• fix the sp

  wangbluo

  committed
  fdd84b9

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 216d54e
  pre-commit-ci[bot]

  committed
  216d54e

• fix the attn

  wangbluo

  committed
  0a01e2a

• fix

  wangbluo

  committed
  683179c

• fix

  wangbluo

  committed
  6eb8832

• fix

  wangbluo

  committed
  f393867

• fix

  wangbluo

  committed
  dc03217

• [zerobubble]Support ZeroBubble Pipeline (hpcaitech#6034)

  Show description for e79d442
  

  duanjunwen

  authored and

  flybird11111

  committed
  e79d442

• [fp8] fix missing fp8_comm flag in mixtral (hpcaitech#6057)

  botbw

  authored
  696fced

• fix

  wangbluo

  committed
  0b14a55

• fix

  wangbluo

  committed
  0ad3129

• fix

  wangbluo

  committed
  b582319

Commits on Sep 14, 2024

• [fp8] Disable all_gather intranode. Disable Redundant all_gather fp8 (hpcaitech#6059)

  Show description for f20b066
  GuangyaoZhang

  authored
  f20b066

• [doc] FP8 training and communication document (hpcaitech#6050)

  Show description for bdb125f
  GuangyaoZhang

  authored
  bdb125f

• fix

  wangbluo

  committed
  827ef3e

• Merge pull request hpcaitech#6061 from wangbluo/sp_fix

  Show description for 37e3523
  wangbluo

  authored
  37e3523

Commits on Sep 16, 2024

• fix

  wangbluo

  committed
  10e4f7d

Commits on Sep 18, 2024

• Merge pull request hpcaitech#6064 from wangbluo/fix_attn

  Show description for 63314ce
  wangbluo

  authored
  63314ce

• [moe] add parallel strategy for shared_expert && fix test for deepseek (hpcaitech#6063)

  botbw

  authored
  4fa6b95

• [ColossalEval] support for vllm (hpcaitech#6056)

  Show description for f9546ba
  

  Camille7777

  and

  pre-commit-ci[bot]

  authored
  f9546ba

Commits on Sep 19, 2024

• [release] update version (hpcaitech#6062)

  ver217

  authored
  dabc2e7

Commits on Sep 29, 2024

• Merge branch 'main' of github.com:flybird11111/ColossalAI into main

  flybird11111

  committed
  b3b3278

• [feat] add zerobubble pp (just a frame now); add POC test for dx_dw; add test for zerobubble;

  

  duanjunwen

  authored and

  flybird11111

  committed
  da4595a

• [update] update text;

  

  duanjunwen

  authored and

  flybird11111

  committed
  e450dd2

• [feat] add test run_fwd_bwd automatic scheduling;

  

  duanjunwen

  authored and

  flybird11111

  committed
  ccc37a4

• [feat] fix poc format

  

  duanjunwen

  authored and

  flybird11111

  committed
  228d71e

• [fix] fix poc test; add comments in poc;

  

  duanjunwen

  authored and

  flybird11111

  committed
  8b0ffed

• [feat] add optim backward_b_by_grad

  

  duanjunwen

  authored and

  flybird11111

  committed
  97f2443

• [feat] fix optimizer bwd b & w; support return accum loss & output

  

  duanjunwen

  authored and

  flybird11111

  committed
  c90bd98

• [fix] fix optim bwd; add license for v_schedule; remove redundant attributes; fix schedule loop "while"--> "for"; add communication dict;

  

  duanjunwen

  authored and

  flybird11111

  committed
  5df5965

• [feat] update test; rm comments;

  

  duanjunwen

  authored and

  flybird11111

  committed
  94a12f6

• [fix] rm zbv in hybridplugin

  

  duanjunwen

  authored and

  flybird11111

  committed
  cc5e7dc

• [fix] fix optim bwd;

  

  duanjunwen

  authored and

  flybird11111

  committed
  ad8ad64

• [fix] fix optim bwd;

  

  duanjunwen

  authored and

  flybird11111

  committed
  f347591

• [fix] rm output.data after send fwd;

  

  duanjunwen

  authored and

  flybird11111

  committed
  4249a36

• [fix] fix bwd step if condition; remove useless comments and format info;

  

  duanjunwen

  authored and

  flybird11111

  committed
  497d545

• [fix] fix mem check;

  

  duanjunwen

  authored and

  flybird11111

  committed
  0825700

• [fix] fix mem assertation

  

  duanjunwen

  authored and

  flybird11111

  committed
  ae4cf5b

• [fix] fix mem; use a new model shape; only assert mem less and equal than theo;

  

  duanjunwen

  authored and

  flybird11111

  committed
  e80179c

• [fix] fix model zoo import;

  

  duanjunwen

  authored and

  flybird11111

  committed
  2683d26

• [feat] moehybrid support zerobubble;

  

  duanjunwen

  authored and

  flybird11111

  committed
  9094cc3

• [fix] fix zerobubble pp for shardformer type input;

  

  duanjunwen

  authored and

  flybird11111

  committed
  3e2f260

• [fix] fix require_grad & deallocate call;

  

  duanjunwen

  authored and

  flybird11111

  committed
  8ce22ae

• [fix] fix mem assert;

  

  duanjunwen

  authored and

  flybird11111

  committed
  f8d6f98

• [fix] fix fwd branch, fwd pass both micro_batch & internal_inputs'

  

  duanjunwen

  authored and

  flybird11111

  committed
  78a439b

• [fix] fix pipeline util func deallocate --> release_tensor_data; fix bwd_b loss bwd branch;

  

  duanjunwen

  authored and

  flybird11111

  committed
  8bc8bb0

• [fix] fix zerobubble; support shardformer model type;

  

  duanjunwen

  authored and

  flybird11111

  committed
  a3a797d

• [fix] fix test_pipeline_utils ci;

  

  duanjunwen

  authored and

  flybird11111

  committed
  4d3eaee

• [plugin] hybrid support zero bubble pipeline (hpcaitech#6060)

  Show description for 2fd9d3e
  

  3 peoplecommitted
  2fd9d3e

• [feat] add zerobubble pp (just a frame now); add POC test for dx_dw; add test for zerobubble;

  

  duanjunwen

  authored and

  flybird11111

  committed
  21c62b6

• [update] update text;

  

  duanjunwen

  authored and

  flybird11111

  committed
  28ee5a7

• [feat] add test run_fwd_bwd automatic scheduling;

  

  duanjunwen

  authored and

  flybird11111

  committed
  d44e7e6

• [feat] fix poc format

  

  duanjunwen

  authored and

  flybird11111

  committed
  49d68eb

• [fix] fix poc test; add comments in poc;

  

  duanjunwen

  authored and

  flybird11111

  committed
  0055c47

• [feat] add optim backward_b_by_grad

  

  duanjunwen

  authored and

  flybird11111

  committed
  21bf510

• [feat] fix optimizer bwd b & w; support return accum loss & output

  

  duanjunwen

  authored and

  flybird11111

  committed
  93ede6b

• [fix] fix optim bwd; add license for v_schedule; remove redundant attributes; fix schedule loop "while"--> "for"; add communication dict;

  

  duanjunwen

  authored and

  flybird11111

  committed
  4ac0d6e

• [feat] update test; rm comments;

  

  duanjunwen

  authored and

  flybird11111

  committed
  262b27e

• [fix] fix optim bwd;

  

  duanjunwen

  authored and

  flybird11111

  committed
  fe99ca3

• [fix] fix optim bwd;

  

  duanjunwen

  authored and

  flybird11111

  committed
  355a3af

• [fix] rm output.data after send fwd;

  

  duanjunwen

  authored and

  flybird11111

  committed
  4420dc1

• [fix] fix bwd step if condition; remove useless comments and format info;

  

  duanjunwen

  authored and

  flybird11111

  committed
  7ba031d

• [fix] fix mem check;

  

  duanjunwen

  authored and

  flybird11111

  committed
  e666f5c

• [fix] fix mem assertation

  

  duanjunwen

  authored and

  flybird11111

  committed
  93b3604

• [fix] fix mem; use a new model shape; only assert mem less and equal than theo;

  

  duanjunwen

  authored and

  flybird11111

  committed
  78ed432

• [fix] fix model zoo import;

  

  duanjunwen

  authored and

  flybird11111

  committed
  df12ae7

• [fix] fix mem assert;

  

  duanjunwen

  authored and

  flybird11111

  committed
  9e90356

• [fix] fix fwd branch, fwd pass both micro_batch & internal_inputs'

  

  duanjunwen

  authored and

  flybird11111

  committed
  993f3db

• [plugin] hybrid support zero bubble pipeline (hpcaitech#6060)

  Show description for 0767948
  

  3 peoplecommitted
  0767948

• Merge branch 'feature/zerobubble' into feature/zerobubble

  flybird11111

  authored
  3251e68

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 797d1ed
  pre-commit-ci[bot]

  committed
  797d1ed

Commits on Oct 8, 2024

• Merge branch 'feature/zerobubble' of github.com:flybird11111/ColossalAI into feature/zerobubble

  flybird11111

  committed
  1637c14

• zbv support zero

  flybird11111

  committed
  a5f0670

• suport zbv all

  flybird11111

  committed
  42f2d0b

• example support zbv

  flybird11111

  committed
  d50c0a1

• fix

  flybird11111

  committed
  b9ac0a6