Code

Dev/optimizer offloading#1205

Code

Open

lostkevin wants to merge 32 commits intoNVIDIA:mainNVIDIA/Megatron-LM:mainfrom lostkevin:dev/optimizer_offloadinglostkevin/PAI-Megatron-LM-240926:dev/optimizer_offloadingCopy head branch name to clipboard

+4,671-17

Commits

Commits on Oct 8, 2024

• init commit

  lostkevin

  committed
  876d926

• update optimizer offloading to 0926

  lostkevin

  committed
  0ed6b43

Commits on Oct 9, 2024

• fix checkpointing issue

  lostkevin

  committed
  3e39676

• fix tp comm overlap

  lostkevin

  committed
  3e868ae

• fix cpu adam op

  lostkevin

  committed
  54a085f

• add model chunk kwargs

  lostkevin

  committed
  07ee88f

• fix compat issue of static optimizer offload in 0926

  lostkevin

  committed
  d724396

• fix bug in auto optimizer offload

  lostkevin

  committed
  490c257

Commits on Oct 14, 2024

• add optimizer offload arguments

  shjwudp

  committed
  2ba77c8

Commits on Oct 15, 2024

• remove deps of megatron-patch

  lostkevin

  committed
  df403ff

Commits on Oct 16, 2024

• fix arguments.py

  

  root

  committed
  465fc48

• fix arguments.py

  

  root

  committed
  5f89336

• Merge remote-tracking branch 'lostkevin/dev/optimizer_offloading' into optimizer_offloading_oct14

  shjwudp

  committed
  0545670

Commits on Oct 22, 2024

• init hybrid optimizer

  shjwudp

  committed
  81c0d97

Commits on Oct 23, 2024

• init abstract class hybrid_device_optimizer

  shjwudp

  committed
  c8bea0b

• Code tidying, fixes

  shjwudp

  committed
  d292946

• remove unused change

  shjwudp

  committed
  9a24930

Commits on Oct 25, 2024

• Merge pull request #1 from shjwudp/optimizer_offloading_oct14

  Show description for c23b674
  shjwudp

  authored
  c23b674

Commits on Nov 5, 2024

• add pre-allocation for each cpu grad

  lostkevin

  committed
  7bed7d1

Commits on Nov 6, 2024

• overlap GPU step / CPU step with two streams

  lostkevin

  committed
  3690420

• HDO e2e in MCore

  shjwudp

  committed
  dfa77a7

Commits on Nov 7, 2024

• add naive benchmark and fix bug

  lostkevin

  committed
  83e784a

• Merge pull request #2 from lostkevin/dev/hdo_io

  Show description for 660a079
  shjwudp

  authored
  660a079

Commits on Nov 8, 2024

• distopt support end to end

  shjwudp

  committed
  1b09ea0

• Merge remote-tracking branch 'lostkevin/dev/optimizer_offloading' into optimizer_offloading_oct14

  shjwudp

  committed
  e838c6d

Commits on Dec 4, 2024

• bug fix & UT update

  shjwudp

  committed
  b145fb3

• Merge pull request #3 from shjwudp/optimizer_offloading_oct14

  Show description for f98bafb
  lostkevin

  authored
  f98bafb

Commits on Dec 26, 2024

• support dist-checkpoint for hdo + distopt

  shjwudp

  committed
  ea7e876

• fix ..

  shjwudp

  committed
  b905841

• remove cuda sync in HDO

  shjwudp

  committed
  341b413

Commits on Dec 27, 2024

• add layer offset for state step dist-ckpt and donot sync sub-optimizers param_groups to hdo

  shjwudp

  committed
  28ada57

• Merge pull request #7 from shjwudp/optimizer_cpu_offload_dev_dec22

  Show description for df617a8
  lostkevin

  authored
  df617a8