Code

feat(zero): add zero35#1

Code

Open

SolenoidWGT wants to merge 11 commits intomasterpjlab-sys4nlp/DeepSpeed:masterfrom feat/add_zero35pjlab-sys4nlp/DeepSpeed:feat/add_zero35Copy head branch name to clipboard

+1,245-85

Commits

Commits on Oct 11, 2023

• feat(zero): add zero35

  SolenoidWGT

  committed
  a019c50

  Copy full SHA for a019c50

Commits on Oct 12, 2023

• fix

  SolenoidWGT

  committed
  0766091

  Copy full SHA for 0766091

Commits on Oct 13, 2023

• fix zero35_g_p_all_gather_coalesced reshape

  SolenoidWGT

  committed
  6ea0571

  Copy full SHA for 6ea0571

• format some api

  SolenoidWGT

  committed
  2a2983e

  Copy full SHA for 2a2983e

• debug low tflop

  SolenoidWGT

  committed
  999f4f2

  Copy full SHA for 999f4f2

Commits on Oct 14, 2023

• fix no zero35 bug

  SolenoidWGT

  committed
  f643ea8

  Copy full SHA for f643ea8

• comment some debug print to avoid cuda tensor print

  SolenoidWGT

  committed
  4a2a32f

  Copy full SHA for 4a2a32f

• debug code

  SolenoidWGT

  committed
  0d5ca75

  Copy full SHA for 0d5ca75

Commits on Oct 17, 2023

• feat: add fake hierarchical comm for zero35

  SolenoidWGT

  committed
  f142782

  Copy full SHA for f142782

• feat: support zero35 shared > 8

  SolenoidWGT

  committed
  90478cd

  Copy full SHA for 90478cd

• Merge branch 'feat/add_zero35' of https://github.com/pjlab-sys4nlp/DeepSpeed into feat/add_zero35

  SolenoidWGT

  committed
  32dede4

  Copy full SHA for 32dede4