Code

Varying experts per layer#43

Code

Open

pglorio wants to merge 85 commits intomainZyphra/Megatron-LM:mainfrom pyramid_moeZyphra/Megatron-LM:pyramid_moeCopy head branch name to clipboard

+97-38

Commits

Commits on Dec 5, 2023

• Update switch_mlp.py

  pglorio

  authored
  9261410

  Copy full SHA for 9261410

• Update switch_mlp.py

  pglorio

  authored
  f2ecc71

  Copy full SHA for f2ecc71

• Update switch_mlp.py

  pglorio

  authored
  c7ea711

  Copy full SHA for c7ea711

Commits on Dec 6, 2023

• Update switch_mlp.py

  pglorio

  authored
  d5f7af4

  Copy full SHA for d5f7af4

• Update transformer_block.py

  pglorio

  authored
  35d3ac4

  Copy full SHA for 35d3ac4

• Update switch_mlp.py

  pglorio

  authored
  0e7eb23

  Copy full SHA for 0e7eb23

• Update transformer_block.py

  pglorio

  authored
  4ad505a

  Copy full SHA for 4ad505a

• Update transformer_block.py

  pglorio

  authored
  3af0059

  Copy full SHA for 3af0059

• Update transformer_block.py

  pglorio

  authored
  13c54d9

  Copy full SHA for 13c54d9

Commits on Dec 7, 2023

• Update arguments.py

  pglorio

  authored
  df2eeb2

  Copy full SHA for df2eeb2

• Update initialize.py

  pglorio

  authored
  1f07db5

  Copy full SHA for 1f07db5

• Update initialize.py

  pglorio

  authored
  b9d6475

  Copy full SHA for b9d6475

• Update initialize.py

  pglorio

  authored
  1c20628

  Copy full SHA for 1c20628

• Update initialize.py

  pglorio

  authored
  413bf43

  Copy full SHA for 413bf43

• Update initialize.py

  pglorio

  authored
  9237b34

  Copy full SHA for 9237b34

• Update arguments.py

  pglorio

  authored
  01dc1e9

  Copy full SHA for 01dc1e9

• Update arguments.py

  pglorio

  authored
  68b99a0

  Copy full SHA for 68b99a0

• Update pretrain_gpt.py

  pglorio

  authored
  28bc3a3

  Copy full SHA for 28bc3a3

• Update arguments.py

  pglorio

  authored
  b819330

  Copy full SHA for b819330

• Update transformer_block.py

  pglorio

  authored
  56ad120

  Copy full SHA for 56ad120

• Update switch_mlp.py

  pglorio

  authored
  8ce18aa

  Copy full SHA for 8ce18aa

• Update arguments.py

  pglorio

  authored
  0875fc5

  Copy full SHA for 0875fc5

• Update transformer_block.py

  pglorio

  authored
  4989c58

  Copy full SHA for 4989c58

• Update transformer_block.py

  pglorio

  authored
  ef743f4

  Copy full SHA for ef743f4

• Update switch_mlp.py

  pglorio

  authored
  25056d5

  Copy full SHA for 25056d5

• Update switch_mlp.py

  pglorio

  authored
  7dad158

  Copy full SHA for 7dad158

• Update switch_mlp.py

  pglorio

  authored
  2aee703

  Copy full SHA for 2aee703

• Update switch_mlp.py

  pglorio

  authored
  babbebb

  Copy full SHA for babbebb

• Update switch_mlp.py

  pglorio

  authored
  179d2a4

  Copy full SHA for 179d2a4

• Update transformer_block.py

  pglorio

  authored
  253c87c

  Copy full SHA for 253c87c

• Update README.md

  pglorio

  authored
  a058ac2

  Copy full SHA for a058ac2

• Merge pull request #47 from Zyphra/main

  Show description for c178073
  Quentin-Anthony

  authored
  c178073

  Copy full SHA for c178073

• Update mlp.py

  pglorio

  authored
  cec3437

  Copy full SHA for cec3437

• Update switch_mlp.py

  pglorio

  authored
  381e899

  Copy full SHA for 381e899

• Update transformer_layer.py

  pglorio

  authored
  cd4b5b1

  Copy full SHA for cd4b5b1

• Update mlp.py

  pglorio

  authored
  3c0a58f

  Copy full SHA for 3c0a58f

• Update switch_mlp.py

  pglorio

  authored
  2644ff8

  Copy full SHA for 2644ff8

• Update mlp.py

  pglorio

  authored
  911ceeb

  Copy full SHA for 911ceeb

• Update arguments.py

  pglorio

  authored
  7d90105

  Copy full SHA for 7d90105

• Update mlp.py

  pglorio

  authored
  0ff85c7

  Copy full SHA for 0ff85c7

• Update mlp.py

  pglorio

  authored
  e91b587

  Copy full SHA for e91b587

• Update mlp.py

  pglorio

  authored
  7f7d9ab

  Copy full SHA for 7f7d9ab

• Update README.md

  pglorio

  authored
  1f052fc

  Copy full SHA for 1f052fc

• Update mlp.py

  pglorio

  authored
  25dd9a8

  Copy full SHA for 25dd9a8

• Update mlp.py

  pglorio

  authored
  e0f37f1

  Copy full SHA for e0f37f1

Commits on Dec 14, 2023

• Update switch_mlp.py

  pglorio

  authored
  7e4a6ae

  Copy full SHA for 7e4a6ae

• Update switch_mlp.py

  pglorio

  authored
  42be8d4

  Copy full SHA for 42be8d4

• Update switch_mlp.py

  pglorio

  authored
  f787add

  Copy full SHA for f787add

Commits on Dec 15, 2023

• Update gpt_model.py

  pglorio

  authored
  03b50ec

  Copy full SHA for 03b50ec

• Update switch_mlp.py

  pglorio

  authored
  02969e7

  Copy full SHA for 02969e7

Commits on Dec 19, 2023

• Update switch_mlp.py

  pglorio

  authored
  fb61494

  Copy full SHA for fb61494

• Update training.py

  pglorio

  authored
  e26ca57

  Copy full SHA for e26ca57

Commits on Dec 20, 2023

• Update switch_mlp.py

  pglorio

  authored
  83bcde1

  Copy full SHA for 83bcde1

• Update switch_mlp.py

  pglorio

  authored
  a8ac369

  Copy full SHA for a8ac369

• Update switch_mlp.py

  pglorio

  authored
  c2edfbb

  Copy full SHA for c2edfbb

• print statement of data shape

  pglorio

  authored
  74b0c7b

  Copy full SHA for 74b0c7b

• print statement of data shape

  pglorio

  authored
  8f7bb1c

  Copy full SHA for 8f7bb1c

• Update transformer_layer.py

  pglorio

  authored
  714cd8d

  Copy full SHA for 714cd8d

• Update gpt_model.py

  pglorio

  authored
  5fd134f

  Copy full SHA for 5fd134f

• Update training.py

  pglorio

  authored
  d95239c

  Copy full SHA for d95239c

• Update gpt_model.py

  pglorio

  authored
  8ac39c9

  Copy full SHA for 8ac39c9

• Update gpt_model.py

  pglorio

  authored
  687687e

  Copy full SHA for 687687e

Commits on Dec 21, 2023

• Update gpt_model.py

  pglorio

  authored
  fff6470

  Copy full SHA for fff6470

• Update gpt_model.py

  pglorio

  authored
  fac5f79

  Copy full SHA for fac5f79

• Update gpt_model.py

  pglorio

  authored
  b157390

  Copy full SHA for b157390

• Update pretrain_gpt.py

  pglorio

  authored
  214e0d0

  Copy full SHA for 214e0d0

• Update gpt_model.py

  pglorio

  authored
  0597d75

  Copy full SHA for 0597d75

• Update transformer_layer.py

  pglorio

  authored
  2b017d1

  Copy full SHA for 2b017d1

Commits on Dec 29, 2023

• Update switch_mlp.py

  pglorio

  authored
  09b1228

  Copy full SHA for 09b1228

• Update switch_mlp.py

  pglorio

  authored
  3800c0e

  Copy full SHA for 3800c0e

• Update gpt_model.py

  pglorio

  authored
  5a583a2

  Copy full SHA for 5a583a2

• Update fused_softmax.py

  pglorio

  authored
  5123488

  Copy full SHA for 5123488

• Update fused_softmax.py

  pglorio

  authored
  0d6f7c9

  Copy full SHA for 0d6f7c9

• Update fused_softmax.py

  pglorio

  authored
  f8c8de9

  Copy full SHA for f8c8de9

• Update fused_softmax.py

  pglorio

  authored
  2bfb7c0

  Copy full SHA for 2bfb7c0

• Update fused_softmax.py

  pglorio

  authored
  36a2376

  Copy full SHA for 36a2376

Commits on Jan 1, 2024

• Update switch_mlp.py

  pglorio

  authored
  7048fc1

  Copy full SHA for 7048fc1

• Update switch_mlp.py

  pglorio

  authored
  431fa63

  Copy full SHA for 431fa63

Commits on Jan 2, 2024

• Update switch_mlp.py

  pglorio

  authored
  fc5390c

  Copy full SHA for fc5390c

• Update switch_mlp.py

  pglorio

  authored
  5791a87

  Copy full SHA for 5791a87

• Update switch_mlp.py

  pglorio

  authored
  03aecf6

  Copy full SHA for 03aecf6

• Update switch_mlp.py

  pglorio

  authored
  0e5b34f

  Copy full SHA for 0e5b34f

• Update switch_mlp.py

  pglorio

  authored
  704726d

  Copy full SHA for 704726d

• Update switch_mlp.py

  pglorio

  authored
  277afeb

  Copy full SHA for 277afeb

• Update switch_mlp.py

  pglorio

  authored
  4a31d0f

  Copy full SHA for 4a31d0f