Code

MoE#639

Code

Open

Muennighoff wants to merge 144 commits intomainallenai/OLMo:mainfrom Muennighoff/MoEallenai/OLMo:Muennighoff/MoECopy head branch name to clipboard

+1,896-15

Commits

Commits on Jun 20, 2024

• Clean MoE implementation

  Muennighoff

  committed
  e725eb9

  Copy full SHA for e725eb9

• Add conf

  Muennighoff

  committed
  db24750

  Copy full SHA for db24750

• Fix return args

  Muennighoff

  committed
  18450de

  Copy full SHA for 18450de

• Rmv outdated kwarg

  Muennighoff

  committed
  4ab7f77

  Copy full SHA for 4ab7f77

• Rmv legacy kwarg

  Muennighoff

  committed
  dba42fd

  Copy full SHA for dba42fd

• Merge branch 'Muennighoff/MoE' of github.com:allenai/LLM into Muennighoff/MoE

  Muennighoff

  committed
  6c5f8a3

  Copy full SHA for 6c5f8a3

• Add distributed_strategy

  Muennighoff

  committed
  6a8e089

  Copy full SHA for 6a8e089

• Allow w/o weight attr

  Muennighoff

  committed
  1a9a317

  Copy full SHA for 1a9a317

• Merge branch 'Muennighoff/MoE' of github.com:allenai/LLM into Muennighoff/MoE

  Muennighoff

  committed
  ddf6fd4

  Copy full SHA for ddf6fd4

• Allow w/o weight attr

  Muennighoff

  committed
  ab55e07

  Copy full SHA for ab55e07

• Add MoE params

  Muennighoff

  committed
  7aeefd4

  Copy full SHA for 7aeefd4

• Rmv kwarg

  Muennighoff

  committed
  3eab45c

  Copy full SHA for 3eab45c

Commits on Jun 21, 2024

• Reduce lb & moe losses

  Muennighoff

  committed
  6d736da

  Copy full SHA for 6d736da

• LN & Emb Dec

  Muennighoff

  committed
  d07c638

  Copy full SHA for d07c638

• Merge branch 'Muennighoff/MoE' of github.com:allenai/LLM into Muennighoff/MoE

  Muennighoff

  committed
  cdb592f

  Copy full SHA for cdb592f

• Do not decay emb

  Muennighoff

  committed
  1399841

  Copy full SHA for 1399841

• Tmp - debug throughput

  Muennighoff

  committed
  a13b5b8

  Copy full SHA for a13b5b8

• Fix

  Muennighoff

  committed
  935167e

  Copy full SHA for 935167e

• Fix

  Muennighoff

  committed
  b96972d

  Copy full SHA for b96972d

• maintain init order

  Muennighoff

  committed
  0079490

  Copy full SHA for 0079490

• Merge branch 'Muennighoff/MoE' of github.com:allenai/LLM into Muennighoff/MoE

  Muennighoff

  committed
  8b1c441

  Copy full SHA for 8b1c441

• Decay emb

  Muennighoff

  committed
  e2c7286

  Copy full SHA for e2c7286

• Keep EA on CPU

  Muennighoff

  committed
  d39a37c

  Copy full SHA for d39a37c

• Do not decay emb

  Muennighoff

  committed
  3acfc04

  Copy full SHA for 3acfc04

• Change norm

  Muennighoff

  committed
  4432261

  Copy full SHA for 4432261

Commits on Jun 22, 2024

• Confs

  Muennighoff

  committed
  cef7707

  Copy full SHA for cef7707

• Adapt wrap

  Muennighoff

  committed
  2a6df33

  Copy full SHA for 2a6df33

Commits on Jun 23, 2024

• Add conf

  Muennighoff

  committed
  7421890

  Copy full SHA for 7421890

Commits on Jun 24, 2024

• decemb conf

  Muennighoff

  committed
  021974e

  Copy full SHA for 021974e

• Updates

  Muennighoff

  committed
  d5a0626

  Copy full SHA for d5a0626

Commits on Jun 25, 2024

• up

  Muennighoff

  committed
  fce086f

  Copy full SHA for fce086f

• Update weka

  Muennighoff

  committed
  daa7c91

  Copy full SHA for daa7c91

• Rev weka

  Muennighoff

  committed
  3a40b7f

  Copy full SHA for 3a40b7f

• Adapt

  Muennighoff

  committed
  2676d03

  Copy full SHA for 2676d03

• Add fine

  Muennighoff

  committed
  2fb4c96

  Copy full SHA for 2fb4c96

Commits on Jun 26, 2024

• Adapt

  Muennighoff

  committed
  448a9a8

  Copy full SHA for 448a9a8

• Adapt

  Muennighoff

  committed
  e361918

  Copy full SHA for e361918

• Add eps

  Muennighoff

  committed
  9377aa5

  Copy full SHA for 9377aa5

Commits on Jun 27, 2024

• Update confs

  Muennighoff

  committed
  bad3a34

  Copy full SHA for bad3a34

• Change eps

  Muennighoff

  committed
  3dbd156

  Copy full SHA for 3dbd156

Commits on Jun 28, 2024

• Add conf

  Muennighoff

  committed
  a6496d6

  Copy full SHA for a6496d6

• Confs

  Muennighoff

  committed
  5dd6135

  Copy full SHA for 5dd6135

Commits on Jun 29, 2024

• Add conf

  Muennighoff

  committed
  195c77d

  Copy full SHA for 195c77d

• Fix path

  Muennighoff

  committed
  4ea9f0a

  Copy full SHA for 4ea9f0a

• Fix path

  Muennighoff

  committed
  44fa5ae

  Copy full SHA for 44fa5ae

Commits on Jun 30, 2024

• Adapt confs

  Muennighoff

  committed
  b7658eb

  Copy full SHA for b7658eb

• add s3 conf

  Muennighoff

  committed
  16edece

  Copy full SHA for 16edece

• fix paths

  Muennighoff

  committed
  f8e061f

  Copy full SHA for f8e061f

Commits on Jul 1, 2024

• Update confs

  Muennighoff

  committed
  c8a51cf

  Copy full SHA for c8a51cf

• Increase bs

  Muennighoff

  committed
  b95a05f

  Copy full SHA for b95a05f

• Adjust paths

  Muennighoff

  committed
  20103ec

  Copy full SHA for 20103ec

• conf

  Muennighoff

  committed
  459ef27

  Copy full SHA for 459ef27

• confs

  Muennighoff

  committed
  84f21db

  Copy full SHA for 84f21db

• confs

  Muennighoff

  committed
  08d3253

  Copy full SHA for 08d3253

• Update confs

  Muennighoff

  committed
  4b07140

  Copy full SHA for 4b07140

• Update conf

  Muennighoff

  committed
  cf6fa33

  Copy full SHA for cf6fa33

• Conf

  Muennighoff

  committed
  d95f978

  Copy full SHA for d95f978

Commits on Jul 2, 2024

• Finegrained

  Muennighoff

  committed
  3c15abf

  Copy full SHA for 3c15abf

• Cx5

  Muennighoff

  committed
  4163e70

  Copy full SHA for 4163e70

• Add conf

  Muennighoff

  committed
  aae0e0b

  Copy full SHA for aae0e0b

• Add cx5

  Muennighoff

  committed
  76e6e6d

  Copy full SHA for 76e6e6d

• Up confs

  Muennighoff

  committed
  31e387b

  Copy full SHA for 31e387b

• Up confs

  Muennighoff

  committed
  f631d8e

  Copy full SHA for f631d8e

Commits on Jul 4, 2024

• Dfix

  Muennighoff

  committed
  142720b

  Copy full SHA for 142720b

• reddit

  Muennighoff

  committed
  412a55e

  Copy full SHA for 412a55e

• flan

  Muennighoff

  committed
  669bba5

  Copy full SHA for 669bba5

• fine

  Muennighoff

  committed
  7d90908

  Copy full SHA for 7d90908

• indent

  Muennighoff

  committed
  3c97519

  Copy full SHA for 3c97519

• Add conf

  Muennighoff

  committed
  6e08b09

  Copy full SHA for 6e08b09

Commits on Jul 5, 2024

• Conf

  Muennighoff

  committed
  1a6a2e2

  Copy full SHA for 1a6a2e2

• Add shared

  Muennighoff

  committed
  103e450

  Copy full SHA for 103e450

• Add

  Muennighoff

  committed
  48b6c14

  Copy full SHA for 48b6c14

• Add conf

  Muennighoff

  committed
  8a387fa

  Copy full SHA for 8a387fa

Commits on Jul 6, 2024

• Add conf

  Muennighoff

  committed
  3203724

  Copy full SHA for 3203724

• Add conf

  Muennighoff

  committed
  5309137

  Copy full SHA for 5309137

• Addqk

  Muennighoff

  committed
  bf66e68

  Copy full SHA for bf66e68

• Make QK Norm parametric

  Muennighoff

  committed
  63c12e1

  Copy full SHA for 63c12e1

Commits on Jul 7, 2024

• dense comp

  Muennighoff

  committed
  1c4aa8d

  Copy full SHA for 1c4aa8d

• add conf

  Muennighoff

  committed
  6a94263

  Copy full SHA for 6a94263

• conf

  Muennighoff

  committed
  b58c316

  Copy full SHA for b58c316

• conf

  Muennighoff

  committed
  3bbfaed

  Copy full SHA for 3bbfaed

Commits on Jul 18, 2024

• Change torch version

  Muennighoff

  committed
  364659f

  Copy full SHA for 364659f

Commits on Jul 20, 2024

• Add anneal

  Muennighoff

  committed
  15f5503

  Copy full SHA for 15f5503

• Resets

  Muennighoff

  committed
  6c516d8

  Copy full SHA for 6c516d8

Commits on Jul 21, 2024

• add conf

  Muennighoff

  committed
  8a0758e

  Copy full SHA for 8a0758e

• conf

  Muennighoff

  committed
  c23b048

  Copy full SHA for c23b048

• Adjust mlp hidden

  Muennighoff

  committed
  285ff10

  Copy full SHA for 285ff10

• Change ratio

  Muennighoff

  committed
  99aec31

  Copy full SHA for 99aec31

• Add conf

  Muennighoff

  committed
  a43eae8

  Copy full SHA for a43eae8

• Conf

  Muennighoff

  committed
  aaefc58

  Copy full SHA for aaefc58

• Adjust

  Muennighoff

  committed
  270271f

  Copy full SHA for 270271f

• add conf

  Muennighoff

  committed
  c084d34

  Copy full SHA for c084d34

• Fix conf

  Muennighoff

  committed
  6789ee2

  Copy full SHA for 6789ee2

• Adjust

  Muennighoff

  committed
  f084fa0

  Copy full SHA for f084fa0

• Add alt

  Muennighoff

  committed
  14ee7e4

  Copy full SHA for 14ee7e4

Commits on Jul 23, 2024

• Fix typo; update script

  Muennighoff

  committed
  f831adf

  Copy full SHA for f831adf

• merge main

  Muennighoff

  committed
  26eb3f3

  Copy full SHA for 26eb3f3

• add moe reorder

  Muennighoff

  committed
  0a3b076

  Copy full SHA for 0a3b076

• Add datafix

  Muennighoff

  committed
  d6ccbf0

  Copy full SHA for d6ccbf0

• Max doc len MoE

  Muennighoff

  committed
  6a2c17e

  Copy full SHA for 6a2c17e

• add fa varlen

  Muennighoff

  committed
  9ccb2f1

  Copy full SHA for 9ccb2f1

• Add conf

  Muennighoff

  committed
  f5291ec

  Copy full SHA for f5291ec

Commits on Jul 24, 2024

• add conf

  Muennighoff

  committed
  ed571a8

  Copy full SHA for ed571a8

• fix conf

  Muennighoff

  committed
  71d2d2a

  Copy full SHA for 71d2d2a

• fixconf

  Muennighoff

  committed
  8b72521

  Copy full SHA for 8b72521

• fixonf

  Muennighoff

  committed
  fc822a0

  Copy full SHA for fc822a0

• fix

  Muennighoff

  committed
  1ccaf9a

  Copy full SHA for 1ccaf9a

Commits on Jul 25, 2024

• Add 8k

  Muennighoff

  committed
  c157441

  Copy full SHA for c157441

Commits on Jul 26, 2024

• add conf

  Muennighoff

  committed
  06b8010

  Copy full SHA for 06b8010

• conf

  Muennighoff

  committed
  cf31e53

  Copy full SHA for cf31e53

Commits on Jul 31, 2024

• Adapt

  Muennighoff

  committed
  a750fde

  Copy full SHA for a750fde

• Merge branch 'main' into Muennighoff/MoE

  Muennighoff

  committed
  c853a43

  Copy full SHA for c853a43

• Sort

  Muennighoff

  committed
  bea80ec

  Copy full SHA for bea80ec

• Simplify configs for merge

  Muennighoff

  committed
  dfdcfc5

  Copy full SHA for dfdcfc5

Commits on Aug 1, 2024

• Update megablocks installation

  Muennighoff

  committed
  4606598

  Copy full SHA for 4606598

• Simplify

  Muennighoff

  committed
  f6c707d

  Copy full SHA for f6c707d

• Simplify

  Muennighoff

  committed
  76f0376

  Copy full SHA for 76f0376

• typing & changes

  Muennighoff

  committed
  9631c80

  Copy full SHA for 9631c80

• Format

  Muennighoff

  committed
  2fa2acb

  Copy full SHA for 2fa2acb

• Change torch.tensor to Any so type check passes?

  Muennighoff

  committed
  c517703

  Copy full SHA for c517703

• Simplify

  Muennighoff

  committed
  b4eb33f

  Copy full SHA for b4eb33f

• Simplify

  Muennighoff

  committed
  ae6f16a

  Copy full SHA for ae6f16a

• Sort

  Muennighoff

  committed
  02781be

  Copy full SHA for 02781be

• Fix type checks

  Muennighoff

  committed
  a62a1ee

  Copy full SHA for a62a1ee

• Format

  Muennighoff

  committed
  0ecd4b8

  Copy full SHA for 0ecd4b8

Commits on Aug 3, 2024

• Fix typo; MoEArgs func

  Muennighoff

  committed
  d8452a0

  Copy full SHA for d8452a0

• Format

  Muennighoff

  committed
  8a28ced

  Copy full SHA for 8a28ced

• Check for act ckpt strategy & moe; fix typo

  Muennighoff

  committed
  91f5553

  Copy full SHA for 91f5553

• fix import

  Muennighoff

  committed
  61ac104

  Copy full SHA for 61ac104

Commits on Aug 20, 2024

• Sort impot

  Muennighoff

  authored
  f4faf8a

  Copy full SHA for f4faf8a

• Merge branch 'main' into Muennighoff/MoE

  Muennighoff

  authored
  fdc1021

  Copy full SHA for fdc1021

Commits on Aug 21, 2024

• Fix typo

  Show description for ed82181
  

  Muennighoff

  and

  2015aroras

  authored
  ed82181

  Copy full SHA for ed82181

• Simplify isinstance

  Show description for b0cc754
  

  Muennighoff

  and

  2015aroras

  authored
  b0cc754

  Copy full SHA for b0cc754

• Clean conf & move constructor

  Muennighoff

  committed
  ca9b41f

  Copy full SHA for ca9b41f

Commits on Sep 4, 2024

• Add ref

  Muennighoff

  committed
  215c0f5

  Copy full SHA for 215c0f5

• Merge main

  Muennighoff

  committed
  43baf74

  Copy full SHA for 43baf74

• Sort imports

  Muennighoff

  committed
  775e514

  Copy full SHA for 775e514

• Format

  Muennighoff

  committed
  cd0004b

  Copy full SHA for cd0004b

Commits on Sep 12, 2024

• No exp ass

  Muennighoff

  committed
  acb23dd

  Copy full SHA for acb23dd

• Revert

  Muennighoff

  committed
  a143469

  Copy full SHA for a143469

• Simplify

  Muennighoff

  committed
  1a4bdae

  Copy full SHA for 1a4bdae

Commits on Oct 3, 2024

• Rmv interleave

  Muennighoff

  committed
  410064a

  Copy full SHA for 410064a

• Merge branch 'main' into Muennighoff/MoE

  Muennighoff

  committed
  671bc8e

  Copy full SHA for 671bc8e

Commits on Oct 4, 2024

• expert_assignments on gpu

  Muennighoff

  committed
  04a2da5

  Copy full SHA for 04a2da5