Code

Draft PR Adding mistral 0.1#1131

Code

Merged

Quentin-Anthony merged 69 commits intoEleutherAI:mainEleutherAI/gpt-neox:mainfrom AIproj:adding-mistral-0.1AIproj/gpt-neox:adding-mistral-0.1Copy head branch name to clipboardFeb 23, 2024

Commits

Commits on Aug 9, 2023

• add support for flash attention 2

  zhangir-azerbayev

  committed
  ab38f60

  Copy full SHA for ab38f60

• change cosine decay to chinchilla style

  zhangir-azerbayev

  committed
  840c09f

  Copy full SHA for 840c09f

• set default warmup to none so that warmup_iters can be set

  zhangir-azerbayev

  committed
  ae26360

  Copy full SHA for ae26360

• fixed bug

  zhangir-azerbayev

  committed
  bf4cab5

  Copy full SHA for bf4cab5

• fixed chinchilla lr

  zhangir-azerbayev

  committed
  ff86462

  Copy full SHA for ff86462

• add s3 checkpoint syncing

  haileyschoelkopf

  committed
  757320b

  Copy full SHA for 757320b

• Merge branch 'add-s3-ckpting' into math-lm-2

  haileyschoelkopf

  committed
  a765819

  Copy full SHA for a765819

Commits on Aug 10, 2023

• rotary embedding in fp32

  zhangir-azerbayev

  committed
  8a11029

  Copy full SHA for 8a11029

• fix for seq_len < max_seq_len

  zhangir-azerbayev

  committed
  d869e47

  Copy full SHA for d869e47

Commits on Aug 11, 2023

• some fixes, still not working

  zhangir-azerbayev

  committed
  52ba5e4

  Copy full SHA for 52ba5e4

• ?'

  Show description for dfedf05
  zhangir-azerbayev

  committed
  dfedf05

  Copy full SHA for dfedf05

Commits on Aug 12, 2023

• Merge branch 'fix_rotary_precision' into math-lm-2-rotary

  zhangir-azerbayev

  committed
  e680f68

  Copy full SHA for e680f68

Commits on Aug 13, 2023

• fix bugs; evaluate on step 0

  zhangir-azerbayev

  committed
  fcbd8a1

  Copy full SHA for fcbd8a1

• Merge branch 'math-lm-2' into math-lm-2-rotary

  haileyschoelkopf

  committed
  416bafa

  Copy full SHA for 416bafa

Commits on Aug 23, 2023

• first attempt at gqa

  zhangir-azerbayev

  committed
  334bbd5

  Copy full SHA for 334bbd5

Commits on Sep 8, 2023

• gqa works in kv_heads==query_heads case

  haileyschoelkopf

  committed
  3c8616f

  Copy full SHA for 3c8616f

Commits on Sep 15, 2023

• gqa working

  haileyschoelkopf

  committed
  e59c873

  Copy full SHA for e59c873

• workaround for FSX quota

  haileyschoelkopf

  committed
  801192e

  Copy full SHA for 801192e

Commits on Oct 5, 2023

• update with llemma

  zhangir-azerbayev

  committed
  e52b749

  Copy full SHA for e52b749

• update with recent PR

  zhangir-azerbayev

  committed
  6bc724b

  Copy full SHA for 6bc724b

Commits on Oct 25, 2023

• README and requirements updated

  AIproj

  committed
  48d394e

  Copy full SHA for 48d394e

• Added Mistral config

  AIproj

  committed
  694bc7f

  Copy full SHA for 694bc7f

• Added sliding window through flash attention 2

  AIproj

  committed
  612de29

  Copy full SHA for 612de29

• Added sliding window

  AIproj

  committed
  9bd58f1

  Copy full SHA for 9bd58f1

Commits on Oct 27, 2023

• Mistral should likely use mp=2 like llama2

  AIproj

  committed
  d5d90dc

  Copy full SHA for d5d90dc

Commits on Nov 1, 2023

• Update gitignore

  AIproj

  committed
  67638e1

  Copy full SHA for 67638e1

• Removed unused CPCargo import

  AIproj

  committed
  b521408

  Copy full SHA for b521408

• Conversion script (WIP)

  AIproj

  committed
  c842ea9

  Copy full SHA for c842ea9

• Fixed missing slurm environ vars

  AIproj

  committed
  aa50fd1

  Copy full SHA for aa50fd1

Commits on Nov 8, 2023

• updated mistral config

  AIproj

  committed
  6a86310

  Copy full SHA for 6a86310

• updated job script

  AIproj

  committed
  b5f2c6a

  Copy full SHA for b5f2c6a

• initial commit conversion mistral hf to sequential

  AIproj

  committed
  44e0397

  Copy full SHA for 44e0397

Commits on Nov 12, 2023

• Added stacking q, k, v appropriately for mp ranks

  AIproj

  committed
  fa71c63

  Copy full SHA for fa71c63

Commits on Jan 20, 2024

• pp=0 support from end of 2023

  AIproj

  committed
  e263367

  Copy full SHA for e263367

Commits on Jan 25, 2024

• Cleaning up config and removing Autoconfig in conversion script

  AIproj

  committed
  bcfb279

  Copy full SHA for bcfb279

• Cleaned up conversion example script

  AIproj

  committed
  753ef0f

  Copy full SHA for 753ef0f

Commits on Feb 13, 2024

• cleanup: add back configs folder, discard Llemma readme

  haileyschoelkopf

  committed
  3488dae

  Copy full SHA for 3488dae

• cleanup: remove llemma lr sched changes, re-add requirements/ folder

  haileyschoelkopf

  committed
  a55d69c

  Copy full SHA for a55d69c

• docs: add explanation of intermediate_size behavior

  haileyschoelkopf

  committed
  a521a82

  Copy full SHA for a521a82

• args: add argument checking for num_kv_heads, clean up usage syntax

  haileyschoelkopf

  committed
  4df0c4e

  Copy full SHA for 4df0c4e

• args: prevent num KV heads < TP worldsize

  haileyschoelkopf

  committed
  beb66d4

  Copy full SHA for beb66d4

• readd triton flash attn func

  haileyschoelkopf

  committed
  08f80fe

  Copy full SHA for 08f80fe

• cleanup: use tools/ dir from main

  haileyschoelkopf

  committed
  7325880

  Copy full SHA for 7325880

• docs: re-add mistral , GQA as supported

  haileyschoelkopf

  committed
  9b2331f

  Copy full SHA for 9b2331f

• cleanup: delete duplicate tools/ files

  haileyschoelkopf

  committed
  d6accd8

  Copy full SHA for d6accd8

• cleanup: use fp32 rope (non-fused) from main

  haileyschoelkopf

  committed
  975d8f8

  Copy full SHA for 975d8f8

• cleanup: no longer block out GQA codepaths in conversion scripts

  haileyschoelkopf

  committed
  23b7577

  Copy full SHA for 23b7577

Commits on Feb 14, 2024

• Merge branch 'main' into adding-mistral-0.1

  Quentin-Anthony

  authored
  9704976

  Copy full SHA for 9704976

• cleanup: gqa code a bit

  haileyschoelkopf

  committed
  54135b4

  Copy full SHA for 54135b4

• add llama2, llemma configs

  haileyschoelkopf

  committed
  594d926

  Copy full SHA for 594d926

Commits on Feb 21, 2024

• add non-flash GQA ; refactor modeling code

  haileyschoelkopf

  committed
  0827bb8

  Copy full SHA for 0827bb8

• clean up mistral config for commit

  haileyschoelkopf

  committed
  558bdd8

  Copy full SHA for 558bdd8

• further cleanup configs dir

  haileyschoelkopf

  committed
  726935f

  Copy full SHA for 726935f

• remove slurm script from llemma

  haileyschoelkopf

  committed
  4cec223

  Copy full SHA for 4cec223

• update seqlen params for codellama, llemma configs

  haileyschoelkopf

  committed
  eca632d

  Copy full SHA for eca632d

• add more comments to GQA code, and make reshapes more readable

  haileyschoelkopf

  committed
  b07e63a

  Copy full SHA for b07e63a

Commits on Feb 22, 2024

• make inv_freq non-persistent

  haileyschoelkopf

  committed
  f0dcf17

  Copy full SHA for f0dcf17

• actually, just ensure mistral has inv_freqs as a persistent buffer

  haileyschoelkopf

  committed
  95afe82

  Copy full SHA for 95afe82

• non-flash GQA works, so ensure arguments.py permits it

  haileyschoelkopf

  committed
  5cfe8ee

  Copy full SHA for 5cfe8ee

• no longer use our own copies of flash attention interface functions

  haileyschoelkopf

  committed
  627a287

  Copy full SHA for 627a287

• remove unused mpu util fn

  haileyschoelkopf

  committed
  63c2fbe

  Copy full SHA for 63c2fbe

• delete unused config file

  haileyschoelkopf

  committed
  e768492

  Copy full SHA for e768492

• fix diff on mpu/utils.py

  haileyschoelkopf

  committed
  caa440d

  Copy full SHA for caa440d

• remove slurm scripts that won't be in this PR

  haileyschoelkopf

  committed
  74fde98

  Copy full SHA for 74fde98

• run pre-commit

  haileyschoelkopf

  committed
  e7d1282

  Copy full SHA for e7d1282

• Merge remote-tracking branch 'upstream/main' into adding-mistral-0.1

  haileyschoelkopf

  committed
  50ed9b5

  Copy full SHA for 50ed9b5

• update tests for conversion scripts

  haileyschoelkopf

  committed
  ace0e94

  Copy full SHA for ace0e94

Commits on Feb 23, 2024

• add flash version check for sliding window

  Quentin-Anthony

  committed
  157ec47

  Copy full SHA for 157ec47

• pre-commit

  Quentin-Anthony

  committed
  db9947e

  Copy full SHA for db9947e