Code

merge upstream#64

Code

Merged

kshitijkg merged 117 commits intoCERC-AAI:mainCERC-AAI/multimodal:mainfrom EleutherAI:mainEleutherAI/gpt-neox:mainCopy head branch name to clipboardApr 4, 2024

+108,893-3,398

Commits

Commits on Aug 12, 2023

• Fixed final value of cosine decay lr (#1011)

  Show description for eda5aae
  

  kshitijkg

  and

  github-actions

  authored
  eda5aae

  Copy full SHA for eda5aae

• turn hyphens into underscores in merge20b.py (#986)

  wunderalbert

  authored
  01b5e22

  Copy full SHA for 01b5e22

Commits on Aug 22, 2023

• Update Dockerfile (#1014)

  Show description for d8bcd97
  xu-song

  authored
  d8bcd97

  Copy full SHA for d8bcd97

Commits on Aug 28, 2023

• README Update (#1017)

  Show description for 43ea51c
  

  3 peopleauthored
  43ea51c

  Copy full SHA for 43ea51c

Commits on Sep 13, 2023

• Bump transformers version and update enwik8 link (#1024)

  Show description for 2922bef
  

  dashstander

  and

  github-actions

  authored
  2922bef

  Copy full SHA for 2922bef

Commits on Sep 15, 2023

• Fix Generation with Sequential Model (#1026)

  xu-song

  authored
  960ed3d

  Copy full SHA for 960ed3d

• Fix broken link (#1022)

  Show description for 97e376c
  

  3 peopleauthored
  97e376c

  Copy full SHA for 97e376c

• add llama training config (#1023)

  xu-song

  authored
  7821aa7

  Copy full SHA for 7821aa7

• Create README_llama.md (#1027)

  Quentin-Anthony

  authored
  cdc94ee

  Copy full SHA for cdc94ee

• Rename README_llama.md to README.md (#1028)

  Quentin-Anthony

  authored
  737c913

  Copy full SHA for 737c913

• Add llama generation script (#1030)

  xu-song

  authored
  c883e8c

  Copy full SHA for c883e8c

Commits on Sep 18, 2023

• Fix bf16 for zero > 0 and pipeline parallelism > 0 (#1032)

  Show description for d9166bf
  

  dashstander

  and

  github-actions

  authored
  d9166bf

  Copy full SHA for d9166bf

• Remove support for lazy dataset implementation (#1033)

  Show description for fcd5f92
  

  3 peopleauthored
  fcd5f92

  Copy full SHA for fcd5f92

• Fix SequentialWrapper Generation (pipe_parallel_size = 0) (#1031)

  Show description for 70af6e8
  xu-song

  authored
  70af6e8

  Copy full SHA for 70af6e8

Commits on Sep 20, 2023

• integrated flash attention 2 (#1035)

  a663E-36z1120

  authored
  8903a96

  Copy full SHA for 8903a96

• Fix register_buffer parameter (#1036)

  Show description for 0ce77ab
  xu-song

  authored
  0ce77ab

  Copy full SHA for 0ce77ab

• Add flash 2.x message to README.md (#1037)

  Show description for 444c0ef
  

  Quentin-Anthony

  and

  github-actions

  authored
  444c0ef

  Copy full SHA for 444c0ef

Commits on Sep 23, 2023

• Add s3 checkpoint syncing (#1010)

  Show description for f9503b7
  

  3 peopleauthored
  f9503b7

  Copy full SHA for f9503b7

• Fixed final value of linear decay lr (#1039)

  foggy-frost-forest

  authored
  390d37c

  Copy full SHA for 390d37c

• Fix final value of exponential decay lr (#1040)

  Quentin-Anthony

  authored
  e431ff5

  Copy full SHA for e431ff5

Commits on Sep 25, 2023

• Remove the NeoX implementation of GPT2Tokenizer (#1042)

  Show description for 2ab05be
  

  dashstander

  and

  github-actions

  authored
  2ab05be

  Copy full SHA for 2ab05be

• Pre-compute RoPE embeddings in fp32 (#1041)

  Show description for 3bfedf4
  

  3 peopleauthored
  3bfedf4

  Copy full SHA for 3bfedf4

Commits on Sep 27, 2023

• Patch LR Annealing Bug (#1046)

  Show description for ba51ca0
  

  3 peopleauthored
  ba51ca0

  Copy full SHA for ba51ca0

• Improve FLOPS Calculation (#1044)

  Show description for 5f36401
  

  3 peopleauthored
  5f36401

  Copy full SHA for 5f36401

Commits on Sep 28, 2023

• adding boilerplate coverity scan to submit to public analysis (#1047)

  Show description for 5fa85ad
  

  3 peopleauthored
  5fa85ad

  Copy full SHA for 5fa85ad

Commits on Sep 29, 2023

• Add section to the README detailing how to start distributed jobs (#1048)

  Show description for f44db66
  

  3 peopleauthored
  f44db66

  Copy full SHA for f44db66

• Fix readme typos (#1049)

  Show description for 2c60645
  

  Quentin-Anthony

  and

  github-actions

  authored
  2c60645

  Copy full SHA for 2c60645

• Update citation list (#1052)

  Quentin-Anthony

  authored
  b14d6f7

  Copy full SHA for b14d6f7

• Update CITATION.cff (#1053)

  Show description for 93cac79
  

  Quentin-Anthony

  and

  github-actions

  authored
  93cac79

  Copy full SHA for 93cac79

Commits on Oct 1, 2023

• Remove duplicated hf_config (#1054)

  xu-song

  authored
  7a8569f

  Copy full SHA for 7a8569f

Commits on Oct 2, 2023

• Organize the tools directory (#1055)

  Show description for 3f43f07
  

  4 peopleauthored
  3f43f07

  Copy full SHA for 3f43f07

Commits on Oct 4, 2023

• Add documentation about using labelled datasets (#1056)

  Show description for f6ac04d
  

  dashstander

  and

  github-actions

  authored
  f6ac04d

  Copy full SHA for f6ac04d

Commits on Oct 17, 2023

• LR scheduler fix no longer breaks inference (#1060)

  Show description for e001a04
  

  dashstander

  and

  github-actions

  authored
  e001a04

  Copy full SHA for e001a04

Commits on Oct 20, 2023

• Lion Optimizer (#1062)

  Show description for b02d989
  

  andylolu2

  and

  adi-kmt

  authored
  b02d989

  Copy full SHA for b02d989

Commits on Oct 31, 2023

• fix lion optimizer documentation (#1067)

  Show description for e277bc7
  

  jahatef

  and

  github-actions

  authored
  e277bc7

  Copy full SHA for e277bc7

• Fix preprocess_data.py link (#1064)

  Show description for f574f22
  

  Quentin-Anthony

  and

  github-actions

  authored
  f574f22

  Copy full SHA for f574f22

Commits on Nov 1, 2023

• Edge-casing for multi-GPU HF-to-NeoX conversion (#1065)

  Show description for fcc5af5
  

  3 peopleauthored
  fcc5af5

  Copy full SHA for fcc5af5

• Create tools __init__.py for import (#1068)

  Quentin-Anthony

  authored
  8c9fc00

  Copy full SHA for 8c9fc00

• Pin version of lm_eval (#1070)

  Show description for a10f69c
  

  haileyschoelkopf

  and

  github-actions

  authored
  a10f69c

  Copy full SHA for a10f69c

• fixed case when ntasks_per_node is used instead (#1069)

  AIproj

  authored
  41f019e

  Copy full SHA for 41f019e

Commits on Nov 5, 2023

• Update README.md

  StellaAthena

  authored
  90aa131

  Copy full SHA for 90aa131

Commits on Nov 7, 2023

• When processing mlp.dense_4h_to_h.bias and attention.dense.bias, tp_ranks are not reflected, so strange results always appear when tp_ranks is greater than 1.

  kyuheejang

  committed
  04dc2ba

  Copy full SHA for 04dc2ba

• Merge pull request #1072 from kyuheejang/Fixing-neox-to-huggingface

  Show description for f214358
  StellaAthena

  authored
  f214358

  Copy full SHA for f214358

Commits on Nov 8, 2023

• Resolve error in the test_neoxargs_usage unit test (#1074)

  mkerin

  authored
  d8028f8

  Copy full SHA for d8028f8

Commits on Nov 16, 2023

• Update neox_args.py (#1081)

  Show description for 10bf788
  

  jahatef

  and

  github-actions

  authored
  10bf788

  Copy full SHA for 10bf788

Commits on Nov 22, 2023

• Update README.md (#1082)

  Show description for f48d3a6
  

  StellaAthena

  and

  github-actions

  authored
  f48d3a6

  Copy full SHA for f48d3a6

Commits on Nov 30, 2023

• Update README.md

  StellaAthena

  authored
  efea81f

  Copy full SHA for efea81f

Commits on Dec 4, 2023

• Extend ci suite (#1080)

  Show description for 3be59a4
  mkerin

  authored
  3be59a4

  Copy full SHA for 3be59a4

• Patch coverity scan (#1090)

  Show description for a2b2020
  

  3 peopleauthored
  a2b2020

  Copy full SHA for a2b2020

Commits on Dec 6, 2023

• Corrects FLOPs formula as per 1093 (#1094)

  Show description for 050f560
  

  StellaAthena

  and

  github-actions

  authored
  050f560

  Copy full SHA for 050f560

Commits on Dec 19, 2023

• Update CODEOWNERS

  Show description for f19b2ec
  StellaAthena

  authored
  f19b2ec

  Copy full SHA for f19b2ec

Commits on Dec 20, 2023

• Bump transformers from 4.30.2 to 4.36.0 in /requirements (#1097)

  Show description for 07166da
  dependabot[bot]

  authored
  07166da

  Copy full SHA for 07166da

• Pins old DeeperSpeed until bug is fixed (#1095)

  Show description for 9283eff
  

  StellaAthena

  and

  github-actions

  authored
  9283eff

  Copy full SHA for 9283eff

Commits on Dec 22, 2023

• Update README.md

  StellaAthena

  authored
  9eef954

  Copy full SHA for 9eef954

• Update README.md

  StellaAthena

  authored
  a48e09e

  Copy full SHA for a48e09e

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  613e5a6

  Copy full SHA for 613e5a6

• Update README.md

  StellaAthena

  authored
  be7eeda

  Copy full SHA for be7eeda

• Update README.md

  StellaAthena

  authored
  2117afc

  Copy full SHA for 2117afc

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  8dba5b6

  Copy full SHA for 8dba5b6

• Add QK Normalization (#1100)

  Show description for f161245
  

  3 peopleauthored
  f161245

  Copy full SHA for f161245

• Update README.md

  StellaAthena

  authored
  7fb3b3c

  Copy full SHA for 7fb3b3c

• Update README.md

  StellaAthena

  authored
  a7509f0

  Copy full SHA for a7509f0

• Merge branch 'main' into StellaAthena-patch-4-1

  StellaAthena

  authored
  8eaac4e

  Copy full SHA for 8eaac4e

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  4d5a811

  Copy full SHA for 4d5a811

• Merge pull request #1099 from EleutherAI/StellaAthena-patch-4-1

  Show description for 05cc29c
  StellaAthena

  authored
  05cc29c

  Copy full SHA for 05cc29c

• Merge branch 'main' into StellaAthena-patch-4

  StellaAthena

  authored
  e25446e

  Copy full SHA for e25446e

• Merge pull request #1102 from EleutherAI/StellaAthena-patch-4

  Show description for 287f9f7
  StellaAthena

  authored
  287f9f7

  Copy full SHA for 287f9f7

Commits on Dec 23, 2023

• Lm eval 0.4.0 support (#1101)

  Show description for b27e409
  

  3 peopleauthored
  b27e409

  Copy full SHA for b27e409

• Update README.md

  StellaAthena

  authored
  1148a0f

  Copy full SHA for 1148a0f

Commits on Dec 26, 2023

• Update neox_args.py (#1107)

  Show description for e5a7ea7
  

  StellaAthena

  and

  github-actions

  authored
  e5a7ea7

  Copy full SHA for e5a7ea7

Commits on Jan 4, 2024

• Fix repo for CI (#1106)

  Show description for eca6b1a
  yang

  authored
  eca6b1a

  Copy full SHA for eca6b1a

• Fix install, Dockerfile, CI (#1104)

  Show description for 98716eb
  yang

  authored
  98716eb

  Copy full SHA for 98716eb

Commits on Jan 5, 2024

• Fused Rotary Embeddings (fixed) (#1108)

  Show description for 77605ca
  

  4 peopleauthored
  77605ca

  Copy full SHA for 77605ca

• Add pythia 14M and 31M configs (#1111)

  Show description for f14782a
  segyges

  authored
  f14782a

  Copy full SHA for f14782a

Commits on Jan 9, 2024

• Add docker compose and change containerized setup instructions to use it (#1113)

  Show description for e6e944a
  

  segyges

  and

  Quentin-Anthony

  authored
  e6e944a

  Copy full SHA for e6e944a

Commits on Jan 11, 2024

• Fix openwebtext2 downloader, backport improvements to DataDownloader (#1116)

  segyges

  authored
  92b1b6f

  Copy full SHA for 92b1b6f

Commits on Jan 13, 2024

• Bump jinja2 from 3.1.2 to 3.1.3 in /requirements (#1120)

  Show description for 90f70ff
  dependabot[bot]

  authored
  90f70ff

  Copy full SHA for 90f70ff

Commits on Jan 19, 2024

• Enable passing of --account to srun / SlurmLauncher (#1126)

  Show description for 6399155
  

  haileyschoelkopf

  and

  github-actions

  authored
  6399155

  Copy full SHA for 6399155

Commits on Jan 24, 2024

• update copyrights (#1128)

  Show description for 7a8fa2f
  

  jahatef

  and

  github-actions

  authored
  7a8fa2f

  Copy full SHA for 7a8fa2f

Commits on Jan 26, 2024

• fused layernorm (#1105)

  Show description for 3d8fec0
  

  yang

  and

  jahatef

  authored
  3d8fec0

  Copy full SHA for 3d8fec0

Commits on Jan 29, 2024

• Contributing Guide (#1138)

  Show description for e5602c3
  

  3 peopleauthored
  e5602c3

  Copy full SHA for e5602c3

Commits on Jan 30, 2024

• moved eval import and added to docs (#1139)

  R0n12

  authored
  1c133bf

  Copy full SHA for 1c133bf

Commits on Feb 1, 2024

• Update lm_eval v0.4 to PyPI dependencies (#1141)

  Show description for 032ec8c
  

  3 peopleauthored
  032ec8c

  Copy full SHA for 032ec8c

Commits on Feb 5, 2024

• Remove gas (beano) (#1144)

  Show description for 91c44bc
  segyges

  authored
  91c44bc

  Copy full SHA for 91c44bc

Commits on Feb 8, 2024

• Improve Conversion Utilities (#1124)

  Show description for f7373f8
  

  3 peopleauthored
  f7373f8

  Copy full SHA for f7373f8

Commits on Feb 21, 2024

• Fixes distributed tests, and skips tests that are broken. (#1149)

  Show description for 412cf6e
  

  3 peopleauthored
  412cf6e

  Copy full SHA for 412cf6e

• Memory profiling (#1153)

  Show description for 46d179c
  

  3 peopleauthored
  46d179c

  Copy full SHA for 46d179c

Commits on Feb 23, 2024

• add profiling to readme (#1154)

  Show description for eee03b2
  

  jahatef

  and

  github-actions

  authored
  eee03b2

  Copy full SHA for eee03b2

• Python version update (#1122)

  Show description for a7638a8
  segyges

  authored
  a7638a8

  Copy full SHA for a7638a8

• Minor changes (#1125)

  Show description for 72d1803
  segyges

  authored
  72d1803

  Copy full SHA for 72d1803

• Draft PR Adding mistral 0.1 (#1131)

  Show description for f36aed7
  

  4 peopleauthored
  f36aed7

  Copy full SHA for f36aed7

Commits on Feb 26, 2024

• [Bug?] Fix profiling argument names (#1155)

  Show description for 9663802
  

  3 peopleauthored
  9663802

  Copy full SHA for 9663802

Commits on Feb 29, 2024

• Update cpu_ci.yml (#1159)

  Show description for 3c03fc7
  

  jaimemcc-intel

  and

  github-actions

  authored
  3c03fc7

  Copy full SHA for 3c03fc7

Commits on Mar 2, 2024

• Improve argument validation for Flash-attn + SWA (#1162)

  Show description for 19596b0
  

  haileyschoelkopf

  and

  github-actions

  authored
  19596b0

  Copy full SHA for 19596b0

Commits on Mar 4, 2024

• Single node Pythia 14M training on ngc pytorch 24.02 container (#1170)

  Show description for 119950c
  

  tf-nv

  and

  Quentin-Anthony

  authored
  119950c

  Copy full SHA for 119950c

• Remove unnecessary fp32/bf16 conversion (#1169)

  Show description for 7b8187a
  

  DayOfThePenguin

  and

  Quentin-Anthony

  authored
  7b8187a

  Copy full SHA for 7b8187a

• Ignore markdown for pre-commit (#1171)

  Show description for 31cfe52
  

  Quentin-Anthony

  and

  github-actions

  authored
  31cfe52

  Copy full SHA for 31cfe52

• Make rotary freqs buffer non-persistent (#1168)

  Show description for e109bf5
  

  3 peopleauthored
  e109bf5

  Copy full SHA for e109bf5

• Support Lion with Zero Optimizer (#1166)

  Show description for df8cf24
  

  DayOfThePenguin

  and

  Quentin-Anthony

  authored
  df8cf24

  Copy full SHA for df8cf24

Commits on Mar 7, 2024

• Add MoE (#1129)

  Show description for 86758c3
  

  3 peopleauthored
  86758c3

  Copy full SHA for 86758c3

Commits on Mar 8, 2024

• remove best_download as dependency (#1179)

  Show description for 63b9fa1
  

  haileyschoelkopf

  and

  github-actions

  authored
  63b9fa1

  Copy full SHA for 63b9fa1

• Fix documentation for --jsonl-keys argument of preprocess_data scripts (#1176)

  KeitaW

  authored
  90d4cb3

  Copy full SHA for 90d4cb3

• clean up dockerfile: (#1175)

  Show description for 8c13642
  tf-nv

  authored
  8c13642

  Copy full SHA for 8c13642

• When using kv cache and flash attention in conjunction, it's crucial to set the causal parameter of flash_varlen_qkv_fn to False. Failing to do so will lead to inaccurate results. (#1178)

  chaochen99

  authored
  c1fa994

  Copy full SHA for c1fa994

• Remove gas from Pythia configs (#1181)

  Show description for 1e7abe7
  yang

  authored
  1e7abe7

  Copy full SHA for 1e7abe7

• Fix moe_loss in gpt_j_residual path (#1180)

  Show description for 82ddc66
  yang

  authored
  82ddc66

  Copy full SHA for 82ddc66

Commits on Mar 10, 2024

• Add Mamba Architecture (#1157)

  Show description for 6809bbc
  

  3 peopleauthored
  6809bbc

  Copy full SHA for 6809bbc

Commits on Mar 13, 2024

• Switch to using Cuda Flash Attn for Alibi (#1183)

  Show description for 03186de
  

  haileyschoelkopf

  and

  github-actions

  authored
  03186de

  Copy full SHA for 03186de

Commits on Mar 15, 2024

• Mamba + Tensor Parallel Support (#1184)

  Show description for 277141e
  

  3 peopleauthored
  277141e

  Copy full SHA for 277141e

Commits on Mar 19, 2024

• [ZeRO-3] Partitioned init with deepspeed.zero.Init() (#1190)

  Show description for 7267a74
  

  R0n12

  and

  Quentin-Anthony

  authored
  7267a74

  Copy full SHA for 7267a74

Commits on Mar 26, 2024

• Small typo in the README

  

  edouardoyallon

  committed
  e6b5261

  Copy full SHA for e6b5261

• Merge pull request #1196 from edouardoyallon/typo_readme

  Show description for 4085302
  StellaAthena

  authored
  4085302

  Copy full SHA for 4085302

• Added more papers

  StellaAthena

  authored
  1960b66

  Copy full SHA for 1960b66

• Update README.md

  StellaAthena

  authored
  3616658

  Copy full SHA for 3616658

Commits on Apr 1, 2024

• making PR triggered CPU test for changes to megatron (#1195)

  Show description for 977448e
  

  3 peopleauthored
  977448e

  Copy full SHA for 977448e

• [AMD] Supporting fused kernels build using JIT (#1188)

  Show description for 51a7de9
  R0n12

  authored
  51a7de9

  Copy full SHA for 51a7de9

• [ZeRO-3] Ensured passing neox deepspeed_config when using partitioned init (#1191)

  Show description for 01657aa
  

  R0n12

  and

  Quentin-Anthony

  authored
  01657aa

  Copy full SHA for 01657aa