Code

re-merge from NVIDIA main#68

Code

Open

RaymondLi0 wants to merge 28 commits intomulti-query-attentionbigcode-project/Megatron-LM:multi-query-attentionfrom NVIDIA-main2bigcode-project/Megatron-LM:NVIDIA-main2Copy head branch name to clipboard

+846-317

Commits

Commits on May 19, 2023

• Add option to specify a data cache path separate from data directory.

  Show description for 8c86034
  jaredcasper

  committed
  8c86034

  Copy full SHA for 8c86034

Commits on May 23, 2023

• Check for write failure of index cache and print error message.

  jaredcasper

  committed
  ae37924

  Copy full SHA for ae37924

Commits on May 30, 2023

• Merge branch 'main' into data-cache

  jaredcasper

  committed
  41ba8a2

  Copy full SHA for 41ba8a2

Commits on May 31, 2023

• Add option to overlap p2p communication.

  jaredcasper

  committed
  f9283c5

  Copy full SHA for f9283c5

• typo

  jaredcasper

  committed
  621c9de

  Copy full SHA for 621c9de

• Consistent arg names.

  jaredcasper

  committed
  2c13d1f

  Copy full SHA for 2c13d1f

Commits on Jun 1, 2023

• Merge branch 'p2p_overlap' into 'main'

  Show description for e6d7e09
  jaredcasper

  committed
  e6d7e09

  Copy full SHA for e6d7e09

• Merge branch 'data-cache' into 'main'

  Show description for e4adfbc
  jaredcasper

  committed
  e4adfbc

  Copy full SHA for e4adfbc

Commits on Jun 2, 2023

• Supporting loading checkpoints without add_position_embedding arg.

  jaredcasper

  committed
  4d564cf

  Copy full SHA for 4d564cf

• Fix GPTDataset assert.

  jaredcasper

  committed
  1997e94

  Copy full SHA for 1997e94

Commits on Jun 5, 2023

• Merge branch 'gptdataset-assert' into 'main'

  Show description for f965380
  jaredcasper

  committed
  f965380

  Copy full SHA for f965380

• Fixed rotary_pos_emb's position in layer's forward args.

  

  lmcafee-nvidia

  authored and

  jaredcasper

  committed
  a6c574d

  Copy full SHA for a6c574d

• Merge branch 'lmcafee/rotary-kwarg-dev' into 'main'

  Show description for 382fd9d
  jaredcasper

  committed
  382fd9d

  Copy full SHA for 382fd9d

• fix indexation for output tensor after gradscaler call

  Show description for 41221b8
  aklife97

  committed
  41221b8

  Copy full SHA for 41221b8

• Merge branch 'outputtensor_index' into 'main'

  Show description for d2891b4
  jaredcasper

  committed
  d2891b4

  Copy full SHA for d2891b4

Commits on Jun 6, 2023

• Perform grad sync at correct place in interleaved pipeline parallelism

  

  timmoon10

  authored and

  jaredcasper

  committed
  ea76ecd

  Copy full SHA for ea76ecd

• Merge branch 'interleaved-pipeline-bugfix' into 'main'

  Show description for 992da75
  jaredcasper

  committed
  992da75

  Copy full SHA for 992da75

• Merge branch 'ckpt-load-fix' into 'main'

  Show description for f6c6d86
  jaredcasper

  committed
  f6c6d86

  Copy full SHA for f6c6d86

Commits on Jun 8, 2023

• Add workarounds for non-determinism in Megatron training

  jon-barker

  committed
  2880267

  Copy full SHA for 2880267

• Merge branch 'jbarker/non_determinism_fix' into 'main'

  Show description for db71a33
  jon-barker

  committed
  db71a33

  Copy full SHA for db71a33

Commits on Jun 10, 2023

• Update gitlab to catch pytest errors

  shanmugamr1992

  committed
  1af380d

  Copy full SHA for 1af380d

• Merge branch 'pytestError' into 'main'

  Show description for c7a0145
  shanmugamr1992

  committed
  c7a0145

  Copy full SHA for c7a0145

Commits on Jun 12, 2023

• Remove use of deprecated np.float in indexed_dataset.py

  

  jon-barker

  authored and

  jaredcasper

  committed
  bf5206e

  Copy full SHA for bf5206e

• Merge branch 'jbarker/np_float64_deprecation' into 'main'

  Show description for 000590e
  jaredcasper

  committed
  000590e

  Copy full SHA for 000590e

Commits on Jun 13, 2023

• Retro fix for tensor parallelism.

  

  lmcafee-nvidia

  authored and

  jaredcasper

  committed
  f479999

  Copy full SHA for f479999

• Merge branch 'lmcafee/retro-dataloader-fix' into 'main'

  Show description for 0604155
  jaredcasper

  committed
  0604155

  Copy full SHA for 0604155

Commits on Jun 23, 2023

• Merge branch 'main' of github.com:NVIDIA/Megatron-LM into NVIDIA-main2

  RaymondLi0

  committed
  e5a9d7b

  Copy full SHA for e5a9d7b

Commits on Jun 27, 2023

• pass on data_cache_pass in build_dataset_group

  

  Raymond Li

  committed
  8b6ceeb

  Copy full SHA for 8b6ceeb