Code

Support FSDP#149

Code

Merged

tocean merged 18 commits intomainAzure/MS-AMP:mainfrom yuxiang/fsdp_optAzure/MS-AMP:yuxiang/fsdp_optCopy head branch name to clipboardJan 16, 2024

+712-13

Commits

Commits on Jan 5, 2024

• integrate with fsdp

  tocean

  committed
  97ee2fa

  Copy full SHA for 97ee2fa

Commits on Jan 8, 2024

• support auto wrap policy

  tocean

  committed
  e6d4822

  Copy full SHA for e6d4822

• FSDPAdam extends from FSDPAdamW

  tocean

  committed
  eb800e8

  Copy full SHA for eb800e8

• add document

  tocean

  committed
  993bb7c

  Copy full SHA for 993bb7c

• add comment

  tocean

  committed
  515e6fc

  Copy full SHA for 515e6fc

• fix linting

  tocean

  committed
  b8da96b

  Copy full SHA for b8da96b

• fix typo

  tocean

  committed
  f005202

  Copy full SHA for f005202

Commits on Jan 9, 2024

• broadcast scaling_inv in optimizer.step

  tocean

  committed
  bcfd7e5

  Copy full SHA for bcfd7e5

• fix bug

  tocean

  committed
  103b7c7

  Copy full SHA for 103b7c7

• add ut for fsdp

  tocean

  committed
  19a5434

  Copy full SHA for 19a5434

• fix comments

  tocean

  committed
  4415d82

  Copy full SHA for 4415d82

• fix lint

  tocean

  committed
  e2d2f03

  Copy full SHA for e2d2f03

• fix lint

  tocean

  committed
  2379e32

  Copy full SHA for 2379e32

• fix bug in optimizer

  tocean

  committed
  6ffc5c1

  Copy full SHA for 6ffc5c1

Commits on Jan 10, 2024

• fix comments

  tocean

  committed
  5ea7679

  Copy full SHA for 5ea7679

• remove state type check in _fp8_post_backward_hook

  tocean

  committed
  8388adc

  Copy full SHA for 8388adc

• fix ut error

  tocean

  committed
  a8ff215

  Copy full SHA for a8ff215

Commits on Jan 11, 2024

• fix comments

  tocean

  committed
  d3acfa2

  Copy full SHA for d3acfa2