Code

[WIP] Sequence packing#65

Code

Merged

pclucas14 merged 41 commits intomainmicrosoft/mttl:mainfrom sequence_packingmicrosoft/mttl:sequence_packingCopy head branch name to clipboardAug 2, 2024

+724-44

Commits

Commits on Jul 18, 2024

• sequence packing

  pclucas14

  committed
  3e2e883

  Copy full SHA for 3e2e883

• Merge branch 'main' of github.com:microsoft/mttl into sequence_packing

  pclucas14

  committed
  4b4c739

  Copy full SHA for 4b4c739

• cmd line support

  pclucas14

  committed
  859fe79

  Copy full SHA for 859fe79

Commits on Jul 24, 2024

• Fixed import and circular dependency

  matheper

  authored
  e319947

  Copy full SHA for e319947

• black isort

  matheper

  authored
  55aa9f6

  Copy full SHA for 55aa9f6

• isort

  matheper

  committed
  70f5c09

  Copy full SHA for 70f5c09

Commits on Jul 25, 2024

• make pack_to_multiple_of a controllable arg

  pclucas14

  committed
  40baf16

  Copy full SHA for 40baf16

• Merge branch 'info-container' of github.com:microsoft/mttl into sequence_packing

  pclucas14

  committed
  0bbd41f

  Copy full SHA for 0bbd41f

• Merge branch 'fix_dependencies' of github.com:microsoft/mttl into sequence_packing

  pclucas14

  committed
  6f7d374

  Copy full SHA for 6f7d374

• flash attention with sequence packing

  pclucas14

  committed
  c96ac64

  Copy full SHA for c96ac64

• make padding side an argument controllable from command line. Flash Attention with Mistral must be left padded

  pclucas14

  committed
  be9b97c

  Copy full SHA for be9b97c

Commits on Jul 26, 2024

• revert to shuffle train ds

  pclucas14

  committed
  5771715

  Copy full SHA for 5771715

• Merge remote-tracking branch 'origin/main' into sequence_packing

  

  Alessandro Sordoni

  committed
  5fa7e3e

  Copy full SHA for 5fa7e3e

• Merge remote-tracking branch 'origin/main' into sequence_packing

  

  Alessandro Sordoni

  committed
  204edc5

  Copy full SHA for 204edc5

Commits on Jul 30, 2024

• rename use_flash_attn to attn_implementation, and update sequence packing code

  pclucas14

  committed
  a2135ca

  Copy full SHA for a2135ca

• renamed attention patch

  pclucas14

  committed
  0ae5c27

  Copy full SHA for 0ae5c27

• max sequences per pack as a cmd line arg

  pclucas14

  committed
  4249127

  Copy full SHA for 4249127

• cleanup wrapper

  pclucas14

  committed
  a9d4209

  Copy full SHA for a9d4209

• cleanup wrapper

  pclucas14

  committed
  f134727

  Copy full SHA for f134727

• enable eval during epoch

  pclucas14

  committed
  75ff5be

  Copy full SHA for 75ff5be

• Merge branch 'main' of github.com:microsoft/mttl into sequence_packing

  pclucas14

  committed
  802a64e

  Copy full SHA for 802a64e

• Merge branch 'sequence_packing' of github.com:microsoft/mttl into sequence_packing

  pclucas14

  committed
  5b26e21

  Copy full SHA for 5b26e21

Commits on Jul 31, 2024

• packing tests and warnings

  pclucas14

  committed
  a79b835

  Copy full SHA for a79b835

• flash attention wrapper

  pclucas14

  committed
  a64abb7

  Copy full SHA for a64abb7

• trying to bump up versions for flash-attn

  pclucas14

  committed
  9723a96

  Copy full SHA for 9723a96

• max sequences per pack as a cmd line arg

  pclucas14

  committed
  d93b27a

  Copy full SHA for d93b27a

• Upgrade pip and use python 3.11 for codeql

  matheper

  committed
  d9ce19a

  Copy full SHA for d9ce19a

• cleanup

  pclucas14

  committed
  b722437

  Copy full SHA for b722437

• remove flash attn from the requirements

  pclucas14

  committed
  615c569

  Copy full SHA for 615c569

• Merge branch 'sequence_packing' of github.com:microsoft/mttl into sequence_packing

  pclucas14

  committed
  04d08cd

  Copy full SHA for 04d08cd

• flash-attn optional-dependencies

  matheper

  committed
  de38d53

  Copy full SHA for de38d53

Commits on Aug 1, 2024

• trust remote code

  

  Alessandro Sordoni

  committed
  53c2243

  Copy full SHA for 53c2243

• attn_implementation in SimpleConfig?

  

  Alessandro Sordoni

  committed
  2193320

  Copy full SHA for 2193320

• fix tests for coalesced

  pclucas14

  committed
  6b2de1b

  Copy full SHA for 6b2de1b

• fix tests

  pclucas14

  committed
  b0afa1e

  Copy full SHA for b0afa1e

• requirements upgrade

  pclucas14

  committed
  4db90d7

  Copy full SHA for 4db90d7

• hmm

  pclucas14

  committed
  384597a

  Copy full SHA for 384597a

Commits on Aug 2, 2024

• fix test

  pclucas14

  committed
  71957a6

  Copy full SHA for 71957a6

• run test

  pclucas14

  committed
  3fc2d8f

  Copy full SHA for 3fc2d8f

• run test

  pclucas14

  committed
  abf17c7

  Copy full SHA for abf17c7

• remove flash attn from the requirements

  pclucas14

  committed
  5c918e6

  Copy full SHA for 5c918e6