Code

Flash attention experiments#18

Code

Draft

jlamypoirier wants to merge 9 commits intomainbigcode-project/transformers:mainfrom flash_attentionbigcode-project/transformers:flash_attentionCopy head branch name to clipboard

+584-194

Commits

Commits on Apr 13, 2023

• Add back experimental features

  jlamypoirier

  committed
  0b4585c

  Copy full SHA for 0b4585c

• Fixes

  jlamypoirier

  committed
  4187786

  Copy full SHA for 4187786

Commits on Apr 14, 2023

• Error message

  jlamypoirier

  committed
  10f4a98

  Copy full SHA for 10f4a98

Commits on Apr 20, 2023

• Attn implementations

  jlamypoirier

  committed
  d08ce17

  Copy full SHA for d08ce17

Commits on Apr 21, 2023

• Optional jit

  jlamypoirier

  committed
  23eacc7

  Copy full SHA for 23eacc7

• No flash for graphs

  jlamypoirier

  committed
  5dd048f

  Copy full SHA for 5dd048f

Commits on Apr 22, 2023

• Unpad for flash attn, remove flash/torch for decode, misc improvements

  jlamypoirier

  committed
  a6efba9

  Copy full SHA for a6efba9

Commits on Apr 24, 2023

• cleanup

  jlamypoirier

  committed
  a2efad2

  Copy full SHA for a2efad2

• Merge branch 'main' into flash_attention

  jlamypoirier

  committed
  5aa0276

  Copy full SHA for 5aa0276