Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PyTorch] Add support for cuDNN FusedAttention + THD + CP #885

Merged
merged 30 commits into from
Jun 10, 2024
Merged

[PyTorch] Add support for cuDNN FusedAttention + THD + CP #885

merged 30 commits into from
Jun 10, 2024

Commits on May 31, 2024

  1. add seq_offsets_qkvo for cudnn thd 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed May 31, 2024
    Configuration menu
    Copy the full SHA
    d9ec133 View commit details
    Browse the repository at this point in the history

  2. add seq_offsets_qkvo to AttnFuncWithCP 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed May 31, 2024
    Configuration menu
    Copy the full SHA
    9211a7c View commit details
    Browse the repository at this point in the history

  3. fix seq_offsets calculation of cudnn thd 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed May 31, 2024
    Configuration menu
    Copy the full SHA
    541268b View commit details
    Browse the repository at this point in the history

  4. remove a thd assert 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed May 31, 2024
    Configuration menu
    Copy the full SHA
    eb3f66d View commit details
    Browse the repository at this point in the history

  5. fix bias for thd test 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed May 31, 2024
    Configuration menu
    Copy the full SHA
    9ac207a View commit details
    Browse the repository at this point in the history

  6. add thd test for cudnn FA with CP 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed May 31, 2024
    Configuration menu
    Copy the full SHA
    4239df9 View commit details
    Browse the repository at this point in the history

  7. skip GQA/MQA test for cuDNN THD 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed May 31, 2024
    Configuration menu
    Copy the full SHA
    9b309ab View commit details
    Browse the repository at this point in the history

  8. make sure seq_offsets are computed with qkv_group of hd_hd_hd while CP>1 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed May 31, 2024
    Configuration menu
    Copy the full SHA
    4891fac View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2024

  1. fix seq_offsets inputs 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    e886136 View commit details
    Browse the repository at this point in the history

  2. remove two comments 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    6ae7384 View commit details
    Browse the repository at this point in the history

  3. fix attn mask type for cudnn thd with cp 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    4a38798 View commit details
    Browse the repository at this point in the history

  4. fix attn_mask_type check 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    a8807e6 View commit details
    Browse the repository at this point in the history

  5. fix attn_mask_type for cudnn fa with thd 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    5a6a0a5 View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2024

  1. fix a typo 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    d657950 View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2024

  1. fix out dout in bwd 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    3b8780c View commit details
    Browse the repository at this point in the history

  2. assert cudnn+thd does not support attn bias 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    b3997ce View commit details
    Browse the repository at this point in the history

  3. check if attn_mask_type has padding 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    e7b9ea7 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into xren/cp_thd

    @xrennvidia
    xrennvidia committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    df2e164 View commit details
    Browse the repository at this point in the history

  5. minor change 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    12cd072 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. change cp test batch size to 2 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    cc3eb4f View commit details
    Browse the repository at this point in the history

  2. fix code format 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    442f347 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. fix two assert info 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    70042d2 View commit details
    Browse the repository at this point in the history

  2. fix assert comment 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    d37b399 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into xren/cp_thd

    @cyanguwa
    cyanguwa authored Jun 5, 2024
    Configuration menu
    Copy the full SHA
    5302419 View commit details
    Browse the repository at this point in the history

  4. fix assert comments 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    a6c620d View commit details
    Browse the repository at this point in the history

  5. Merge branch 'xren/cp_thd' of github.com:xrennvidia/TransformerEngine… 

    … into xren/cp_thd
    @xrennvidia
    xrennvidia committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    a98abd4 View commit details
    Browse the repository at this point in the history

  6. minor fix 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    dfe2f03 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2024

  1. fix assert comments 

    Signed-off-by: Xiaowei Ren <[email protected]>
    @xrennvidia
    xrennvidia committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    a8cebd7 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into xren/cp_thd

    @cyanguwa
    cyanguwa authored Jun 6, 2024
    Configuration menu
    Copy the full SHA
    f22bc19 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2024

  1. Merge branch 'main' into xren/cp_thd

    @cyanguwa
    cyanguwa authored Jun 7, 2024
    Configuration menu
    Copy the full SHA
    b13e3aa View commit details
    Browse the repository at this point in the history