Keyword arguments for liger functional #368
Assignees
Comments
|
CE is fixed by #375. But need to add functional wrapper for other modules |
|
I can take this one 😄 |
This was referenced Nov 20, 2024
ByronHsu
added a commit
that referenced
this issue
Nov 21, 2024
## Summary <!--- This is a required section; please describe the main purpose of this proposed code change. ---> This PR enables the keyword arguments of liger functional #368. <!--- ## Details This is an optional section; is there anything specific that reviewers should be aware of? ---> 1. Warp the Liger Operator Functions (`torch.autograd.Function`) with an extra layer that can take key word arguments. 2. For each of the liger functions, updating its unit test function `test_{operator_name}.py::test_correctness_functional` to reflect that keyword args can be accepted. ## Testing Done <!--- This is a required section; please describe how this change was tested. ---> <!-- Replace BLANK with your device type. For example, A100-80G-PCIe Complete the following tasks before sending your PR, and replace `[ ]` with `[x]` to indicate you have done them. --> - Hardware Type: A10G - [x] run `make test` to ensure correctness - [x] run `make checkstyle` to ensure code style - [x] run `make test-convergence` to ensure convergence --------- Signed-off-by: Hongpeng Guo <[email protected]> Co-authored-by: Byron Hsu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
🚀 The feature, motivation and pitch
Liger's functional doesn't support keyword arguments because it's implemented by replacing
torchautograd.Function.apply.kwargs support is necessary for LigerCrossEntropyLoss to patch transformers model after GA fix
https://github.com/huggingface/transformers/blob/a06a0d12636756352494b99b5b264ac9955bc735/src/transformers/models/llama/modeling_llama.py#L1224
Alternatives
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: