Add rotation matrix to equalized layers #1086
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Related to Xilinx#1073 Add `RotationEqualizedLayer` class to `equalized_layer.py` to hold two learnable matrices for rotation. * Implement `forward` method in `RotationEqualizedLayer` to apply rotation and call the wrapped layer. * Add method to fuse rotation matrices into the wrapped layer. * Add unit tests for `RotationEqualizedLayer` class in `test_equalized_layer.py`. * Test forward pass with rotation matrices. * Test fusing rotation matrices into the wrapped layer.
|
Hi @vishwamartur, I took a look at this PR and unfortunately I might have to close it, since it is very far away from the solution I had envisioned for this problem, and I am not even sure this solution would allow us to reach our goal, which is to replicate the setup used in some recent works like SpinQuant. As mentioned in the other PR, it is always recommended to use the issues to discuss what solution you would like to propose. In the next days I will update the relevant issue with a broad description of the solution I had planned for this problem. Apologies for the miss-understanding and if you're still willing to work on this, stay tuned for the updates on the issue! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related to #1073
Add
RotationEqualizedLayerclass toequalized_layer.pyto hold two learnable matrices for rotation.forwardmethod inRotationEqualizedLayerto apply rotation and call the wrapped layer.RotationEqualizedLayerclass intest_equalized_layer.py.