Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Draft] Split qkv_proj into q_proj, k_proj and v_proj #90

Closed
wants to merge 6 commits into from
Closed

[Draft] Split qkv_proj into q_proj, k_proj and v_proj #90

wants to merge 6 commits into from

Conversation

pclucas14
Copy link
Contributor

  1. For models that stack the query, key, value projections in the attention layer, this PR splits the qkv_proj into q_proj, k_proj, and v_proj to allow for individual patching of the layers with LoRA.

  2. This remove the need for phi_2_align_heads and replaces it with split_qkv in the LoRAConfig class.

@pclucas14 pclucas14 changed the title Split qkv_proj into q_proj, k_proj and v_proj [Draft] Split qkv_proj into q_proj, k_proj and v_proj Aug 19, 2024
@pclucas14
Copy link
Contributor Author

closing for now

@pclucas14 pclucas14 closed this Sep 3, 2024
@sordonia sordonia deleted the split_qkv branch February 13, 2025 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@pclucas14