Update dependency peft to v0.14.0 #793
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![renovate[bot]](https://avatars.githubusercontent.com/in/2740?s=60&v=4){kind=small}
![renovate-approve[bot]](https://avatars.githubusercontent.com/in/7394?s=60&v=4){kind=small}
![renovate-approve-2[bot]](https://avatars.githubusercontent.com/in/62175?s=60&v=4){kind=small}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains the following updates:
==0.13.2->==0.14.0Release Notes
huggingface/peft (peft)
v0.14.0: Version 0.14.0: EVA, Context-aware Prompt Tuning, Bone, and moreCompare Source
Highlights
New Methods
Context-aware Prompt Tuning
@tsachiblau added a new soft prompt method called Context-aware Prompt Tuning (CPT) which is a combination of In-Context Learning and Prompt Tuning in the sense that, for each training sample, it builds a learnable context from training examples in addition to the single training sample. Allows for sample- and parameter-efficient few-shot classification and addresses recency-bias.
Explained Variance Adaption
@sirluk contributed a new LoRA initialization method called Explained Variance Adaption (EVA). Instead of randomly initializing LoRA weights, this method uses SVD of the base layer weights to initialize the LoRA weights and is also able to re-allocate the ranks of the adapter based on the explained variance ratio (derived from SVD). Thus, this initialization method can yield better initial values and better rank distribution.
Bone
@JL-er added an implementation for Block Affine (Bone) Adaption which utilizes presumed sparsity in the base layer weights to divide them into multiple sub-spaces that share a single low-rank matrix for updates. Compared to LoRA, Bone has the potential to significantly reduce memory usage and achieve faster computation.
Enhancements
PEFT now supports LoRAs for
int8torchao quantized models (check this and this notebook) . In addition, VeRA can now be used with 4 and 8 bit bitsandbytes quantization thanks to @ZiadHelal.Hot-swapping of LoRA adapters is now possible using the
hotswap_adapterfunction. Now you are able to load one LoRA and replace its weights in-place with the LoRA weights of another adapter which, in general, should be faster than deleting one adapter and loading the other adapter in its place. The feature is built so that no re-compilation of the model is necessary iftorch.compilewas called on the model (right now, this requires ranks and alphas to be the same for the adapters).LoRA and IA³ now support
Conv3dlayers thanks to @jsilter, and @JINO-ROHIT added a notebook showcasing PEFT model evaluation using lm-eval-harness toolkit.With the
target_modulesargument, you can specify which layers to target with the adapter (e.g. LoRA). Now you can also specify which modules not to target by using theexclude_modulesparameter (thanks @JINO-ROHIT).Changes
DynamicCachecaching infrastructure of transformers (see #2096). If you are using this PEFT version and a recent version of transformers with an old prefix tuning checkpoint, you should double check that it still works correctly and retrain it if it doesn't.lora_biasparameter to LoRA layers to enable bias on LoRA B matrix. This is useful when extracting LoRA weights from fully fine-tuned parameters with bias vectors so that these can be taken into account.from_pretrainednow warns the user if PEFT keys are missing.modules_to_saveis now properly and transparently handled.What's Changed
SFTConfiginstead ofSFTTrainerkeyword args by @qgallouedec in https://github.com/huggingface/peft/pull/2150evalandno dropoutby @ariG23498 in https://github.com/huggingface/peft/pull/2122rank_patternandalpha_patterntogether inLoraConfigby @sirluk in https://github.com/huggingface/peft/pull/2195metadevice check bug + add multi-gpu functionality by @sirluk in https://github.com/huggingface/peft/pull/2218Nonecheck forloftq_configattribute inLoraConfigby @sirluk in https://github.com/huggingface/peft/pull/2215task_typein PEFT Configurations by @d-kleine in https://github.com/huggingface/peft/pull/2210New Contributors
Full Changelog: huggingface/peft@v0.13.2...v0.14.0
Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Enabled.
♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR was generated by Mend Renovate. View the repository job log.