Initializing server with an adapter sets it as the default #370
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, initializing the LoRAX server with an adapter would merge its weights into the base model and prevent dynamic adapter loading. In practice, this runs counter to the value prop of LoRAX, which is to support many adapters at runtime, and is incompatible with planned extensions like speculative decoding adapters.
This PR changes the behavior so that specifying
--adapter-idduring init will set that as the default adapter used when no adapter ID is provided for the request. This allows for setting fallback adapters that can be generally useful for most requests when the user does not know which adapter they should use at runtime, and in cases where using something like a strong baseline speculative decoding adapter is a reasonable default for most queries.