Print per-token reward over an RM #9
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Will review later today! |
Comment on lines
+58
to
+95
| args = get_args() | ||
| quantized = True # only Starling isn't quantized for now | ||
| custom_dialogue = False | ||
| # some models need custom code to be run | ||
| if "oasst" in args.model or "oasst" in args.chat_template: | ||
| from herm.models import openassistant # noqa | ||
|
|
||
| model_builder = AutoModelForSequenceClassification.from_pretrained | ||
| pipeline_builder = pipeline | ||
| elif "Starling" in args.model or "Starling" in args.chat_template: | ||
| from herm.models.starling import StarlingPipeline, build_starling_rm | ||
|
|
||
| model_builder = build_starling_rm | ||
| pipeline_builder = StarlingPipeline | ||
| quantized = False | ||
| elif "openbmb" in args.model or "openbmb" in args.chat_template: | ||
| from herm.models.openbmb import LlamaRewardModel, OpenBMBPipeline | ||
|
|
||
| model_builder = LlamaRewardModel.from_pretrained | ||
| pipeline_builder = OpenBMBPipeline | ||
| elif "PairRM" in args.model or "PairRM" in args.chat_template: | ||
| from herm.models.pairrm import DebertaV2PairRM, PairRMPipeline | ||
|
|
||
| custom_dialogue = True | ||
| model_builder = DebertaV2PairRM.from_pretrained | ||
| pipeline_builder = PairRMPipeline | ||
| elif "SHP" in args.model or "SHP" in args.chat_template: | ||
| from herm.models.shp import SHPPipeline | ||
|
|
||
| custom_dialogue = True | ||
| model_builder = T5ForConditionalGeneration.from_pretrained | ||
| pipeline_builder = SHPPipeline | ||
| else: | ||
| model_builder = AutoModelForSequenceClassification.from_pretrained | ||
| pipeline_builder = pipeline | ||
|
|
||
| if custom_dialogue: | ||
| raise ValueError("Custom dialogue formatting not yet supported in this script") |
There was a problem hiding this comment.
In case we're going to reuse this code block in the future, we should factor this logic out (so that we can reuse it on run_rm.py), but imo for v1 it's fine for now 🤔
There was a problem hiding this comment.
Yeah I agree @ljvmiranda921 , and maybe add a test case.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Brief documentation in
analysis/README.mdWe can add visualizing it soon :)
E.g.
Reward: -0.544 | Substring: I
Reward: -0.556 | Substring: I love
Reward: -0.566 | Substring: I love to
Reward: 0.099 | Substring: I love to walk
Reward: 0.096 | Substring: I love to walk the
Reward: 0.092 | Substring: I love to walk the dog
Reward: 0.09 | Substring: I love to walk the dog,
Reward: 0.087 | Substring: I love to walk the dog, what
Reward: 0.085 | Substring: I love to walk the dog, what do
Reward: 0.089 | Substring: I love to walk the dog, what do you
Reward: 0.09 | Substring: I love to walk the dog, what do you like
Reward: 0.093 | Substring: I love to walk the dog, what do you like?