Update yaml configs. Add validation dataset #2608
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/2608
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 348793b with merge base 0991f97 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
|
|
||
| # we need to manually build the mistral classifier model | ||
| # because our reward model checkpoint has a larger vocabulary size (due to an added padding token) | ||
| reward_and_value_model: |
There was a problem hiding this comment.
this looks like an accidental change and probably could be worth putting back for the PPO recipe to work
| dataset_val: | ||
| _component_: torchtune.datasets.alpaca_cleaned_dataset | ||
| split: train[95%:] | ||
| batch_size_val: 8 |
There was a problem hiding this comment.
For many of the larger models this batch size may be too high and result in OOM.
|
@krammnic thank you for opening this PR. One last change: could you actually set I'm happy to merge after this. |
Hey, sure! |
|
@SalmanMohammadi This is really weird |
|
I think you also need to remove the fields from configs for recipes which we don't support validation datasets for: QAT, and LoRA and FFT single device. |
|
@SalmanMohammadi let's start CI |
|
@SalmanMohammadi oops, one more time (I forgot that our lint triggers on configs)/ |
| @@ -115,4 +115,3 @@ profiler: | |||
| wait_steps: 5 | |||
| warmup_steps: 3 | |||
| active_steps: 2 | |||
| num_cycles: 1 | |||
There was a problem hiding this comment.
Why was this changed?
| dataset: | ||
| _component_: torchtune.datasets.alpaca_dataset |
There was a problem hiding this comment.
This is a single device recipe, it doesn't support validation datasets
| seed: null | ||
| shuffle: True | ||
|
|
||
| # Validation |
There was a problem hiding this comment.
Also a single device recipe
|
@krammnic could you please take another pass through and make sure that no single-device recipes have been modified? |
Context
What is the purpose of this PR? Is it to
Please link to any issues this PR addresses.
Changelog
What are the changes made in this PR?
Test plan
Please make sure to do each of the following if applicable to your PR. If you're unsure about any one of these just ask and we will happily help. We also have a contributing page for some guidance on contributing.
pre-commit install)pytest testspytest tests -m integration_testUX
If your function changed a public API, please add a dummy example of what the user experience will look like when calling it.
Here is a docstring example
and a tutorial example