fix: upsample the phase10 knowledge dataset

[RobotSail]

When we mix the knowledge dataset with skills today, we do not account for the potential discrepancy
in size between the generated knowledge data and skills data. This leads to the models potentially
forgetting the data it was trained on in the knowledge phase. As a simple workaround, we simply
upsample the knowledge samples before mixing them in with the generated skills dataset.

Signed-off-by: Oleg S [email protected]

[bbrowning]

Is upsampling a special case here? Or is it just that we need to adjust our mixing recipe in use for these knowledge leaf node(s) to have a fixed sampling size or a sampling ratio larger than the default of 1.0? See _sample_ds and _adjust_train_sample_size in datamixing.py for examples of what we already do today. And, you'll see that there's an optional fourth parameter to _gen_leaf_node_data that is the sampling size, which means we could pass in the desired fixed number of samples or ratio of samples when generating the knowledge leaf node data and then that would get written out to the recipe.yaml file as well as used for our final data mixing to scale up the knowledge samples.

[bbrowning]

This looks like a very reasonable solution to the upscaling problem in such a short time. I actually don't think it's quite as hacky as the code comments imply, but agree it's not an ideal solution to this general problem.

I'm running the full data generation pipeline against a sample taxonomy with a skill leaf node, a knowledge leaf node, and with a precomputed skills dataset getting mixed in via a customized default skills recipe (ie using https://github.com/instructlab/sdg/blob/main/docs/data_mixing.md#using-instructlab-community-pre-generated-dataset). However, I don't think this will finish on my available hardware before I head out for the night. If for some reason it errors out overnight because of these changes, I'll leave a note tomorrow.

Other than the one nit about replacing the stdout print with a logger, this looks ready to go. Since I'll be scarce tomorrow, going ahead and giving this one approval.

Thanks for the detailed PR, taking a couple of iterations on this to make it far less hacky than originally proposed, and the attention to detail with code comments and type hints!

[khaledsulayman]

This is a good approach, thanks for working to get this in!

[aakankshaduggal]

Thanks @RobotSail 🚢