Incremental Fine Tuning in Ludwig

[msmmpts]

Hi all,

I was wondering if we could fine tune an LLM with 1K samples and saving the model and later add additional 500 samples while preserving the knowledge from previous 1K samples in Ludwig during the next fine tuning phase

Is there any way to do that?

Thanks in advance

[w4nderlust]

This is a pretty challenging thing, and despite lots of work on avoiding catastrophic forgetting i nthe literature, there doesn't seem to be a silver bullet for this.

That said, the most convincing mechanism (in terms of efficacy and simplicity) that I've seen so far is what people do in RLHF for instance: adding an additional piece to the loss that penalizes the model from producing distributions with a high KL Divergence from the distribution of the "original" (as in previous) model.
We dont' have such a mechanism in Ludwig today, but I've been thinking about it for a while, and one could achieve something similar by means of a loss that tries to keep the logits close to the logits of the original model on the new datapoints and potentially on old ones too. Something like a l2 distance in the last layer embedding space.
Would you be interested in exploring working together on this @msmmpts and contributing it to Ludwig? Obviously no guarantee it will work for your specific use case, but it's the best idea I have so far 😊