Hellaswag result not reproducing

[shash42]

Hi,

Thanks a lot for the codebase! It works quite smoothly and I was able to reproduce results across many datasets using the default sft_config provided, but not on hellaswag. I've attached my plot (weak_ft, strong_ft are floor and ceil respectively).

Specifically, I'm simply running python run.py --dataset="$arg_1" using the default Qwen-1.5-0.5b and Llama-3-8b models as the weak, strong pair. Are the configurations used to produce the main weak to strong plot different (specifically across datasets) than the ones provided by default in sft_config.py. If so, could you please provide detailed config files for reproduction, especially if there are any changes needed for Hellaswag?

Thanks!