why finetuning get worse result than zero-shot

[gyangfan]

finetuning own my dataset，but get worse result than zero-shot. Is this reasonable?

[jackshiels]

Are you perhaps overfitting your data? What does your training set look like?

[gyangfan]

my data is temperature records with 1h freq，finetuning lr le-4，there should be no overfitting because predict with checkpoint saved at epoch1

[jackshiels]

Could you elaborate more on the state of your experiment?

What is your config like?

Are you cleaning and preparing your data correctly?

Is the data actually time series data (could be more stochastic)?

[FrankHo-Hwc]

I met the same phenomenon when fine-tuning with my own dataset. One of the reasons I guess maybe the size of the fine-tuning dataset you use.

[ikvision]

Is your training loss going down? Do you monitor your validation loss

[tomar-s]

I am seeing the same behavior while finetuning on zero shot datasets(ETTh and Ercot) keeping the config files same but only changing the prediction length according to the given dataset. I use the command :

Fine-tune amazon/chronos-t5-mini for 1000 steps with initial learning rate of 1e-3
CUDA_VISIBLE_DEVICES=0 python training/train.py --config /path/to/modified/config.yaml
--model-id amazon/chronos-t5-mini
--no-random-init
--max-steps 1000
--learning-rate 0.001

And then run the evaluate.py loading the model saved from the above fine-tuning:
pipeline = ChronosPipeline.from_pretrained("/path/to/fine-tuned/model/ckpt/dir/")

Below are the MASE for each dataset:

• ETTh : 0.900
• Ercot : 0.630
  Both the results are worse than zero-shot performance in the results table in Chronos paper. This is just a single run and I am not averaging across multiple runs yet.

[ikvision]

Without a plot of the validation loss over iterations it is very difficult do understand if the fine tune is done with the correct hyper-parameters