Default num_samples may be too small

[tanz63]

I've recently read a paper that benchmark with Chronos for zero-shot forecasting(https://arxiv.org/abs/2402.02368), it comes out with inferior performance for Chronos. However, based on the results of internal test we conducted, I think Chronos' performance is over underestimated in that paper(even with the dataset they benchmarked on). Maybe the main reason is that the default num_sample is set to be too low (num_samples=20 based on the example usage in readme file). A value larger than 30 (general empirical rule from Central Limit Theorem) might be much better. In our case, we set it to be 100 and Chronos came out to be the leading time series foundation model(TSFM) with a significant margin compared with other published TSFMs, which is no longer significant at all when num_samples=20. [Maybe also with the low topK parameter]
I think Chronos is highly competitve, but it's academic and industrial recognition might be over underestimated with this num_samples. (For users who just want to give a simple trial based the default parameters)

[lostella]

@tanz63 interesting observations, thanks for sharing this.

In our case, we set it to be 100 and Chronos came out to be the leading time series foundation model(TSFM) with a significant margin compared with other published TSFMs, which is no longer significant at all when num_samples=20.

I'm wondering, did you observe this also with smaller-sized chronos models? Because increasing num_samples too much incurs in slower inference obviously, which may be OK with smaller models but not so much with larger ones.

[Maybe also with the low topK parameter]

Did you try only increasing top_k, and leaving num_samples to its default? The reason I'm asking is, increasing top_k comes at essentially no cost in terms of inference speed.

[tanz63]

I don't really remember whether we've conducted systematic ablation study to analysis the effect, but the main finding is that we got much better result when using larger num_samples and top_k on T5-base. Since we don't test much with large volume data like Hourly,Minute, but mainly with Daily,weekly,monthly dataset, the inference speed is fast enough even under these configurations.
I'd be happy to see that upcoming papers with a default benchmark parameter setting that fully release Chronos's forecasting power.
By the way, how does it come out to be num_samples=20 as default value?(just for curiosity because I would guess it to be 30 or larger , perhaps because of a common statistical guideline)

[lostella]

I think we just went for a conservative value for num_samples, to guard against too-long prediction time in case one uses default settings. There's a tradeoff there, obviously.