The size of num_samples greatly affects the inference speed of the model

[sudongwang-upc]

Thank you for this job!
I see that during model inference, the context of encoder inputs will be copied by num_samples and then followed by the decoder steps. In this case, the number of num_samples will significantly affect the maximum speed of inference (under given GPU conditions), which is important in practical application scenarios.
Is there a better way to improve inference speed without significantly reducing model performance and robustness?

[abdulfatir]

You're right. More samples will typically improve performance but will require more GPU memory/inference time. As this is autoregressive generation, I am afraid I am not aware of a way to draw more samples without incurring this cost.