Understanding large prediction intervals

[jesuinovieira]

Setup

• Estimator: pre-trained model
• Conformity score: residual normalized score
• Residual estimator: fitted by mapie (i.e., linear regression)

Description

In some cases, the prediction intervals produced can be very large or even infinite. Based on my experience, this issue often arises from insufficient data for calibration and fitting of the residual estimator – which may lead to a poorly fitted residual estimator, unrepresentative samples for calibration, or both.

Question

Given that there is sufficient data and the estimator is well-fitted to the training data, what other factors could cause unexpectedly large prediction intervals? For example, I guess that out-of-distribution data is one of the reasons. Are there other potential issues to be aware of that might indicate when the prediction intervals might not be reliable?

[Valentin-Laurent]

Indeed, out-of-distribution data can be the cause of large prediction intervals. The model used to estimate residuals may also be poorly fitted (overfitted/underfitted), resulting in overestimating or underestimating the residual variance. Maybe our conformal predictions experts @vincentblot28 or @thibaultcordier have other reasons in mind?

[vincentblot28]

Hi @jesuinovieira, I agree with the Valentin's answer. Maybe to better understand why you have such large prediction intervals you could do some sanity checks such as:

• When you have very large prediction sets, what is the value predicted by your residual estimator ?
• What is the quantile of your non-conformity scores ? (if it's very large, maybe you have some outlier non-conformity scores in your calibration set)
• To assess the global "adaptivness" of your prediction intervals, you can check the spearman correlation between the size of your intervals and the error of your model. If you have a significant correlation it means that your intervals adapt to the difficulty of the observation (even if some prediction intervals are very large)