[BUG] Scaled CRPS is possibly incorrect

[elephaint]

Our implementation of scaled crps scales each timeseries according to the absolute value of the observed values of each timeseries, but the original GluonTS implementation (and the one we also follow in hierarchicalforecast) scales the CRPS based on the norm calculated across all timeseries.

Is this intentional?

To make things concrete, the following shows how the code for a Pandas DF should be adapted to reflect the GluonTS / HF behavior.

loss = loss.set_index(id_col)
sizes = sizes.set_index(id_col)
assert isinstance(df, pd.DataFrame)
- norm = df[target_col].abs().groupby(df[id_col], observed=True).sum()
+ norm = df[target_col].abs().sum()
- res = 2 * loss.mul(sizes['counts'], axis=0).div(norm + eps, axis=0)
+ res = 2 * loss.mul(sizes['counts'].sum(), axis=0).div(norm + eps, axis=0)        
res.index.name = id_col
res = res.reset_index()

[jmoralez]

I think the whole point of having a scale is for the metric to be comparable across series, similar to MASE. If you use a global scale you're just dividing everything by a constant and the metric is dominated by series with large values.