Look at performance by month

[jsadler2]

I am curious to see how the models compare on month-wise basis.

It should be pretty straightforward to look at this.

[jsadler2]

[jsadler2]

One question that came up in our call yesterday was whether the really good replicate of the 2_multitask_dense model in the winter was the same as the really bad one in the summer:

I plotted just the 2_multitask_dense RMSE's and colored those by rep_id

So, as you can see, the really good winter model is not the same as the really bad summer model for do_min.

One thing that did pop out to me, though, was that that green replicate (rep_id 2) is clearly the most accurate at predicting do_min, but it is nearly the worst at predicting do_max.

Here's another look at that (averaged across the months). It doesn't seem like that's always true though (i.e., the worst replicates for predicting do_max are not always the best at predicting do_min).