Suggestions for improvements for the estimation model.

[pmensalt]

The estimation model that is being used at the moment has some limitations. In this discussion, I want to give my view on how to improve it. This discussion is extracted from this issue: #4323

First, let me start with some of the limitations and real world examples:

• Estimated value of nuclear jumps and is overestimated by a lot. Energy production of nuclear cannot make such large jumps, it should normally only be increase or decrease gradually.
  
• Estimated value of wind energy drops by a lot. Same thing applies here: normally, it increases/decreases gradually.
  (Also, look at the increase of carbon intensity. The previous value was 63g, the estimated next value is 195g (that's more than tripple the previous value).

This is caused by the simplicity of the estimation model: It looks like it takes the previous values of previous days into account and calculates the average. However, for some energy sources, the values of the previous days are not related to the current day and should not be used.

I identified several types of energy sources:

1. The curve of a day is related to the curve of the previous day(s), eg. solar energy.
2. The value is related to the previous value (and not related to the previous days), eg. wind/nuclear/hydro,...
3. Depending on the surplus of energy, eg. batteries and hydro storage. They are mostly related to renewable energy sources, like wind and solar energy and/or the price of energy. (When energy prices are low, there is a greater change that energy will be put into the batteries).
4. Other energy sources that are used to complete the total need of energy, on top of the above sources, eg coal, gas, ... This is also a curve which is related to the previous day(s).

To improve the estimation model, I would like to suggest the following improvements (each point refferres to the same point above):

1. Take the value of the same timeslot of the previous days into account. Instead of calculating the average value, calculate the average value increase or decrease in percent.
   The rationale is the following: The previous days can be very sunny, whereas the current day can be cloudy. An average value of the previous days would therefor be a massive overestimation. However, if we take the average value increase in percent into account, the value will be much closer to the real value: eg: the light intensity at 11:00 will normally always be higher than at 10:00, on sunny and on cloudy days. As the value on the cloudy day at 10:00 will be low, the estimated value at 11:00 will also be low, but higher than at 10:00.
   For an (imaginary) example, see Estimate for nuclear in Belgium looks odd #4323 (comment).
   The calculated value should not be higher than than the max capacity of the energy source!
2. For those kind of energy sources, just take the same value as the previous one. We cannot estimate a better value, because those values are not related to historical data. Without other input (like weather forecast), the same value is the best estimation we can make.
3. I'm not completely sure how this type can be best dealt with. As I already mentioned, we should take the estimated values of renewable energy sources (like wind and solar) into account, which are already calculate in 1. When high amounts of energy is produced by those sources, there is a higher change that batteries will be used to store the energy. However, we should also take into account the state of the batteries: Use the previous values to see if there is still capacity left to be filled. Also take into account the energy price of the current timeslot. To get this all right, this will most likely be challenging (but not impossible). For a first version, this can be simplified and improved later on.
4. To calculate the remaining sources, another strategy is needed. For those sources, I would suggest to calculate the total energy needed, subtract the values calculated above, and distribute the remaining amount over those types. In more details, we should:
   1. Calculate the total amount needed. To do this, again, calculate the average value increase or decrease in percent of the same timeslot of the previous days to get an indication of the energy needed. However, on weekend days, less energy is typically needed than on week days. To improve the calculation, not only the previous days should be taken into account, but also the values of the same day of the previous weeks. To further improve this, I would take a weighted average, where the same day of the previous weeks should have a higher weight than the values of the previous day.
   2. Subtract the value values calculated in 1., 2. and 3. from the value calculated in 4.i. This is the total remaining amount of energy needed.
   3. Distribute the total remaining amount of energy needed of the remaining energy sources. To do this distribution, there might be several ways, but I propose the following method: For each of the remaining energy types, calculated the average value of the previous couple of hours. Now calculate the average distribution value from those values. Eg: coal: 36%, gas: 58%, biomass: 6%. Use those percentages to distribute the value calculated in 4.ii.
      One verification is needed: the estimated value of an energy source must not exceeds its maximum capacity. If this would be the case, distribute the excessive amount over the remaining types.

Some other things to take into account:

• Don't use energy sources that are not being used in a country (eg. coal in Belgium), which the current estimation model is using. Actually, no special logic is required, just don't exceed the maximum capacity of an energy source.
• For unknown, the same method described in 4. can be used. For the current estimation model, some weird things are happening for unknown, eg. for Belgium, the actual value for unknown is always '?' (looks like 0 is being used), but the estimated value is a larger value, which has a negative impact on the carbon intensity. This should not happen.

[VIKTORVAV99]

I believe the estimation models need to follow a few ground rules to get good enough to be relied on. And these are:

1. The estimated production of all sources + imports - exports (or forecasted import and exports) should never exceed the load or forecasted load when available. (with a margin of error calculated for each zone)
2. It should prioritize forecasted data over estimated data when available (should help with most issues regarding wind and solar when the source have good estimations).
3. Nuclear and perhaps geothermal should use a linear estimation model based on the vector of the last 6 (exact time would have to be refined later) hours of production.
4. There should be region specific models that fill the gap between the other estimation methods and the load curve based on controllable power production (hydro, gas, biomass, coal etc)
5. Production should never exceed capacity - unavailable capacity for each mode (new parser(s) needed). (should also be a margin of error here as capacities are usually lagging by 1-2 years).

I think if we adapted all or most of these changes + added mode and region specific models we could have extremely accurate estimations of at least the next 1-24 hours (based on the available forecast data).

I realize this will take a while to properly implement if we do decide to do it, it would require significant changes to the backend, parsers and general methodology.

The new parsers would be for:

• Capacity Data to keep the upper bounds updated (should fall back to the ones declared in zones.json when unavailable or fails validation)
• Unavailable Data, to temporarily restrict the upper bounds. Currently we don't take this into account at all as far as I know but it's critical when it comes to estimating at the very least nuclear power over a longer period of time with high accuracy.

[pmensalt]

Same is happening again, but now with gas. Gas usage is increased by a lot lately. Estimated values still use previous values, when gas consumption was much lower. Therefore, estimated value for gas is way too low.

[KabelWlan]

From my observations, I check through most of the zones like twice a day, it happens to a few zones. I've seen it happen for West and East Denmark, Poland, Estonia, Germany, Norway, Sweden, all the Italy zones and as already stated for Belgium.

The proposed fixes would also improve the estimations for all the US zones, where the zone US-NW-WACM still shows 8.29GW of pumped hydro generation (already mentioned in #3907), US-SW-WALC has the same problem, reporting 21.8 GW of pumped hydro generation and US-NW-IPCO reports coal generation even though there is no coal power plant in this particular zone.

I feel like this is a major issue, and it should be addressed as soon as possible. It is to some extent misleading and distorts the coloring quite a bit.

[pmensalt]

Another example: estimated carbon intensity is way too high. Wind is under estimated, gas is over estimated.

[pierresegonne]

Hey all, reviving this thread as I'm trying to piece together suggestions for improving our estimations.

Have you noticed these kinds of issue in the last weeks?

Thanks!

[VIKTORVAV99]

Are you looking for issues that just affect our "realtime" estimations or the estimations that try to fill past data as well?

I have spotted a few issues where the estimation models are under estimating nuclear and over estimating solar (the other modes are probably affected as well) for the Swedens estimated historical data but I have a solution for the country level data at least.

As for the realtime estimations I have only seen a issue with the estimations estimate over the installed capacity notably for nuclear in Germany and Coal in Portugal (there has been issues opened about these) but they have been surprisingly good lately.

[pierresegonne]

I have spotted a few issues where the estimation models are under estimating nuclear and over estimating solar (the other modes are probably affected as well) for the Swedens estimated historical data but I have a solution for the country level data at least.

Please share :)

As for the realtime estimations I have only seen a issue with the estimations estimate over the installed capacity notably for nuclear in Germany and Coal in Portugal (there has been issues opened about these) but they have been surprisingly good lately.

Great, we did improve some things back there, seems to be working :)

[VIKTORVAV99]

I have spotted a few issues where the estimation models are under estimating nuclear and over estimating solar (the other modes are probably affected as well) for the Swedens estimated historical data but I have a solution for the country level data at least.

Please share :)
I'm not sure it's a solution on it's own but it should at least help, #4921 would allow us to get the correct nuclear production on a country level and I think we can assign all of that production to SE-SE3 as well but I'll have to check when the last nuclear power plant in SE-SE4 closed.

Wind, hydro, solar and unknown will still be tricky tho.
But my hope is that the historical capacity parsers I proposed in slack combined with average load factors could be used to help estimate solar at least (combined with the existing weather based system).

As for the realtime estimations I have only seen a issue with the estimations estimate over the installed capacity notably for nuclear in Germany and Coal in Portugal (there has been issues opened about these) but they have been surprisingly good lately.

Great, we did improve some things back there, seems to be working :)

And it appears to have paid off! Now we just need to figure out the exchanges to get a "total" estimate 😅

[pmensalt]

I haven't seen any issues anymore in the last couple of weeks, estimated values are now in line with what one would expect. So a big thank you to the team 👍