ETraGoSpecs (eTraGo-eDisGo-Interface)

[maltesc]

In addition to #2, the interface function get_etragospecs is implemented in:
eGo/ego/tools/specs.py (see: specs.py)

However, I need help with extending the function and verifying a few parameters (I don't necessarily need the exact answer - also a hint to the corresponding code section might help)

1. Information is needed on how to merge Ding0 grids with eTraGo buses. Is there a table that associates Ding0 grid IDs with transmission grid bus IDs?

2. What exactly is the column e_annual in table model_draft.ego_grid_pf_hv_result_load? What is the unit?

3. Is there a definition of 'extendable storage'? In my point of view this is the extendable short term storage (batteries) and thus there should be only one per TG bus. However, e.g. at bus 27334 there are 6. For eDisGo the installed capacities and active power series of the batteries are of particular interest.

4. eDisGo needs the Curtailment for wind and solar. In the eTraGo results I find the installed capacity and the corresponding dispatch. Where can I find the hourly potential in order to calculate the curtailment?

5. Zählpfeilsystem in eTraGo results in my point of view: storage charged = negative P and discharged = positive P. Is this correct (I have only seen dummy results yet)?

[gplssm]

Today I'll review implementation by @maltesc and report back here.

I cannot answer the above question. Could you @ulfmueller @IlkaCu @wolfbunke @mariusves or @BartelsJ help out here?

[ulfmueller]

1. @IlkaCu knows better how the grid district ids evolve in the hv-mv bus ids of the powerflow data. @gplssm and @nesnoj should know where ding0 gets the grid districts ids. But it should be consistent :)

2. e_annual is the overall load per bus per year. It is not used by eTraGo. More details from @IlkaCu

3. extendable storages could be also for longterm storage. If a bus has a potential for both. Two storage units should be possible. 6 seems awkward. @lukasoldi knows about this stuff in detail.

4. the p_max_pu * p_nom defines the potential for each snapshot and power plant

5. eTraGo uses the pypsa convention on the "Zählpfeilsystem" ;), which is a Generator-Zählpfeilsystem. So yes, generation is positive and load is negative.

[gplssm]

1. Indeed, we know where we take this information from: the column "subst_id" from grid.ego_dp_mv_griddistrict. But this wasn't actually the question. The question is: how to join this subst_id (ranging from 1..3608) with the bus_id (by far higher range) for example found in "model_draft.ego_grid_pf_hv_result_bus".

[ulfmueller]

I think I understood the question. I would say what you explained is one side of the medal. The other side can be answered by @IlkaCu . She knows in which table(s) the subst_id gets translated to 'our' final id.

[ulfmueller]

it has to do with the data which comes from osmTGmod, which has 'otg_ids'. But I think it is easier if @IlkaCu explains this.

[IlkaCu]

Table model_draft.ego_grid_hvmv_substation contains the otg_id (which is the id from osmTGmod) as well as the subst_id. We use this table in the DP to link both IDs.

[lukasoldi]

extendable storages could be also for longterm storage. If a bus has a potential for both. Two storage units should be possible. 6 seems awkward. @lukasoldi knows about this stuff in detail.

that is correct. 6 seems not correct. Are you sure this is for one scenario? Could you share the 6 items so we can check for doubling?

[maltesc]

Thank you for the numerous and informative feedback.

@lukas and @ulfmueller The mentioned case occurred in the current model_draft results version (result ID = 1, no clustering). Not sure if this is for one scenario, however, the result_id is supposed to be unique. There are many similar cases among the table, however here is the example:

table: model_draft.ego_grid_pf_hv_result_storage

At bus 27334, there are 6 extendable storages:
Storage IDs: 50886, 10967, 40990, 39318, 1071, 29422

[ulfmueller]

I think I know why that occurred. The model_draft data is the result of the debugging of the data processing. In that process the script which installs the extendable storages was executed more than once. To make a long story short: The model_draft data is unstable up until the clean-run was performed. This bug should not appear in the grid schema data.

[maltesc]

The function get_etragospecs_from_db() (see: specs.py) has now been completely implemented. I queries eTraGo result-data from a database and "converts" it into the eDisGo input-class ETraGoSpecs.

One remaining issue are the 'load_types'. eTraGo aggregates all loads and thus, from the results these can not be disaggregated. However, eDisGo needs the distinguished loads ('residential', 'industrial', etc.). Currently in get_etragospecs_from_db this is "hacked", assuming all loads are 'residential'. Al the necessary data should be available in the oedb. @gplssm, @ulfmueller: The question is, if this query should be implemented at get_etragospecs_from_db() (which makes it somehow less consistent, since it is designed to only query eTraGo results), at another place in eDisGo or at another input function?

Furthermore, I made the assumption that the desired 'battery_capacity' for eDisgo is the short-term (less than 20 hours) extendable storage from eTraGo. This implies that no already existing storages can be passed to eDisgo (which I think makes sense, cause eDisGo should only analyze the redistribution only "new", thus "extended" storages). I hope you agree.

[ulfmueller]

The question is, if this query should be implemented at get_etragospecs_from_db() (which makes it somehow less consistent, since it is designed to only query eTraGo results), at another place in eDisGo or at another input function?

If the get_etragospecs_from_db() function is designed as an interface I think it would be ok to translate the loads into a desired resolution taking information from another place in that same function (if it is more consistent to do this somewhere else, ok for me too). In my mind ding0 would have the information about the structure of the load areas (or you get it from the data processing). With the help of that information you can split up the load again into several sectors...

Furthermore, I made the assumption that the desired 'battery_capacity' for eDisgo is the short-term (less than 20 hours) extendable storage from eTraGo. This implies that no already existing storages can be passed to eDisgo (which I think makes sense, cause eDisGo should only analyze the redistribution only "new", thus "extended" storages). I hope you agree.

ok, but if there are already existing storages installed they should also be considered in eDisGo - just as an existing power plant (e.g. PV). So eDisGo has to get their dispatch as well in order to get consistent power flows, right?

[maltesc]

ok, but if there are already existing storages installed they should also be considered in eDisGo - just as an existing power plant (e.g. PV). So eDisGo has to get their dispatch as well in order to get consistent power flows, right?

This is an interesting point. As far as I know this is not intended in the ETraGoSpecs class, since I think only generators and battery storage is implemented. However, @ulfmueller might be right and this is necessary. I would suggest to query the existing storages and pass them to ETraGoSpecs just as generators (with corresponding storage-type). The ETraGoSpecs battery-storage series is more for spacial redistribution I guess.

In my mind ding0 would have the information about the structure of the load areas (or you get it from the data processing)

I assume it is the easiest to get the information directly from data processing.

[maltesc]

In the current version, the specs.py represent the value of the TG buses (and NOT the MV side of the transformer, as demanded by eDisGo). Therefore, I assume that the current specs need to be adjusted to subtract load and feed-in, connected to the HV side (such as industrial load etc.).

Find a schematic drawing here (https://www.dropbox.com/s/lcgnty28hay70s0/HVMVgap.jpeg?dl=0).

Since for this purpose, data needs to be queried from the Data Processing at different levels, this task is somewhat complicated and confusing. The specs would thereby become so to speak part of the model and remain not only an interface. @gplssm and @ulfmueller: The question is if this is desired, or if the eTraGo or eDisGo model needs to be adjusted in order to "fill the gap".

In case the Specs shall fill this gap, the following data is needed:

1. Load
   Currently the eTraGo results only contain aggregated loads. I assume the load_id from the results represents the load area. As said before in (ETraGoSpecs (eTraGo-eDisGo-Interface)  #11 (comment)), these need to be disaggregated into all load_types.
   In order to subtract the e_annual of large scale consumers (connected to HV side), I got the information to use the table ego_demand_hv_largescaleconsumer. @IlkaCu, is the un_id of this table identical with the load_id from the results? I assume that I can query the sectoral e_annual demand for the MV loads from table ego_demand_loadarea. @IlkaCu, but where can I find the exact time series of the sectoral load?

2. Generators
   Generators are slightly easier to handle. Since from the eTraGo results it is easy to calculate the normalized dispatch (and all generators of the same type are dispatched identically), this normalized value can be passed to eDisGo without any problems. With the installed generator capacity of the MV grids (available in the ding0 grids), eDisGo can calculate the absolute dispatch easily. Therefore the generator capacity from the eTraGo results (minus the installed capacity on the HV side) would only be a check-up. @ulfmueller and @gplssm, is this check-up necessary?
   In case this check shall not be disregarded, this get's a bit complicated. In this case I need to identify every single generator associated to the corresponding TG bus and query its voltage level, capacity and type. @IlkaCu, how can I get (coming from the aggregated result "generator_id") to this information?

3. Storage
   I assume that the storage can be handled in a similar way as the generators. @ulfmueller, is it correct that same as generator types, the storage types within a type are all dispatched identically? In this case also normalized values can be used and I assume ding0 contains all (already existing) MV-connected storage devices. Extendable storage is not a problem, since these are explicitly part of the results.

4. P/Q
   I assume that P*/Q* are only checkup-values, since with the correct dispatch etc., the eDisGo results should be (more or less) identical. However, it is probably very important to keep this information. In this case it get's again complicated since we need the time series for every device connected to the HV side in order to subtract from the current eTraGo P/Q.

5. HV/MV transformer
   I assume that furthermore it becomes necessary to eventually model the HV/MV transformer. Is this already done at some place, or is there any data available?

[ulfmueller]

Since for this purpose, data needs to be queried from the Data Processing at different levels, this task is somewhat complicated and confusing. The specs would thereby become so to speak part of the model and remain not only an interface. @gplssm and @ulfmueller: The question is if this is desired, or if the eTraGo or eDisGo model needs to be adjusted in order to "fill the gap".

Generally I think eDisGo needs the information of what load and gneration is installed in 'their' mv and lv grids (similar to that we have to know whether a load/generator is in the hv or ehv level) independently from the interface. This information can be used by the interface.
Since the load is inelastic you can model the load in the mv and lv level just coming from the input data. If it is easier to take the load curve from eTraGo and subdevide it to eDisGo loads - that would be possible but then eDisGo itself could not really work without being connected to eTraGo.
The interface supplies the power flow from the hv side of the hv/mv transformer (but like you state later, this is rather a check up value) and most importantly the dispatch of the generators and storages per "marginal cost type" so far equivalent to 'carrier'.

Currently the eTraGo results only contain aggregated loads. I assume the load_id from the results represents the load area.

No, they represent the entire load of the mv grid district plus industrial consumers which are directly connected to the HV level. Considering 'our' terminology a grid district consists of n load areas.

Therefore the generator capacity from the eTraGo results (minus the installed capacity on the HV side) would only be a check-up. @ulfmueller and @gplssm, is this check-up necessary?

It would be a nice-to-have check-up which I would appreciate if it is done (at least by sample). But it is not a must-have for the final methodology because if everyone worked well there should be no problem 😉

I assume that the storage can be handled in a similar way as the generators. @ulfmueller, is it correct that same as generator types, the storage types within a type are all dispatched identically?

every power plant or storage is dispatched (once built) with respect to their marginal costs. The most accurate is to group by marginal cost groups. That is for sure and reliable. Even though if they are considered same right now (which they are) it is more robust to just take the information of the marginal costs.

4. P/Q

Agree!

[IlkaCu]

Load
Currently the eTraGo results only contain aggregated loads. I assume the load_id from the results represents the load area. As said before in (#11 (comment)), these need to be disaggregated into all load_types.
In order to subtract the e_annual of large scale consumers (connected to HV side), I got the information to use the table ego_demand_hv_largescaleconsumer. @IlkaCu, is the un_id of this table identical with the load_id from the results?

Yes, that's right! The corresponding SQL script which assigns the load to the corresponding bus is ego_dp_powerflow_assignment_load.sql

I assume that I can query the sectoral e_annual demand for the MV loads from table ego_demand_loadarea. @IlkaCu, but where can I find the exact time series of the sectoral load?

The aggregated time series are created in ego_dp_powerflow_griddistrict_demand.py. As far as I know only the aggregated time series are stored in the database. @gplssm: Do I have this right?

[maltesc]

Thank you for the collaboration. Today, during the conference call, we identified some specific approaches and tasks:

• For the moment, we decided to exclude any check-ups from the interface and thus, limit its extent only to the necessary data. This includes the deployment of normalized values.
• This implies adjusting not only of the interface, but also the ETraGoSpecs class in eDisGo.
• In the long run it is desirable to conduct some checks.

1. Load
   We decided to generally exclude load from the interface. Since load timeseries can be generated with eDisGo, this imformation is not needed from eTraGo anymore. @ulfmueller and @gplssm furthermore decided to generate all load series more centrally, so that eTraGo and eDisGo can use the same DP source. In the long run, this source can also be used to conduct the check (and subtract the large scale consumer timeseries from the TG bus load).

2. Generators
   We decided to exclude generator capacities from the interface. Alike loads, the capacity is available in eDisGo. As before, dispatch and curtailment will be normalized values. Problematic are wind and solar, since the aggregated generators are furthermore divided into weather cells. On the one hand we decided to adapt eTraGo so that the weather cell ID remains in the results and on the other hand adapt the ETraGoSpecs class in eDisGo so that it can take different weather cells.

3. Storage
   Currently storage (currently only battery storage in eDisGo) takes absolute values. Similar to generators, this must be changed to normalized values (normalized state of charge and active power).

@ulfmueller, will you manage the eTraGo adjustments (e.g. weather ID in the results)?
@gplssm,will you adjust the ETraGoSpecs class in eDisGo? Instead, I could also offer to implement these changes and thus, follow a more "this is what we can give"-approach instead of the current "this is what we would like to have"-approach. What do you think? I would of course try to stay as close as possible at the current implementation.

[ulfmueller]

I just talked about 1. and 2. with @IlkaCu . Concerning 1. @IlkaCu wanted to talk to @gplssm . Concerning 2. @IlkaCu had another simple idea (which she will post here) to get the weather_id for each aggregate without adding it to the eTraGo data....

[maltesc]

I also talked to @gplssm today and we discussed nr. 3.

1. Load will be completely integrated in eDisGo and will therefore, not be part of the interface.
2. Thank you @IlkaCu for your effort. I will appreciate help in this point.
3. A bit problematic is the normalization of the storage. In particular this is the further aggregation of the storage types (e.g. pump storage) to one value that will be passed to eDisGo (in particular e.g. "dispatch of pump storage"). The problem is that in the real world it can be assumed that the ratio of turbine power to storage capacity (depending on max hours and efficiencies) may differ between facilities and thus, a normalization on turbine power would lead to errors in capacity. However, from looking at the data, I can see that max hours for pump storage is constantly 6h. If this is assumed generally in open_eGo, this aggregation in the interface should be alright. Otherwise, all storage units of the eTraGo results need to be desaggregated via DP tables and passed individually to eDisGo. @ulfmueller, I will call you later concerning this issue.

[maltesc]

3. An alternative would be to exclude all storage types, except batteries (extendable storage short term), from the interface (assuming all others are connected to HV, which I think is the case). I think this is currently also assumed by eDisGo.

[maltesc]

2. Yesterday I talked to @ulfmueller and we discussed the integration of weather IDs in the interface. I will adjust the interface now and discuss the further necessary adjustments of eDisGo with @gplssm.

3. Concerning my last comment (ETraGoSpecs (eTraGo-eDisGo-Interface)  #11 (comment)):
   I checked the database tables model_draft.ego_supply_conv_powerplant and model_draft.ego_supply_conv_powerplant_2035 on fuel = 'pumped_storage' and worked out that no pumped storage facility is connected to lower than 110kV. That means, for now, we could exclude all pumped storages from the interface.
   @gplssm and @ulfmueller, excluding all storage types (except extendable) from the interface keeps it very simple and means almost no adjustments on the eDisGo side. On the other hand, one might loose some flexibility and the interface becomes more project specific. What is the right way?

[maltesc]

I almost finished the new version of the specs with the discussed adjustments.

I am surprised by the fact that some aggregated generators can have more than one weather ID. See e.g.:

SELECT DISTINCT w_id FROM model_draft.ego_supply_pf_generator_single
WHERE aggr_id = 31355

I assume this is because it is a conventional generator. However, this means I have to query the weather_id more specificly for the corresponding sources.

@IlkaCu and @ulfmueller, which sources/types are split into weather IDs in eTraGo? wind, solar, run of river etc.?

[ulfmueller]

only wind and solar, right @IlkaCu ?

[ulfmueller]

@gplssm and @ulfmueller, excluding all storage types (except extendable) from the interface keeps it very simple and means almost no adjustments on the eDisGo side. On the other hand, one might loose some flexibility and the interface becomes more project specific. What is the right way?

for me it would be ok to leave it out if it helps you @maltesc and @openego/edisgo (because a fast solution might be better than loosing too much time on rather 'minor' issues) . It would be nice if a change to a more generic solution would be thought of in a way that it could be added in future.

[maltesc]

@wolfbunke and @ulfmueller, the documentation of the new Specs can be found here: openego/eDisGo#90

[wolfbunke]

great, thank you!

[maltesc]

I just created an interesting plot:
I plotted the mean feed (200 hours, active power) in on the HV side (calculated by eTraGo, thus with aggregated load and generation) against the slack-bus feed-in in the MV side (caclulated by eDisGo, using the ETraGoSpecs as Interface) for about 400 MV grids.

In most of the cases the values match pretty well. However, in some cases they differ considerably.
@gplssm and @ulfmueller, do you have an idea what could be the reason?
I think we should use this thread to collect possible reasons and examine them.

I think this is because of load and generation capacity connected to the HV side. However, there might be many more reasons...

[nesnoj]

I cannot contribute that much here, but are you talking about SQ or 2035/eGo100?
I'm asking since the conventional generators are not connected yet in eDisGo in the future scenarios (openego/eDisGo#21 - shame on me, this is my issue).. this could explain the deviations

[maltesc]

Thank you @nesnoj, this is an important information!
This dataset is based on NEP 2035 Scenario.
For the generator import I use:

if scn_name == 'NEP 2035':
    network.import_generators(types=['wind', 'solar']) 
network.analyze()

I haven't really dug into the import_generators() function. Do I have to import more generators? What is exactly missing on the MV side?

[nesnoj]

The conventional genos are not updated according to future scenarios.
Whereas the RES have unique IDs in SQ, NEP2035 and eGo100 which makes an update (add, remove, modify) pretty straightforward, the IDs of conv. genos change from SQ to the future scenarios..
A discussion on this issue took place in openego/eDisGo#59 , it has not been implemented yet. Me or @gplssm is going to do this.

What you could do is to remove all MV Grid Districts which changing conv. capacities from the data you used for the above plot to find out whether this is the only reason for the deviation or not..

[nesnoj]

Addition: It might be bullshit what I'm telling you - does your query refer to the eDisGo geno import only or eTraGo as well?
If you import only wind and solar in eDisGo the HV-MV-trafo results are quite likely to differ, right? ;)

[gplssm]

Thanks for the plot! I would like to see the plot again with @nesnoj 's comments. No conventional; importing all types, not only wind and PV.