Handle missing type and subtype of Generators #35

gplssm · 2017-09-04T15:37:20Z

Problem

Some generators imported from Ding0 data do not have specified and type and/or subtype. We need this information for several things

Determine names of the components
Aggregated capacity according to type and subtype
Assign the right time series for each generator
...

Thus, we somehow need to define type/subtype if this information is missing.

Ideas/ Questions

We can solve it either in Ding0, eDisGo or in the data processing (DP)
@birgits votes for solving the issue in the DP. In general, I agree on that, but I will take us a while until we have a versioned version of this data (meanwhile we can work with model_draft but even this takes time).
Thus the idea would be to temporarily implement a workaround in eDisGo that assign data for missing type and subtype.
What should the pattern be like?
We could replace each missing type/subtype by 'other' or 'unknown'. This is simply straightfoward but causes other issues. For example if the type is missing, we should anyhow guess its type in order to find the right time series for this generator. For this we can create a translation dict that re-maps subtypes (when type is missing) to the actual type.

nesnoj · 2017-09-05T05:02:56Z

I'd like to get a better understanding why this happens in DP. Is it a bug or a feature?
Which DP version are you using? The current (unstable) one?
(in MVGD 134 (former 3545) the type/subtype exists).

nesnoj · 2017-09-05T11:37:53Z

I will check if problem occurs with unstable dataset.

nesnoj · 2017-09-05T12:47:29Z

The type-subtype list in both versions match.
I'm wondering why there are missing types. Executing

SELECT count(id), generation_type, generation_subtype
  FROM supply.ego_dp_res_powerplant
  WHERE version='v0.2.10'
  GROUP BY generation_type, generation_subtype
  ORDER BY generation_type, generation_subtype;

all types are filled, there's only one empty subtype in type solar... @gplssm

gplssm · 2017-09-05T13:45:25Z

Thanks for the investigation on this!

I'm wondering that missing type specification only occurs at subtype of solar. @miguelparadacontzen found missing subtype of run_of_river

Besides that, I gives sense that both RES datasets in open_eGo have the same missing subtype (solar). Both table found on the same EE Anlagenregister.

Further questions/ thoughts

If only the subtype of solar is sometimes missing, we could use a shortcut and just do an exception handling for this type. Good idea?
I guess the missing subtypes is caused by missing data in the EE Anlagenregister. Based on my current knowledge, I would suggest to handle missing subtypes in the DP, when the RES tables are created.
How to we handle solar generators with missing subtype in Ding0?

nesnoj · 2017-09-05T19:41:06Z

@miguelparadacontzen Is the subtype still missing in the current data? (see link above)

Concerning your questions:

I'd set the subtype in ding0, not in eDisGo.
Explanation:

We do not use the subtypes in dingo. Instead, we allocate generators using the nominal power. In this procedure, we implicity assume a subtype. Therefore, I'd set the subtype of solar in ding0. A detailed investigation reveals that we have solar plants without subtype in voltage levels 4..7.
Procedure:

v_level 7: P <= 30kW: roof mounted, 30kW < P <= 100kW: roof mounted, P > 100kW: either roof- or ground-mounted ???
v_level 6: roof mounted (since 86% of all solar plants in v_level 6 are roof mounted)
v_level 5+4: ground mounted (these plants are, in most cases, too large to fit on a roof)

We cannot really solve this problem in DP, can we? (I assume the method described above does not affect our project partners?!)
Furthermore, other users would not get this information in the dingo datasets if we used a workaround in eDisGo.

miguelparadacontzen · 2017-09-06T10:46:02Z

With the current version of the dev branch, running dingo over all MV districts from 4 to 3607, gives the following generation type/subtypes at MV level:

Occurances of biomass/biofuel                      69
Occurances of biomass/biogas                     1464
Occurances of biomass/biogas_dry_fermentation       7
Occurances of biomass/biogas_from_grid            170
Occurances of biomass/biomass                     745
Occurances of biomass/other                        32
Occurances of biomass/waste_wood                    8
Occurances of biomass/wood                         24
Occurances of coal/other                            2
Occurances of gas/gas                              89
Occurances of gas/gas_landfill                     83
Occurances of gas/gas_mine                         23
Occurances of gas/gas_sewage                       48
Occurances of gas/other                            50
Occurances of geothermal/geothermal                10
Occurances of lignite/other                         4
Occurances of oil/other                             8
Occurances of other_non_renewable/other             5
Occurances of reservoir/other                       1
Occurances of run_of_river/hydro                  427
Occurances of run_of_river/other                   35
Occurances of solar/other                        1191
Occurances of solar/solar_ground_mounted         1227
Occurances of solar/solar_roof_mounted           1714
Occurances of waste/other                          31
Occurances of wind/wind_onshore                  1408

"other" comes whenever subtype==None

miguelparadacontzen · 2017-09-06T10:59:15Z

At LV level:

Occurances of biomass/biofuel                  203
Occurances of biomass/biogas                  3835
Occurances of biomass/biogas_from_grid         459
Occurances of biomass/biomass                 2808
Occurances of biomass/wood                       9
Occurances of gas/gas                          144
Occurances of gas/gas_landfill                  97
Occurances of gas/gas_mine                       1
Occurances of gas/gas_sewage                   157
Occurances of geothermal/geothermal             20
Occurances of run_of_river/hydro              6149
Occurances of solar/other                   170631
Occurances of solar/solar_ground_mounted     37327
Occurances of solar/solar_roof_mounted      299170
Occurances of wind/wind_onshore               1245

nesnoj · 2017-09-06T11:06:34Z

Thx!
The missing data (except the solar plants discussed above) origin from the conv. PP table! In this table we do not have data on type/subtype, that's why we use column fuel in dingo..so all "other"-subtypes origin from conv. PP table.
It looks like we have to set the subtype manually if needed.

gplssm · 2017-09-11T14:15:56Z

Had short chat with @wolfbunke: Missing subtypes of solar PP originates from from missing data in EE Anlagenregister which cannot be resolved that easily in DP. Means, we need to use assumptions in order to this. Hence, we can resolve it entirely in Ding0.

nesnoj · 2017-09-12T07:44:56Z

I can take care of it..

nesnoj · 2017-09-14T18:47:12Z

@miguelparadacontzen How can I get these statistics

Occurances of biomass/biofuel                      69
Occurances of biomass/biogas                     1464
Occurances of biomass/biogas_dry_fermentation       7
Occurances of biomass/biogas_from_grid            170
Occurances of biomass/biomass                     745
Occurances of biomass/other                        32
Occurances of biomass/waste_wood                    8
Occurances of biomass/wood                         24
...

in dingo? I tried your stats branch but got only CSVs with detailed data..

miguelparadacontzen · 2017-09-18T07:54:19Z

I wrote a small script to get those numbers from the saved .csv file with the stats that I have locally in my hard drive:

#############################################
    ########## MV stats
    stats = pd.DataFrame.from_csv('dingo_grids_1_to_3607_mv_stats.csv')
    cols  = [c for c in stats.columns if c[:15] == 'Gen. Cap. of MV']
    stats = stats[cols]
    cols  = [c for c in stats.columns if c[:20] != 'Gen. Cap. of MV at v']
    stats = stats[cols]

    stats.columns = [ 'Occurances of '+c[16:] for c in stats.columns]

    stats = stats.T.astype(bool).sum(axis=1)
    print(stats)

    ########## LV stats
    stats = pd.DataFrame.from_csv('dingo_grids_1_to_3607_lv_stats.csv')
    cols  = [c for c in stats.columns if c[:14] == 'Gen. Cap. type']
    stats = stats[cols]

    stats.columns = [ 'Occurances of '+c[15:] for c in stats.columns]

    stats = stats.T.astype(bool).sum(axis=1)
    print(stats)

nesnoj · 2017-09-18T08:56:04Z

Thanks!

@gplssm I'm running out of time and have to dig deeper into #21, could you please take care of the type & subtype stuff?

gplssm · 2017-09-18T09:11:43Z

@gplssm I'm running out of time and have to dig deeper into #21, could you please take care of the type & subtype stuff?

Yes, but not immediately.

nesnoj · 2017-09-18T09:28:14Z

Cheers!

If a subtype of a RES generator is unknow it is set to 'unknown'. Conventional generators' subtype is set to 'unknown' by default. Tested with MV grid districts in Ding0 and eDisGo. * 391 * 2507 * 1066 * 1117 * 1167 Above MVGDs include both RES generators with unknown subtype and conventional generators. For further discussion details see [here](openego/eDisGo#35).

gplssm · 2017-09-22T12:35:00Z

This should be fixed by openego/ding0@effb54e. If it's still an issue, please reopen.

nesnoj · 2017-09-22T12:42:45Z

Great! But why didn't you work with the assumptions from above?

Do you want to be strict not to use further assumptions additionally to the original data provided by OPSD?

gplssm · 2017-09-22T12:57:21Z

To be honoest, I overlooked it. I'm undecided if we want to mix up data provided by OPSD dataset and own assumptions. They are quite valid (I would guess) but we cannot be sure... I'll give you a call on that soon

nesnoj · 2017-09-22T14:24:11Z

We keep it for now.
Reopen as soon as we need assumptions on subtypes..(e.g. for LV paper)

gplssm added help wanted question labels Sep 4, 2017

gplssm added this to the Power flow analysis with timeseries milestone Sep 4, 2017

gplssm assigned gplssm, birgits and nesnoj Sep 4, 2017

nesnoj unassigned gplssm and birgits Sep 14, 2017

gplssm assigned gplssm and unassigned nesnoj Sep 18, 2017

gplssm closed this as completed Sep 22, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle missing type and subtype of Generators #35

Handle missing type and subtype of Generators #35

gplssm commented Sep 4, 2017

nesnoj commented Sep 5, 2017

nesnoj commented Sep 5, 2017

nesnoj commented Sep 5, 2017 •

edited

Loading

gplssm commented Sep 5, 2017

nesnoj commented Sep 5, 2017

miguelparadacontzen commented Sep 6, 2017 •

edited

Loading

miguelparadacontzen commented Sep 6, 2017

nesnoj commented Sep 6, 2017 •

edited

Loading

gplssm commented Sep 11, 2017

nesnoj commented Sep 12, 2017 •

edited

Loading

nesnoj commented Sep 14, 2017

miguelparadacontzen commented Sep 18, 2017

nesnoj commented Sep 18, 2017 •

edited

Loading

gplssm commented Sep 18, 2017

nesnoj commented Sep 18, 2017

gplssm commented Sep 22, 2017 •

edited by nesnoj

Loading

nesnoj commented Sep 22, 2017

gplssm commented Sep 22, 2017

nesnoj commented Sep 22, 2017 •

edited

Loading

Handle missing type and subtype of Generators #35

Handle missing type and subtype of Generators #35

Comments

gplssm commented Sep 4, 2017

Problem

Ideas/ Questions

nesnoj commented Sep 5, 2017

nesnoj commented Sep 5, 2017

nesnoj commented Sep 5, 2017 • edited Loading

gplssm commented Sep 5, 2017

nesnoj commented Sep 5, 2017

miguelparadacontzen commented Sep 6, 2017 • edited Loading

miguelparadacontzen commented Sep 6, 2017

nesnoj commented Sep 6, 2017 • edited Loading

gplssm commented Sep 11, 2017

nesnoj commented Sep 12, 2017 • edited Loading

nesnoj commented Sep 14, 2017

miguelparadacontzen commented Sep 18, 2017

nesnoj commented Sep 18, 2017 • edited Loading

gplssm commented Sep 18, 2017

nesnoj commented Sep 18, 2017

gplssm commented Sep 22, 2017 • edited by nesnoj Loading

nesnoj commented Sep 22, 2017

gplssm commented Sep 22, 2017

nesnoj commented Sep 22, 2017 • edited Loading

nesnoj commented Sep 5, 2017 •

edited

Loading

miguelparadacontzen commented Sep 6, 2017 •

edited

Loading

nesnoj commented Sep 6, 2017 •

edited

Loading

nesnoj commented Sep 12, 2017 •

edited

Loading

nesnoj commented Sep 18, 2017 •

edited

Loading

gplssm commented Sep 22, 2017 •

edited by nesnoj

Loading

nesnoj commented Sep 22, 2017 •

edited

Loading