Add read_netcdf function

[LinhHo]

Please confirm that this PR has done the following:

• Tests Added
• Documentation Added
• Name of contributors Added to AUTHORS.rst
• Description in RELEASE_NOTES.md Added

Adding to RELEASE_NOTES.md (remove section after adding to RELEASE_NOTES.md)

Please add a single line in the release notes similar to the following:

- (#XX)[http://link-to-pr.com] Add read_netcdf() function

Description of PR

This PR adds a read_netcdf() function to read netcdf data (.nc). Note that this function currently does not support sub-annual data and meta indicators have the exact indices from META_IDX (currently ‘model’ and ‘scenario’).

[danielhuppmann]

Thanks for this first stab, looks nice!

There are a few suggestions inline - most importantly, pyam can handle long- and wide-format natively, no need to do these transformations in your code.

One bigger suggestion is to place the entire new code in a new file netcdf.py (to avoid core.py becoming ever longer) and add

from pyam.netcdf import read_netcdf

to __init__.py.

Also, please take a look at pep8 and use ruff or black to format your code.

And one question: What's the difference between "standard" netcdf vs. the aggregated calliope netcdf?

[danielhuppmann]

test_df already has meta indicators, see

pyam/tests/conftest.py

Line 154 in e27e90e

def test_df(request):

No need to add more of them as part of the test

Suggested change

	# add column to `meta` and write to datapackage
	test_df.set_meta(["a", "b"], "string")
	# read from csv assert that IamDataFrame instances are equal

[LinhHo]

Thanks for the information! When calling test_df to compare with dataframe from read_netcdf, an AttributeError occurs as below. As test_df is usually passed to functions such as pyam.to_csv(file), I think another else clause is needed in as_series(s) to get data directly from test_df for comparison in this case.

[LinhHo]

Thanks a lot for your comments! To answer your question about aggregated Calliope netCDF, the current output from Calliope (“standard” netCDF) has five dimensions: nodes, techs, carriers, timesteps, and costs. They contain variables according to the model’s specifications, e.g., flow cap, flow in, flow out, flow export, etc.

Aggregated Calliope netCDF (the input data for pyam.read_netcdf) are aggregated or disintegrated from standard Calliope netCDF files into variables compatible with the IAMC standard, such as Primary Energy | Coal. They have four dimensions: model, scenario, region, and time. Unit is an attribute of each variable. In the future, Calliope will be updated to also produce aggregated netCDF as an optional output.

I will be on vacation for the next two weeks and will address any further comments from you when I'm back.
Best regards,
Linh