-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix import from bw2 generated packages #220
base: main
Are you sure you want to change the base?
Conversation
Thanks @tfardet! I think we need some tests here. I can do this, but if you already have some older (and small :) fixtures then you are welcome to add them. I will try this weekend, but am not promising anything, there is already a list. |
OK, I tried to add the very simple attached package (which I generated with brightway 2) If I do from bw2io import BW2Package
BW2Package.import_file("bw2_compat_test.bw2package") it works fine, however, the following test import os
import pytest
from bw2data.tests import bw2test
from bw2io import BW2Package
FIXTURES = os.path.join(os.path.dirname(__file__), "..", "fixtures", "bw2package")
@bw2test
def test_bw2_compat():
obj = BW2Package.import_file(os.path.join(FIXTURES, "bw2_compat_test.bw2package"))[0]
a = obj.get("7599062216496486961")
self.assertTrue(a["name"] == "partial_respiration")
self.assertTrue(a["unit"] == "g")
self.assertTrue(a["type"] == "process")
a = obj.get("3866902554231372371")
self.assertTrue(a["name"] == "C_inactivated")
self.assertTrue(a["unit"] == "g")
self.assertTrue(a["type"] == "production") fails on import @bw2test
def test_bw2_compat():
> obj = BW2Package.import_file(os.path.join(FIXTURES, "bw2_compat_test.bw2package"))[0]
tests/bw2package/compat_bw2_import.py:13:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
bw2io/package.py:238: in import_file
return [cls._create_obj(o) for o in loaded]
bw2io/package.py:238: in <listcomp>
return [cls._create_obj(o) for o in loaded]
bw2io/package.py:129: in _create_obj
instance.write(data["data"])
../../../.pipenv/lib/python3.11/site-packages/bw2data/project.py:432: in writable_project
return wrapped(*args, **kwargs)
../../../.pipenv/lib/python3.11/site-packages/bw2data/backends/base.py:535: in write
self.process()
../../../.pipenv/lib/python3.11/site-packages/bw2data/backends/base.py:755: in process
dp.add_persistent_vector_from_iterator(
../../../.pipenv/lib/python3.11/site-packages/bw_processing/datapackage.py:441: in add_persistent_vector_from_iterator
) = resolve_dict_iterator(dict_iterator, nrows)
../../../.pipenv/lib/python3.11/site-packages/bw_processing/utils.py:73: in resolve_dict_iterator
array = create_structured_array(
../../../.pipenv/lib/python3.11/site-packages/bw_processing/array_creation.py:82: in create_structured_array
array = create_chunked_structured_array(iterable, dtype)
../../../.pipenv/lib/python3.11/site-packages/bw_processing/array_creation.py:44: in create_chunked_structured_array
for chunk in chunked(iterable, bucket_size):
../../../.pipenv/lib/python3.11/site-packages/bw_processing/array_creation.py:21: in <lambda>
return iter(lambda: list(itertools.islice(iterable, chunk_size)), [])
../../../.pipenv/lib/python3.11/site-packages/bw_processing/utils.py:72: in <genexpr>
data = (dictionary_formatter(row) for row in iterator)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = Brightway2 SQLiteBackend: partial_respiration_db
sql = "SELECT e.data, a.id, b.id, e.input_database, e.input_code, e.output_database, e.output_code\n FROM exc... == e.output_database\n WHERE e.output_database = ?\n AND e.type = 'biosphere'\n "
dependents = {'biosphere3'}, flip = False
def exchange_data_iterator(self, sql, dependents, flip=False):
"""Iterate over exchanges and format for ``bw_processing`` arrays.
``dependents`` is a set of dependent database names.
``flip`` means flip the numeric sign; see ``bw_processing`` docs.
Uses raw sqlite3 to retrieve data for ~2x speed boost."""
connection = sqlite3.connect(sqlite3_lci_db._filepath)
cursor = connection.cursor()
for line in cursor.execute(sql, (self.name,)):
(
data,
row,
col,
input_database,
input_code,
output_database,
output_code,
) = line
# Modify ``dependents`` in place
if input_database != output_database:
dependents.add(input_database)
data = pickle.loads(bytes(data))
check_exchange(data)
if row is None or col is None:
> raise UnknownObject(
(
"Exchange between {} and {} is invalid "
"- one of these objects is unknown (i.e. doesn't exist "
"as a process dataset)"
).format(
(input_database, input_code), (output_database, output_code)
)
)
E bw2data.errors.UnknownObject: Exchange between ('biosphere3', '9ec076d9-6d9f-4a0b-9851-730626ed4319') and ('partial_respiration_db', '7599062216496486961') is invalid - one of these objects is unknown (i.e. doesn't exist as a process dataset)
../../../.pipenv/lib/python3.11/site-packages/bw2data/backends/base.py:680: UnknownObject I'm not sure what I'm missing here... |
Hi @cmutel do you think you could have a look at what the problem is? (conflicting db versions?) |
Fixes #135
As @renaud correctly analyzed, we need to map the 2 and 2.5 backend structures so the import can happen.
Note that this is important for people who don't have access to Ecoinvent at the moment and want to check the BW notebook using Forwast, for instance, as it was generated with bw2
EDIT: I imagine that bw25 package will always be bundled together so I did not check
bw2data
version, let me know if this is necessary