Flatten-Tool leaks file descriptors

[jpmckinney]

To reproduce, clone cove-ocds and run:

pytest -W error

For example:

pytest -W error tests/test_general.py

Error looks like:

ResourceWarning: unclosed file <_io.BufferedReader name='/var/folders/cs/3pnj707s581bjnsfv0wndqkh0000gn/T/flattentool-ce6fe78e-8ec2-4133-af9d-aca7a85f34e8'>

I tried this diff, but it didn't change anything:

diff --git a/flattentool/json_input.py b/flattentool/json_input.py
index d802a6c..d591b6a 100644
--- a/flattentool/json_input.py
+++ b/flattentool/json_input.py
@@ -126,10 +126,10 @@ class JSONParser(object):
                 tempfile.gettempdir() + "/flattentool-" + str(uuid.uuid4())
             )
             # zlibstorage lowers disk usage by a lot at very small performance cost
-            zodb_storage = zc.zlibstorage.ZlibStorage(
+            self.zodb_storage = zc.zlibstorage.ZlibStorage(
                 ZODB.FileStorage.FileStorage(self.zodb_db_location)
             )
-            self.db = ZODB.DB(zodb_storage)
+            self.db = ZODB.DB(self.zodb_storage)
         else:
             # If None, in memory storage is used.
             self.db = ZODB.DB(None)
@@ -606,9 +606,10 @@ class JSONParser(object):
         return self
 
     def __exit__(self, type, value, traceback):
+        self.connection.close()
+        self.db.close()
         if self.persist:
-            self.connection.close()
-            self.db.close()
+            self.zodb_storage.close()
             os.remove(self.zodb_db_location)
             os.remove(self.zodb_db_location + ".lock")
             os.remove(self.zodb_db_location + ".index")

Note that zc.zlibstorage.ZlibStorage copies the close() method of ZODB.FileStorage.FileStorage, so I don't expect it would do anything to also call close() on the ZODB.FileStorage.FileStorage object.

To be paranoid, can try this:

diff --git a/flattentool/json_input.py b/flattentool/json_input.py
index d802a6c..2af9462 100644
--- a/flattentool/json_input.py
+++ b/flattentool/json_input.py
@@ -126,10 +126,9 @@ class JSONParser(object):
                 tempfile.gettempdir() + "/flattentool-" + str(uuid.uuid4())
             )
             # zlibstorage lowers disk usage by a lot at very small performance cost
-            zodb_storage = zc.zlibstorage.ZlibStorage(
-                ZODB.FileStorage.FileStorage(self.zodb_db_location)
-            )
-            self.db = ZODB.DB(zodb_storage)
+            self.file_storage = ZODB.FileStorage.FileStorage(self.zodb_db_location)
+            self.zodb_storage = zc.zlibstorage.ZlibStorage(self.file_storage)
+            self.db = ZODB.DB(self.zodb_storage)
         else:
             # If None, in memory storage is used.
             self.db = ZODB.DB(None)
@@ -606,9 +605,11 @@ class JSONParser(object):
         return self
 
     def __exit__(self, type, value, traceback):
+        self.connection.close()
+        self.db.close()
         if self.persist:
-            self.connection.close()
-            self.db.close()
+            self.file_storage.close()
+            self.zodb_storage.close()
             os.remove(self.zodb_db_location)
             os.remove(self.zodb_db_location + ".lock")
             os.remove(self.zodb_db_location + ".index")

[jpmckinney]

Now that I've switched to FlattenToolWarning, I've noticed that there are still ResourceWarnings about unclosed files (solution is to close files).