Updating ROOT file with Pandas DataFrame error

[niamh-h]

Hi! I'm trying to use a SciKit classifier on some data by reading it in from multiple ROOT TTrees to Pandas DataFrames (so I use uproot.open()). I then want to update the existing files with the classification data (e.g. confidence scores/probabilities etc) using uproot.update(). I'm having trouble with updating the files as they don't seem to be closing correctly despite being opened in a with statement. I've tried different ways of updating the files (below) but they all lead to the following error when I open the file in ROOT to check:

Error in <TBufferFile::CheckByteCount>: object of class TList read too few bytes: 17391 instead of 17773
Error in <TBufferFile::CheckByteCount>: Byte count probably corrupted around buffer position 64:
	17773 for a possible maximum of -9

I think this means it's not closing correctly? If I then retry the python code with the same ROOT file I get an error when trying to open it in a writable form using uproot.update() (but no error when using uproot.open()), I'm assuming because it's been corrupted: (FYI I have updated to uproot 4.1.5)

ValueError: Uproot can't read TKey version -1 for writing, only version 4

This is the code for two ways I've tried updating the file:

with uproot.update('Datafile.root') as file:
  file.mktree('t', {'branch1': np.int64, 'branch2':np.int64}, 'title')
  file['title'].extend({'branch1:df.loc[:,'branch1'], 'branch2':df.loc[:,'branch2']})

file = uproot.update('Datafile.root)
file['df'] = df
file.close()

(I've also tried this way inside a with statement).
Hope I've explained this ok, any help would be appreciated!

[jpivarski]

It's not a matter of not closing correctly, since you're using a with statement and, anyway, each modification to a file in Uproot is intended to take it from a valid state to a valid state, ending with a flush so close() doesn't actually invoke any clean-up—it's already cleaned up. (All that with is doing here is making us doubly confident that it has finished flushing.)

What seems to be happening here is a more general kind of bug on the Uproot side. You'll have to show me (1) a copy of a pre-updated file, (2) what steps you take to write to it with a dummy DataFrame, not your whole analysis, just one that can be created on the command line, and (3) how it fails to be re-openable with uproot.update. Then I'll figure out what Uproot is doing wrong in step (2).

That should be opened as an issue, rather than a discussion.