Leaf-lists (structs) and non-leaf-lists in the same Pandas DataFrame

[bhbam]

Hi, I am trying to using uproot to read root file into pandas data frame. The root file have some structured data branches some not.

tree.show()
name | typename | interpretation
---------------------+--------------------------+-------------------------------
Event | struct {int64_t run; ... | AsDtype("[('run', '>i8'), (...
GenPhoton1 | struct {double pt; do... | AsDtype("[('pt', '>f8'), ('...
GenPhoton2 | struct {double pt; do... | AsDtype("[('pt', '>f8'), ('...
Photon1 | struct {double pt; do... | AsDtype("[('pt', '>f8'), ('...
Photon2 | struct {double pt; do... | AsDtype("[('pt', '>f8'), ('...
Vertex0 | struct {double vx; do... | AsDtype("[('vx', '>f8'), ('...
PrimaryVertex | struct {double vx; do... | AsDtype("[('vx', '>f8'), ('...
nPV | int32_t | AsDtype('>i4')
BeamSpot | struct {double x0; do... | AsDtype("[('x0', '>f8'), ('...

I can see branch "nPV" is different from other branch that why I think I'm getting error when I'm trying to read this to pandas data frame.
KeyError: ('nPV', nan) -> This is the error showing.
If I remove "nPV" branch than it working fine.

[jpivarski]

Could you include the whole stack trace and maybe a copy of the file?

Is it able to read all of the branches with library="np" or library="ak", but not library="pd"?

The "KeyError: ('nPV', nan)" is a bug; some step in the operation is trying to find a key in a dict that is a string and a number: ('nPV', nan), but NaN is a "not a number" value that is never equal to itself:

>>> float("nan") == float("nan")
False

So if a key in that dict included NaN, it shouldn't (in principle*) ever be accessible:

>>> internal_dict = {("nPV", float("nan")): 3.14, ("nPV", 123): 2.71}
>>> internal_dict[("nPV", float("nan"))]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: ('nPV', nan)
>>> internal_dict[("nPV", 123)]
2.71

The Uproot code should not be allowing the second item of this tuple to be NaN. It's probably a computed value, and your data includes a NaN in it somewhere that is making the computed value also NaN. But I can't debug it if I don't see the details.

---

*As a caveat, to say "in principle," NumPy's np.nan sometimes matches itself in dicts and sometimes doesn't. I haven't figured out why, but that might even be a bug in NumPy or Python. (An extreme corner case; NaN is not a good value to use in dicts!)

>>> internal_dict = {("nPV", np.nan): 3.14, ("nPV", 123): 2.71}
>>> internal_dict[("nPV", np.nan)]
3.14
>>> np.nan + 1
nan
>>> internal_dict[("nPV", np.nan + 1)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: ('nPV', nan)

[bhbam]

Iibrary = 'np' is working but I am using library='pd'

Here is the error

---

KeyError Traceback (most recent call last)
in
----> 1 df = tree.arrays(library ='pd')

/Applications/anaconda3/lib/python3.8/site-packages/uproot/behaviors/TBranch.py in arrays(self, expressions, cut, filter_name, filter_typename, filter_branch, aliases, language, entry_start, entry_stop, decompression_executor, interpretation_executor, array_cache, library, how)
1173 ]
1174
-> 1175 return library.group(output, expression_context, how)
1176
1177 def iterate(

/Applications/anaconda3/lib/python3.8/site-packages/uproot/interpretation/library.py in group(self, arrays, expression_context, how)
968
969 if all(isinstance(x.index, _pandas_rangeindex()) for x in arrays.values()):
--> 970 return _pandas_memory_efficient(pandas, arrays, names)
971
972 indexes = []

/Applications/anaconda3/lib/python3.8/site-packages/uproot/interpretation/library.py in _pandas_memory_efficient(pandas, series, names)
792 out = series[name].to_frame(name=name)
793 else:
--> 794 out[name] = series[name]
795 del series[name]
796 if out is None:

KeyError: ('nPV', nan)

[jpivarski]

It's being fixed in #449.

The columns of your DataFrame must have multiple levels of structure (e.g. "Event" as an outer name and "run" as an inner name; "GenPhoton1" as an outer name and "pt" as an inner name) and also a single level of structure (just "nPV"). To make them the same length, I had been using None as a filler, so that [("Event", "run"), ..., ("GenPhoton1", "pt"), ..., (nPV, None), ...] are passed to Pandas to build the MultiIndex of column names. Pandas turned the None into NaN, and then it couldn't look up the key because that key wasn't equal to itself. So I changed the filler to "" (the empty string).

However, it means that you'll have to access the "nPV" column as ("nPV", "").

The error itself had nothing to do with reading, which was indicated by the fact that library="np" worked.

[bhbam]

Thanks for the explanation. Now it makes lots of sense for me.
Sorry I am again throwing some questions, I am totally new to pandas and uproot
df1 = tree.arrays(filter_name='nPV', library ='pd')
This command separately worked fine. Read nPV branch of root file to pandas data frame.

df2= tree.arrays(filter_name=['Event','GenPhoton1','GenPhoton2','Photon1','Photon2','Vertex0','PrimartVertex','BeamSpot'],library ='pd')
This command reads all remaining branches branches.

df=pd.concat([df2, df1], axis=1)
Finally with this command I am able to join to data set to get final data frame.

I can tell from the data frame, all other quantities have multi-indexing but nPV have single. Can I change double index quantities into single for example ('Event', 'run')->'Event_run'. I saw this type of indexing structure in root_pandas but it is now outdated. Is there any way we can apply same analogy here and read all root branches in pandas data frame? OR is there any better way to do this?

[jpivarski]

You can do "nPV" and "everything but nPV" because the bug (before PR #499) is in merging two-level column names with one-level.

Uproot creates DataFrames in one way with the expectation that if you need a different DataFrame format, you can use Pandas to manipulate it. Providing more flexibility on Uproot's side would complicate the parameterization: library="pd" is one function argument, if we wanted to parameterize it, we'd have to add more function arguments that only apply to the Pandas case. That could get confusing.

I don't know if there's a fancy way to convert ("Event", "run") column names into "Event_run", but it could be done with a for loop/list comprehension. (The number of column names is not large; speed is not a concern.) The "Event_run" form contains less information than the ("Event", "run") form, since names or subnames that originally had underscores in them can no longer be distinguished. That's why we went for the general form.

[bhbam]

Thank you for the clarifications this helps lots.