Problem with reading (awkward) TClonesArray of TLorentzVectors

[denehoffman]

I've refrained from asking a question about this since my understanding of the ROOT data structure is fairly limited, but I work with TTrees which contain branches that have TClonesArrays of TLorentzVectors, and they don't seem to read properly when I use the usual Uproot methods. I noticed in one of the recent pull requests (442) that something related to TClonesArrays was implemented, but I'm still having the same problem, so I was wondering if it is an issue with my usage of Uproot or if it's an issue with how my collaboration generates the TTrees.

For some context, we have a single TTree with multiple branches. However, for each event, in our branches of 4-vectors, we have a TClonesArray object that doesn't necessarily contain the same number of vectors for each event (we say there are multiple "combos" or combinations of particles which can constitute each event, and this is the way they're stored). Unfortunately, I believe this is the cause of a problem when I try to load one of these branches into an array, since I get the error:

>>> tree['Beam__X4_Measured'].array()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/behaviors/TBranch.py", line 2105, in array
    False,
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/behaviors/TBranch.py", line 3510, in _ranges_or_baskets_to_arrays
    uproot.source.futures.delayed_raise(*obj)
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/source/futures.py", line 46, in delayed_raise
    raise exception_value.with_traceback(traceback)
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/behaviors/TBranch.py", line 3461, in basket_to_array
    library,
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/interpretation/objects.py", line 141, in basket_array
    form = self.awkward_form(branch.file, index_format="i64")
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/interpretation/objects.py", line 122, in awkward_form
    self._branch.file, index_format, header, tobject_header, breadcrumbs
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/containers.py", line 312, in awkward_form
    self._model, file, index_format, header, tobject_header, breadcrumbs
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/_util.py", line 553, in awkward_form
    file, index_format, header, tobject_header, breadcrumbs
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/model.py", line 1153, in awkward_form
    file, index_format, header, tobject_header, breadcrumbs
  File "<dynamic>", line 35, in awkward_form
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/model.py", line 677, in awkward_form
    classname_decode(cls.__name__)[0]
uproot.interpretation.objects.CannotBeAwkward: TObjArray

Is there some simple thing I can do about this? I can see the data in a single event easily with

>>> tree['Beam__X4_Measured'].debug(0)
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
 12  84  67 108 111 110 101 115  65 114 114  97 121   0  64   0   1  57   0   4
---   T   C   l   o   n   e   s   A   r   r   a   y ---   @ --- ---   9 --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1   0   0   0   0   3   0   0   0  15  84  76 111 114 101 110 116 122  86
--- --- --- --- --- --- --- --- --- --- ---   T   L   o   r   e   n   t   z   V
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
101  99 116 111 114 115  16  84  76 111 114 101 110 116 122  86 101  99 116 111
  e   c   t   o   r   s ---   T   L   o   r   e   n   t   z   V   e   c   t   o
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
114  59  52   0   0   0   4   0   0   0   0   1  64   0   0  60   0   4   0   1
  r   ;   4 --- --- --- --- --- --- --- --- ---   @ --- ---   < --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   0   0   0   3   0   0   0  64   0   0  36   0   3   0   1   0   0   0   0
--- --- --- --- --- --- --- ---   @ --- ---   $ --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  3   0   0   0  63 139 127  19 184  78 153 110 191 177  75  31 139  25  47 220
--- --- --- ---   ? --- --- --- ---   N ---   n --- ---   K --- --- ---   / ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
 64  77 248  15 190 160 129 246 192  38  67  41 156 187  69 128   1  64   0   0
  @   M --- --- --- --- --- --- ---   &   C   ) --- ---   E --- ---   @ --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
 60   0   4   0   1   0   0   0   0   3   0   0   0  64   0   0  36   0   3   0
  < --- --- --- --- --- --- --- --- --- --- --- ---   @ --- ---   $ --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  1   0   0   0   0   3   0   0   0  63 139 127  19 184  78 153 110 191 177  75
--- --- --- --- --- --- --- --- ---   ? --- --- --- ---   N ---   n --- ---   K
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
 31 139  25  47 220  64  77 248  15 190 160 129 246 192   8 228 140 178 237  22
--- --- ---   / ---   @   M --- --- --- --- --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   1  64   0   0  60   0   4   0   1   0   0   0   0   3   0   0   0  64   0
--- ---   @ --- ---   < --- --- --- --- --- --- --- --- --- --- --- ---   @ ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0  36   0   3   0   1   0   0   0   0   3   0   0   0  63 139 127  19 184  78
---   $ --- --- --- --- --- --- --- --- --- --- --- ---   ? --- --- --- ---   N
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
153 110 191 177  75  31 139  25  47 220  64  77 248  15 190 160 129 246 192   9
---   n --- ---   K --- --- ---   / ---   @   M --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
 88  79  18 237  22   0   1  64   0   0  60   0   4   0   1   0   0   0   0   3
  X   O --- --- --- --- ---   @ --- ---   < --- --- --- --- --- --- --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
  0   0   0  64   0   0  36   0   3   0   1   0   0   0   0   3   0   0   0  63
--- --- ---   @ --- ---   $ --- --- --- --- --- --- --- --- --- --- --- ---   ?
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
139 127  19 184  78 153 110 191 177  75  31 139  25  47 220  64  77 248  15 190
--- --- --- ---   N ---   n --- ---   K --- --- ---   / ---   @   M --- --- ---
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
160 129 246  64  20  26 208  70 137 117   0
--- --- ---   @ --- --- ---   F ---   u ---

So this event appears to have a TClonesArray with two entries, or two combos, but other events might have one, or three, or more. Originally I thought my problem was just something about Uproot not playing well with TClonesArrays, but now I'm fairly certain it has to do with the arrays having different sizes. I'm admittedly out of my depth here, if there's no solution, or no easy solution to this, please let me know.

Edit: For those curious, I there is a simple method (in ROOT or PyROOT) for flattening these structures such that each branch contains one event per combo, which is simply looping over all events, and sublooping over each combo and writing a new event to a new branch for each combo. If there's a way to do this using uproot, that would also be of interest to me.

[jpivarski]

Sorry that I didn't get back to this earlier, but maybe try library="np"? Since this is not strictly numerical data, that NumPy array would have dtype="O" and it would effectively be a Python list. Iteration over it would run at Python speeds, but slow access is better than no access.

The error message is saying that it can't convert the data into an Awkward Array. Your array contains a TObjArray, which is a dynamically typed list: it can contain any TObject; you don't know what types are in it until you read it. Awkward Arrays have to be statically typed so that we can split up the data into columnar buffers, which requires knowing which fields all nested records contain. Maybe I need to expand upon that error message.

I'm not guaranteeing that it will work with library="np", since TClonesArray encodings are full of exceptions. (TClonesArray was early ROOT; the patterns settled down and got more regular later.) But it's worth a try. (Digging any deeper into it would require a copy of the file.)

[denehoffman]

It's a tiny file relatively, only ~20Mb I think. I don't think it's nested at all like it's showing here, but I don't have a firm understanding of how this data file was created in the first place, since I work with events post-construction. I was able to get (most of?) the stack trace using the execption.__traceback__ object:

File "demo.py", line 10, in <module>
    a = branch.array(library='np')
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/behaviors/TBranch.py", line 2105, in array
    False,
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/behaviors/TBranch.py", line 3510, in _ranges_or_baskets_to_arrays
    uproot.source.futures.delayed_raise(*obj)
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/source/futures.py", line 46, in delayed_raise
    raise exception_value.with_traceback(traceback)
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/behaviors/TBranch.py", line 3461, in basket_to_array
    library,
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/interpretation/objects.py", line 159, in basket_array
    self._model, branch, context, byte_offsets, data, cursor_offset
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/interpretation/objects.py", line 595, in to_numpy
    output[i] = self[i]
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/interpretation/objects.py", line 616, in __getitem__
    self._branch,
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/containers.py", line 321, in read
    return cls.read(chunk, cursor, context, file, selffile, parent)
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/model.py", line 1310, in read
    chunk, cursor, context, file, selffile, parent, concrete=concrete
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/model.py", line 821, in read
    self.read_members(chunk, cursor, context, file)
  File "<dynamic>", line 7, in read_members
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/model.py", line 821, in read
    self.read_members(chunk, cursor, context, file)
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/models/TObjArray.py", line 56, in read_members
    chunk, cursor, context, file, self._file, self._parent
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/deserialization.py", line 220, in read_object_any
    bcnt = numpy.int64(cursor.field(chunk, _read_object_any_format1, context))
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/source/cursor.py", line 215, in field
    return format.unpack(chunk.get(start, stop, self, context))[0]
  File "/home/nhoffman/.local/lib/python3.6/site-packages/uproot/source/chunk.py", line 402, in get
    self._source.file_path,

The last line isn't a typo, and looking at that line in chunk.py, it looks like the error is a DeserializationError.

[jpivarski]

The proximate cause of the error is that while Uproot was interpreting bytes of data in the chunk that it had extracted from the file, it ran to the end of the chunk of data. If it hadn't been limited to this pre-assigned chunk, it probably would have kept going and run out of data at the end of the file. That's what the last two lines are: a Cursor trying to get a field of data (an integer bcnt, "byte count") from one step beyond the end of the Chunk. Since this is coming from a TTree, the Chunk is the entire TBasket.

I was thinking this might be a runaway process, resulting in a StackOverflow that got caught and reissued as a DeserializationError. You've shown that it's an explicit DeserializationError. It's probably still a runaway process, but it ran out of data before it ran out of stack.

I've downloaded the file and will take a look at it.

[jpivarski]

It's resolved in PR #467.

It was the case that the apparent deep nesting was because it was walking over the same bytes repeatedly, thinking it was finding objects within objects. The real problem (ultimate cause!) was that we just didn't have any implementation of non-split TClonesArrays. All the examples of TClonesArray I've seen have been split, with each field of the nested objects in a different branch. I think your file could have been written that way, but wasn't.

Usually, we get the data layout of a class from its TStreamerInfo—that's how we can deal with the thousands of classes in the ROOT codebase and user-defined classes outside of C++, without a library of code with pre-written deserializers. Occasionally, the TStreamerInfo's prescription does not match how the class was actually serialized, sometimes because somebody overrode the standard streamer with a customized one for performance, and sometimes because the class in question predated the adoption of TStreamerInfo (around ROOT 3, year 2000). TClonesArray is probably one of those old ones.

TClonesArray's custom reading code does a lot of strange things. Its TStreamerInfo says that it's laid out like a subclass of TObjArray with no other data, but it's nothing like that at all. The TObjArray code was able to read the name "TLorentzVectors" (plural) because they both start with TObject and an fName string, but after that, they differ wildly. TClonesArray follows the fName with a string containing the class name and version of the contained objects, and it parses that string to get the version number (here in C++ and here in Python)! Then TClonesArray also has a "do you exist?" byte before each object, while TObjArray goes through a more generic routine that could allow for missing data (and arbitrarily typed data).

So the unsplit TClonesArray is indeed weird; I'm surprised this has never come up before.

[jpivarski]

The tests finished, so I merged the PR into Uproot's main branch (on GitHub; no release to PyPI yet).

[denehoffman]

Thank you so much, I really appreciate it, this is going to save me a ton of time (and sweat and tears)!