Pad awkward array with nested lists, and changing column names

[vischia]

Hi,

I need to read a tree with 39 columns with a mix of int, bool, float, and lists of ints and floats, and then I need to pad everything to the maximum length of any of the dimensions. Finally, I have to write that in to an HDF5 file.

After a lot of struggling (it seems it is not trivial to mix data types like this), I arrived to a solution.

Now I would like to extract three columns from my padded array, and change their names. Before I was forced to use awkward arrays instead of pandas dataframes by the procedure above, I was doing

my_columns = branches[['name1', 'name2', 'name3']].copy()
my_columns.columns = ['newname1', 'newname2', 'newname3']

However, now that branches is an awkward array the above code doesn't work anymore (no member copy).
I tried variations of:

my_columns = copy.deepcopy(branches[['name1', 'name2', 'name3']])
my_columns.columns = ['newname1', 'newname2', 'newname3']

but I keep getting complaints that named arrays are unmodifiable, so I cannot change column names.

How can I change column names?

Thank you very much in advance!

Cheers,
Pietro

[vischia]

P.S. afaik the relative documentation is here, where it says the content does not exist yet.

[vischia]

Sorry for the spam.

I solved by creating a new thing

my_columns = {'newname1': branches['name1'], 'newname2': branches['name2'], 'newname3': branches['name3']}

It is somehow inconvenient though, it would be great if down the line there were dedicated functions that trivially do this.

[jpivarski]

You've created a Python dict of arrays; if you wanted them to be columns of a new Awkward Array, you would use ak.zip.

>>> array = ak.Array([{"x": 1, "y": 2, "z": 3}, {"x": 1.1, "y": 2.2, "z": 3.3}])
>>> array.show(type=True)
type: 2 * {
    x: float64,
    y: float64,
    z: float64
}
[{x: 1, y: 2, z: 3},
 {x: 1.1, y: 2.2, z: 3.3}]
>>> array2 = ak.zip({"a": array["x"], "b": array["y"], "c": array["z"]})
>>> array2.show(type=True)
type: 2 * {
    a: float64,
    b: float64,
    c: float64
}
[{a: 1, b: 2, c: 3},
 {a: 1.1, b: 2.2, c: 3.3}]

I hear what you're saying about the inconvenience of listing each of the columns individually. Since the above can be done manually, it can also be automated, for instance by using Python dict comprehensions. There isn't a standard function for that (and can't be) because it would depend on how you're getting your columns. The specific transformation you want to do is different from the transformation someone else wants to do. We're relying on you to use Python tools like dict comprehensions, for loops, and such to generalize any metadata management, such as changing column names. (That's why all of the examples show how to do this manually.)

[vischia]

Thank you very much, creating a new awkward array seems indeed the way to go. However, if I use ak.zip then another package complains that it cannot actually recognize the new names of the columns, whereas it does not complain if I pass the dict of arrays, so I think that for my use case I can just live with the dict of arrays.

[jpivarski]

If another library needs a dict, then make a dict and not ak.zip. (See how we can't provide a canned function? :))

[jpivarski]

I moved this to the Awkward Array repo, since this question is about manipulating arrays, not getting data from Uproot.

You mention padding arrays. You have a solution, but in case it's useful, the usual way is through ak.pad_none and ak.fill_none.

Let's leave these Discussions open, too, so that they'll be helpful to more people in the future.

[vischia]

Ultimately I used padded_branches=ak.to_dataframe(branches), which seems to do it automatically.
I had tried through ak.pad_none and ak.fill_none, but I got in trouble when defining the maximum length (code complaining that num() was acting on an axis of unsuitable length, or complaining that .count does not exist, at various stages of desperation).