Returning Awkward Arrays vs returning NumPy arrays from Awkward functions

[HDembinski]

What is the intended use case for ak.flatten? If it converts an arbitrarily complex awkward array into a flat 1d array, then I think it should return a numpy array, not a 1D awkward array. I wanted to call some of my array transforms, e.g.

def pt_eta(px, py, pz): ...

with ak.flatten instead of np.asarray(array.layout.content), but that still does not work in my code, since neither the type returned by ak.flatten nor by array.layout.content has the dtype property.

[HDembinski]

In other words, I don't understand why awkward arrays do not transform into numpy arrays when the result is representable as a numpy array. Requiring the developer to do the conversion explicitly with np.asarray seems not user-friendly.

[jpivarski]

The opposite—mixing Awkward-brand and NumPy-brand arrays in return values—was a problem that we wanted to fix from Awkward 0. Not every flatten would return a flat array:

>>> array = ak.Array([[[0, 1, 2], []], [], [[3, 4]], [[5], [6, 7, 8, 9]]])
>>> array
<Array [[[0, 1, 2], []], ... 5], [6, 7, 8, 9]]] type='4 * var * var * int64'>

>>> ak.flatten(array)
<Array [[0, 1, 2], [], ... [5], [6, 7, 8, 9]] type='5 * var * int64'>

>>> ak.flatten(ak.flatten(array))
<Array [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] type='10 * int64'>

Having return types (ak.Array vs np.ndarray) depend on values (the structure within the input ak.Array) makes it difficult for developers and data analysts to use a library. Knowing that you always have to np.asarray is better than knowing that you sometimes have to np.asarray. (This function, ak.to_numpy, is also ak.Array.__array__, so any API that expects an array-like should be calling __array__ to get a NumPy-brand array.)

For normal data analysis, though, you shouldn't be extracting things from layout. The node structure of the array was hidden in there for this same reason: users were confused about sometimes getting a JaggedArray, sometimes an IndexedArray, sometimes a ChunkedArray, etc. in Awkward 0. We need all of these different node types to construct the abstraction, but then we deliberately hide them all inside a single type, ak.Array, with a fixed set of properties and methods. Those properties do not (yet?) include dtype, but the rule is this: you always np.asarray(·).dtype to get the dtype. Then there's no guessing.

[HDembinski]

Sorry Jim, but you are closing issues so fast that I feel like you are ending a discussion one-sidedly.

In numpy, flatten always reduces to 1D, so if ak.flatten does not do that, then that's a break in similarity. The awkward behavior of flatten is certainly very unexpected.

[jpivarski]

I only closed this one and a duplicate (which was resolved with the requested feature, by the way). That's because in this one, you're asking for something that is way outside the design of the library—returning different Python types depending on array type was the bug we fixed with this redesign.

In all of these, you are coming from a different point of view: I think your needs are simpler. You have reason to believe that your arrays are ListOffsetArrays of one level deep. If that's always the case, it's less onerous to work with the nodes and indexes directly (including mutation) and you don't need the full abstraction. This redesign was motivated by the problems people were having with more complex cases, Earth missing values, unions, and chunking. There are good reasons to have multiple ways of expressing the same array, with and without an IndexedArray, for instance (which delays rearrangement), but those reasons are technical and would be very distracting in a data analysis. However, this is a difference in layout that could make the difference between an Awkward Array and a NumPy array.

Finally, we do make sure that any function with the same name as NumPy has the same behavior. However, NumPy doesn't have a np.flatten function. The semantics of ak.flatten (flattening only one level) come from functional programming: LISP, Scala, and Spark, for instance.

[HDembinski]

I thank you for the explanation, but I am talking about another aspect.

Firstly, you are right, numpy does not have a free function called flatten, but it has a method called flatten on each array. Whether it is a free function or a method does not really matter, right?

The primary point is what you think ak.flatten should do. In numpy, it converts an N-dimensional array to a one-dimensional array. By default the order of the items in the returned array is the order in memory, so that this operation is O(1). This is very useful in practicle, because after processing I usually reach a point where I can ignore substructures and just pass all the numbers into a histogram. This is not what ak.flatten does, which only seems to remove one layer of the nested structure instead of all of them. I am merely telling you that I didn't expect this and I find the name misleading if it does not do what the numpy equivalent does. The word "flatten" indicates removal of all substructure in my ears.

My interest in this is purely practical and based on the analysis that my group does in LHCb. I argue that this is a very useful function and it could just do the same for (some) awkward arrays. Whether ak.flatten returns a numpy array or an awkward array is a secondary point, although I don't understand why you don't return a data structure that is representable as a numpy array as a numpy array. I don't see how that can cause problems. You have a NumpyArray class in awkward, after all.

[jpivarski]

I think of methods vs free functions as being a different interface, though that's a matter of definition. Awkward Arrays don't have sum, max, etc. methods on purpose because they were interfering with the user-defined methods. NumPy's use of both methods and free functions with similar meanings is often described as a mistake, and I agree. New languages, like Julia, are going the route of only free functions, but the choice we made here was to use the distinction as a dividing line: nearly all generic functionality is in Awkward free functions, leaving most of the method-space available for user-defined functions.

So by "generalizing NumPy," we mean "generalizing a subset of NumPy's free functions." Since np.flatten isn't one of them, I didn't think ak.flatten is constrained.

The feature you want is ak.flatten with axis=None. That eliminates all structure, no matter how many levels, and eliminates all field distinctions among records. It's usually too much; that's why the default is axis=1.

One of the unpleasant consequences of generalizing NumPy is that reducers like ak.sum and ak.max must have axis=None as a default. Most of the time, we want axis=1, but we were constrained by NumPy and it complicates analysis code to have to always insert the axis=1.

Is the argument here that the default ak.flatten axis should be None?

As for its performance characteristics, ak.flatten is O(1) when it can be (e.g. ListOffsetArray), O(n) when it can't (e.g. ListArray, IndexedArray, etc.). I think I could cook up a NumPy example that can't be O(1). Here's one that must copy:

>>> base = np.arange(2*3*5).reshape(2, 3, 5)
>>> base
array([[[ 0,  1,  2,  3,  4],
        [ 5,  6,  7,  8,  9],
        [10, 11, 12, 13, 14]],

       [[15, 16, 17, 18, 19],
        [20, 21, 22, 23, 24],
        [25, 26, 27, 28, 29]]])
>>> sliced = base[::-1, ::2, 1:-1]
>>> sliced
array([[[16, 17, 18],
        [26, 27, 28]],

       [[ 1,  2,  3],
        [11, 12, 13]]])
>>> result = sliced.flatten()
>>> result
array([16, 17, 18, 26, 27, 28,  1,  2,  3, 11, 12, 13])
>>> result.base

Non-contiguous ListArrays are the moral equivalent of that.

[agoose77]

I'm just adding to this issue now that we're a little further along the line, in case new users discover this discussion.

Awkward now has ak.ravel, which overloads np.ravel via the __array_function__ mechanism. This function, like its NumPy counterpart, fully flattens the array. Given that Awkward only intends to maintain compatibility via the module API and not the array object itself, this is a reasonable level of compatibility.