Non-Python implementations of Awkward Array (e.g. Rust, Go)

[rw]

I'm interested in using this library (when it's ready) from Rust and Golang.

Are there plans for this? I think it would mean directly using the C++ headers, and skipping anything that uses Python.

[jpivarski]

Actually, all the array transformations are put under an extern "C" interface ("layer 4") precisely so that other language implementions can be written. The FFI only expresses C concepts, not C++. The Numba implementation can be seen as the first language to use this, beyond C++.

While possible, it should be clear that this is a big task. A new "layer 3" in Rust or golang would be responsible for defining array types, allocating, tracking ownership, and deleting array memory. When Rust or golang call the "layer 4" functions, the arrays already have to be made—the shared code only fills preallocated arrays.

If Rust or golang have C++ bindings (i.e. not FFI, but something more specific, an equivalent of pybind11 or Cxx.jl), then less work would have to be done. You could put a Rust or golang wrapper over "layer 3" the same way that it's done in Awkward with pybind11.

In most languages, though, that's not an option. FFI is there common denominator, not any C++ interface.

[jpivarski]

Oh, a couple of other things—while there would be less to write if building a new "layer 2" (on top of C++, equivalent to pybind11 or Cxx.jl) than a new "layer 3" (on top of the extern "C" interface), you would have to link the target language's reference counts with std::shared_ptr reference counts. The price of inheriting the array-ownership code is having to interface with it. Arrays can be created by the target language (NumPy in the Python case) or by Awkward Array (in C++ or in Numba), and both types need to deleted exactly once.

The other thing is that you don't need to wait for Awkward to be ready. It's not ready for general data analysis users, but it's sufficiently well-defined to start building new interfaces. We're at the stage where all the array types have been defined for general use (though a few more will be needed for interface with ROOT files and Arrow buffers) and we're writing functions on all of these types. __getitem__ was a major project, but it's finished, and it's a good starting point for a new language wrapper. (It's where I started when adding Numba.)

We can discuss the particulars, but there's no need to wait.

[rw]

@jpivarski This is great news! Thanks for planning this out and giving so much detail.

For Rust, I'd probably try to use the new cxx library to do type-safe interop between C++ and Rust.

[jpivarski]

I'd like to see how this works out! To help with your planning, I can point out exactly which parts of C++ memory management we use.

All of the arrays of primitive numbers and booleans, that which is directly shared with NumPy, are in std::shared_ptr<T> for integer types that have to be interpreted on the C++ side (only int8_t, int32_t, uint32_t, and int64_t now and in the foreseeable future) or std::shared_ptr<void> (for the numerical data that don't need to be interpreted on the C++ side, like the contents of NumpyArray). If these buffers are created on the C++ side, we use an array_deleter and if they're created on the Python side, we use a pyobject_deleter, which connects C++ reference counts to Python reference counts.

It's often the case that multiple array nodes use the same buffer, so they share the shared_ptr. Since they might be viewing a subinterval of that buffer, they have an offset and length, but they have to share the same base pointer to properly count references.

Integer arrays that the C++ needs to understand to do its work (e.g. the starts and stops in a ListArray) all have an Index type. Indexes are usually passed around by value, but they contain a std::shared_ptr to the actual buffer, so really we're just copying that reference count table, offset and length.

All leaves of the nested structure are either NumpyArray (intended for Python, but can be used in C++) or RawArray (only intended for C++; a templated and header-only class). These carry std::shared_ptr<void> or std::shared_ptr<T> but are themselves carried around as std::shared_ptr<Content>, as all array nodes are.

All array nodes are passed around as std::shared_ptr<Content> so that we can mix them dynamically at runtime. (The Content class is full of virtual methods.) Arbitrary data structures do not require recompilation.

Each of these nodes can optionally carry Identities, which are two-dimensional arrays fulfilling a similar role to Pandas indexes. They're optional (the std::shared_ptr<Identities> can point to nullptr) and only a few functions will use them (SQL-like joins), but all functions pass them through. An "identity" is a unique marker on all logical list/record/numbers. There are only two flavors of Identities: 32-bit and 64-bit; the specialization and possibility of nullptr are the reasons it's passed by pointer.

Each array node also passes through a string → JSON mapping that indicates how it should be interpreted in the high-level interface. For instance, internally, strings are just ListOffsetArrays of 8-bit NumpyArrays, but they carry a parameter "__class__": "\"string\"" that the high-level view uses to present them as strings, rather than lists of numbers.

The identities and parameters are the only mutable attributes that the array nodes have. All the rest are immutable, and the data in the arrays themselves should be regarded as immutable. Actually, there's one exception to that: RecordArray will have mutable contents as requested by #103. It hasn't happened yet, but it's coming. So three—three immutable attributes.

To make structures, we build them up with an append-only FillableArray, which is dynamically typed. It discovers the type of the data as it's being filled. Faster filling when the type is known or partially known in advance will be special cases, not filled with FillableArray (see root.h for a first example of filling data when it's known to be vector^N<number>).

std::unique_ptr is not used anywhere. I thought at first that I might be using it, but the similarity of std::shared_ptr to the Python memory model made it much more convenient. One of the planned array node types, RedirectArray, will be using std::weak_ptr, but that's about it. I use only a few memory features of C++, most of them old (not even std::move), because all of the performance concerns are in "layer 4," under the extern "C" interface. The idea is that a typical dataset would have only a few dozen array nodes at most (representing the array type), but billions of array elements (the array values). All of the loops in C++ (except in FillableArray) walk over array type and not array values.

I don't know how well this memory model maps onto Rust, however: maybe some kind of Box? It should map onto golang well, since that's a garbage collected language.

[jpivarski]

On second thought, __setitem__ is not going to be mutable (in development in #107), and setidentities and setparameters will eventually become immutable, too. The high-level ak.Array in Python is a thin wrapper around everything, and it can have a mutable __setitem__ that replaces its self._layout in place so that it looks to data analysts like they're changing these things in place, but in C++ I'm moving toward complete immutability.

Since these nodes are lightweight handles pointing to large arrays (shared wherever possible), there's no performance advantage to modifying things in place. There can be a conceptual advantage to high-level users ("I just want to add this one field!"), but that mutability can be concentrated in a layer that's easy to debug.

[jpivarski]

This isn't any less relevant, but it's not a to-do item and I'd like to close it to see the list of remaining work more clearly.

If you're planning to build other-language versions on top of either the C layer or the C++ layer, you don't have to wait for me. Awkward 1.0 is "doneish" right now (and it will become more "doneish" as time goes on).