-
|
I'm a little confused about the versioning of this project. What is v2? Is there an imminent set of breaking changes? And is this described somewhere? All I could find is the roadmap which only has Trying to figure out if we should be expecting things to break, and what those things might be! |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 2 replies
-
|
@jpivarski may correct me here, but if you're a high-level API user, that is, any public |
Beta Was this translation helpful? Give feedback.
-
|
What @agoose77 wrote is correct: a design goal of Awkward 2.0 is to avoid changes to the high-level interface, which includes the functions named Therefore, upgrading from version 1.x to version 2.0 will require minimal, if any, changes to data analyst user code. What's changing is the view that you see if you look at The main motivation for this refactoring (given in detail here) is to enable third-party Python libraries to be able to see more of the Awkward internals, in particular Dask. This is related to the biggest change from version 1.x to 2.0: the VirtualArray and PartitionedArray node types are being dropped in favor of an Awkward-aware Dask collection (in development here, through a collaboration with Anaconda). That is, Awkward's built-in lazy arrays are being replaced by industry-standard lazy arrays. Awkward's lazy arrays were a perpetual source of trouble (see, for example, #230, #400, #432, #479, #541, #560, #597, #603, #655, #679, #783, #865, #899, #940, #1052...), largely because it evaluates the array "when some information is needed," rather than "when the user asks for it" (Dask's interface, which requires users to type
(slide from the Dask Summit, slides here). Replacing Awkward's built-in lazy arrays with Dask is a big change for Coffea's NanoEvents, which is thoroughly virtual. The biggest part of the Awkward v1 → v2 transition will be to upgrade NanoEvents, and we'll give the Coffea team a good transition period for that work. On the plus side, it will allow Dask to be more thoroughly integrated into Coffea analyses than it currently is. Another thing that will be different from the Awkward v0 → v1 transition is that names will not change in PyPI/conda-forge: there won't be any This GitHub repo will need a new name, since scikit-hep/awkward-1.0 won't make sense anymore. scikit-hep/awkward-0.x was originally named scikit-hep/awkward-array. To be super-careful about the possibility of old links, I'm thinking the name of this repo should not become scikit-hep/awkward-array, but instead be scikit-hep/awkward, a never-before-used name. (GitHub automatically forwards links when a repo changes its name, but we don't want old links to point to the new repo, even though those old links would have to be years old. Google has already updated.) Note that arrays pickled or otherwise saved through ak.to_buffers/ak.from_buffers (e.g. this tutorial) will be interchangeable between versions 1.0.1+ and 2.0+. While they can't be loaded in the same Python process, they can be serialized and deserialized between the two versions (or sent through an Arrow buffer without touching disk, etc.). Awkward version 1.0.1, the minimum for pickle compatibility, is from Dec 14, 2020.) I'll be talking more about the pros and cons (mostly cons) of Awkward 1.x's mixed C++/Python codebase at ACAT: Lessons learned in Python-C++ integration. Short story—lots of linking issues: #209, #211, #217, #233, #281, #316, #430, #476, #483, #562, #774, #778. If you have any concerns about the performance impact of replacing C++ code with Python code,
(from a 2020 SciPy talk) was repeated with today's v1, v2, and the equivalent Python:
The main point of this performance test is to demonstrate that we're not putting the parts that scale with array size in pure Python: they're still in compiled code. Also, feature requests that would have required a whole new algorithm are being implemented in version 2.0 only instead of v1 and v2 because it's easier to write once than write and port. For instance, #838 needed the |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the very informative responses! We're looking at including support for awkward in |
Beta Was this translation helpful? Give feedback.
-
|
A bit more context—the graphic below comes from a talk I'm writing.
I've also been counting lines of code and I estimate that we're 75% done with writing v2. (Assuming that "lines of code" is proportional to time, that means the v2 green/development bar will continue for another 2 months.) During the overlap period (both v1 and v2 are yellow):
After the overlap period (v1 is gray and v2 is yellow):
Dropping Python 2.7 support will happen before the v1 → v2 transition, since that's driven by loss of tooling at the end of this year (#1010). FYI @henryiii, @nsmith-, @lgray, @ioanaif, @ianna, @agoose77, @stormiestsin |
Beta Was this translation helpful? Give feedback.
v2is foremostly a rewrite of the middle C++ layer of Awkward Array in pure Python. Originally, this C++ layer was intended to be usable by other C++ software (e.g. ROOT, FastJet), but it was later realised that it presented some challenges. Among these, integration with other tools such as Jax was made more difficult by the opaqueness of the C++ layer. In addition to fixing a design problem,v2is also being used to re-think the lazy-loading & parquet routines in Awkward. There is an on-going project to replace theVirtualArrayandPartitionedArrayfeatures with a separate third-party (dask-awkward) library that can leverage the expertise and resources behind the Dask project to scale Aw…