How to compare awkward arrays for equality

[amoschoomy]

Hello, I need help to compare two awkward arrays for equality,

>       assert anndata.obsm["airr"] == anndata.obsm['airr']

tests\test_io.py:313: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
env\Lib\site-packages\awkward\_operators.py:50: in func
    return ufunc(self, other)
env\Lib\site-packages\awkward\highlevel.py:1423: in __array_ufunc__
    return ak._connect.numpy.array_ufunc(ufunc, method, inputs, kwargs)
env\Lib\site-packages\awkward\_connect\numpy.py:425: in array_ufunc
    out = ak._broadcasting.broadcast_and_apply(
env\Lib\site-packages\awkward\_broadcasting.py:1037: in broadcast_and_apply
    out = apply_step(
env\Lib\site-packages\awkward\_broadcasting.py:1016: in apply_step
    return continuation()
env\Lib\site-packages\awkward\_broadcasting.py:984: in continuation
    return broadcast_any_list()
env\Lib\site-packages\awkward\_broadcasting.py:632: in broadcast_any_list
    outcontent = apply_step(
env\Lib\site-packages\awkward\_broadcasting.py:1016: in apply_step
    return continuation()
env\Lib\site-packages\awkward\_broadcasting.py:984: in continuation
    return broadcast_any_list()
env\Lib\site-packages\awkward\_broadcasting.py:697: in broadcast_any_list
    outcontent = apply_step(
env\Lib\site-packages\awkward\_broadcasting.py:1016: in apply_step
    return continuation()
env\Lib\site-packages\awkward\_broadcasting.py:988: in continuation
    return broadcast_any_record()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def broadcast_any_record():
        if not options["allow_records"]:
>           raise ValueError(f"cannot broadcast records{in_function(options)}")
E           ValueError: cannot broadcast records in equal
E           
E           This error occurred while calling
E           
E               numpy.equal.__call__(
E                   <Array [[{c_call: 'IGHM', ...}, ...], ...] type='10 * var * {c_call...'>
E                   <Array [[{c_call: 'IGHM', ...}, ...], ...] type='10 * var * {c_call...'>
E               )

env\Lib\site-packages\awkward\_broadcasting.py:474: ValueError

This is the error that I get even when I try to perform equality comparison on the same object

[jpivarski]

The reason you get an error message is because binary operators such as + and == are not allowed on record types. The motivation was so that wrong interpretations wouldn't happen by accident. Specifically, people had 3D vectors represented by records, sometimes with fields x, y, z and sometimes with field rho, phi, theta. The naive + operation makes a lot of sense on x, y, z, but not on rho, phi, theta. By preventing + (etc.) on generic records, behavioral overloads, such as the ones in the Vector package, could be overlaid to convert these vectors correctly, and the error message was an indication that the overload was forgotten or didn't get applied correctly.

But == and != are special binary operators. Perhaps they should be allowed? Even if so, you might be surprised by the output:

ak.Array([{"x": 1, "y": 2}, {"x": 3, "y": 4}]) == ak.Array([{"x": 1, "y": 2}, {"x": 3, "y": 4444}])

would result in

ak.Array([{"x": True, "y": True}, {"x": True, "y": False}])

rather than

ak.Array([True, False])

as you might be expecting. Is that the case—are you expecting == on records to replace the whole record with a boolean true or false?

---

@agoose77, this could be implemented as a default behavior for records, for == and != only. It would just need to check the equality of all of its fields, applying ak.all to any nested lists until they're flat at the level of the record.

If the two sides of == have a different set of fields, either an error or false would be the appropriate result. Error is safer, but false would always work in unions.

I think it could be implemented in such a way that nested records would perform this operation recursively, so that the boolean result would always come in at the top-most record.

I could take a look at implementing that, but first, what do you think? Do you think this sounds reasonable? Since we'd be replacing an error condition with a meaningful result, we could do it without a deprecation cycle.

[agoose77]

This indeed is something we should be able to do (c.f. #2367). I'm in favour of implementing this.

[amoschoomy]

I think the latter makes much more sense to me but thanks anyway for the reply and glad to see updates to this being considered