C++ error when pyarrow>8.0.1 installed in environment

[mattjbr123]

Getting a strange error when trying to run it in parallel, which I haven't seen before...

terminate called after throwing an instance of 'std::out_of_range'
  what():  basic_string::erase: __pos (which is 16) > this->size() (which is 15)

which is a C++ error in a basic erase function which is trying to erase index 16 of a string which doesn't exist because the string ends at index 15.
It's not clear where this error is happening, the rest of the text surrounding this line seems unhelpful.

It is something to do with my python environment - a particular version of a particular package must be breaking things as the original workflow for the G2G data runs fine in my old environment, but not in the new.

To figure out which package is the culprit I will clone the old environment and gradually update the key packages in it, until it breaks, at which point I can look and see what dependencies the new package required and repeat the process with these dependencies until I have it.

The problematic packages seem to be

• pyarrow
• protobuf
• grpcio
  Pyarrow depends on the other two packages.

Fortunately, older versions of the packages seem to work, specifically:

• pyarrow 8.0.1
• protobuf 3.20.3
• grpcio 1.47.1 or 1.46.4

Any pyarrow version above 8.0.1 seems to reproduce the error. This limits the environment to python<=3.10 which isn't ideal, but at least it works for now and I can proceed with the rest of the project...

[mattjbr123]

Have posted a query in the pangeo discourse forums to see if anyone else there might be able to shed more light on the problem!
https://discourse.pangeo.io/t/strange-error-using-pangeo-forge-recipes-apache-beam-in-parallel/4540
Some places to start debugging are suggested 👌