Linux ARM support

[thomasaarholt]

This is same as #237, but for linux.

Polars added linux arm support a few days ago (issue) (PR), which will also be wanted here, since Docker images running on M1 macs will be running arm architecture as well.

connectorx successfully installs in my arm docker image when I run pip install connectorx, but then fails when I try to import it:

import connectorx
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/fabrica/projects/forecast-compare/Untitled.ipynb Cell [1](vscode-notebook-cell://attached-container%2B7b22636f6e7461696e65724e616d65223a222f666f7265636173742d636f6d70617265227d/fabrica/projects/forecast-compare/Untitled.ipynb#ch0000052vscode-remote?line=0)8' in <module>
----> 1[ import connectorx

File /usr/local/lib/python3.9/site-packages/connectorx/__init__.py:3, in <module>
      ]()[1](file:///usr/local/lib/python3.9/site-packages/connectorx/__init__.py?line=0)[ from typing import Optional, Tuple, Union, List, Dict, Any
----> ]()[3](file:///usr/local/lib/python3.9/site-packages/connectorx/__init__.py?line=2)[ from .connectorx_python import read_sql as _read_sql
      ]()[5](file:///usr/local/lib/python3.9/site-packages/connectorx/__init__.py?line=4)[ try:
      ]()[6](file:///usr/local/lib/python3.9/site-packages/connectorx/__init__.py?line=5)[     from importlib.metadata import version

ModuleNotFoundError: No module named 'connectorx.connectorx_python']()

If I require it to use the binary from pypi, then it does not find a match:

pip install -U connectorx --only-binary connectorx 
ERROR: Could not find a version that satisfies the requirement connectorx (from versions: none)
ERROR: No matching distribution found for connectorx

[wangxiaoying]

Hi @thomasaarholt , thanks for the issue. The source file on pypi is not ready yet so we have removed it from our new alpha version. We are working on it.

[messense]

Feel free to ping me if you need any help.

[wangxiaoying]

Hi @messense , can you please help to check this error (build wheel for arm linux): https://github.com/sfu-db/connector-x/runs/5337702956?check_suite_focus=true ? I have no idea how to fix it. The yaml file can be found here. Thank you!

[messense]

@wangxiaoying Try add manylinux: 2014 option to maturin-action.

[messense]

This is harder than we thought since you have system dependencies which aren't present in maturin-action's cross compilation docker images(they are based on Ubuntu 20.04).

connector-x/.github/workflows/release.yml

Line 46 in fd7a61d

yum install -y openssl11-devel mysql-devel postgresql-devel freetds-devel

And

connector-x/.github/workflows/release.yml

Line 12 in fd7a61d

container: quay.io/pypa/manylinux2014_x86_64

forces x86_64 so you may need to change to use QEMU and switch to the official aarch64 manylinux docker image instead, but it's gonna be at least 6x slow on CI.

It will be much easier if we're statically link system dependencies.

[messense]

For now the easiest way forward is to use a CI with native ARM support for example https://www.drone.io/

https://docs.drone.io/pipeline/docker/syntax/platform/

[wangxiaoying]

For now the easiest way forward is to use a CI with native ARM support for example https://www.drone.io/

https://docs.drone.io/pipeline/docker/syntax/platform/

Thank you so much! I will take a look at this.

[thomasaarholt]

Has anyone taken a look at this since February? I just met this again now - installing connectorx succeeds on aarch64, but it is not importable, resulting in the same error as in the first post.

[wangxiaoying]

Hi @thomasaarholt , we are now successfully supporting apple m1 with wheel files. But still having issue to cross compile for linux arm. Currently the only way to install connector-x on linux arm is to build it from source: https://sfu-db.github.io/connector-x/install.html#build-from-source-code

[viniciusd]

Sorry for bumping this, but I find it more adequate than creating yet another issue (in addition to #186, #386, and #240).

I understand there was an attempt to build ARM releases, but it was removed at 739b8b5, but I can't really tell why.

Do you need help figuring out the ARM gh action @wangxiaoying? Is there something we can do?

[wangxiaoying]

Sorry for bumping this, but I find it more adequate than creating yet another issue (in addition to #186, #386, and #240).

I understand there was an attempt to build ARM releases, but it was removed at 739b8b5, but I can't really tell why.

Do you need help figuring out the ARM gh action @wangxiaoying? Is there something we can do?

Yes, the previous PR for supporting aarch platform was failed (more details here). We are still looking for help on this one. Unfortunately, the logs for the previous actions are already expired. But I think you can fork the repo and test the gihub action (.github/workflows/release.yml) locally. Please feel free to submit a PR.

[pangjunrong]

Hey @wangxiaoying, I've tried to get CI workflow up for Linux ARM and so far I am able to do so when I completely remove integrated-auth-gssapi from the feature list of tiberius found in the dependencies.

I've tried many different strategies (using features functionality, installing pre-requisite tools, conditional dependencies etc.) but unfortunately when cross-compiling for Linux ARM, integrated-auth-gssapi simply does not work as long as it appears in the dependency section.

I was able to build the wheel and verify it successfully using manylinux2_28 without integrated-auth-gssapi, which is based on CentOS 7, with glibc 2.28. Any older versions such as manylinux2_24 or manylinux2014 will result in compilation failure as at least 1 of the transitive dependency is not backward compatible — specifically [email protected].

So technically it is possible to get something up for Linux ARM running on more recent OSes if we were to forgo the system authentication feature for MSSQL, however given that I can't get the conditional dependencies definitions in Cargo.toml to work at the moment we may need to find some workaround so that a proper release can still include integrated-auth-gssapi for other OS & architectures.