Add linux aarch64 support #386
Comments
|
Thank you for your answer. I'll follow the docs build instructions! |
|
The build instructions do not work out of the box, especially as it points to ever changing Rust nightly versions. Lacking official Linux aarch64 support hinders the adoption of this library, as it is basically the version that every developer using a mac M1 and docker will end up needing for local testing. Could we have better instructions/support for that? |
|
Hi @Sirsirious , thanks for the comment. For mac M1 we do have precompiled wheel file support. You can just run |
|
Hi @wangxiaoying, I am a big fan of this library, use it a lot for EDA in Jupyter Notebooks with Polars with big datasets >= 200GB. I want to now use it in a production env on Airflow! Unfortunately my whole team uses mac M1 and every team member builds a local a local dev airflow environment from a docker-compose file on a shared github repo! I currently use polars but read in the data using pd.read_sql_query and then transform the data to a polars dataframe with pl.from_pandas! I would prefer to use connectorX with Polars! Below is the error log trying to install ConnectorX. Currently building from source for the whole team is not an option. Is there any update on the support for aarch64 support? |
|
I am also interested if this is on the roadmap, linux ARM64 support is very important for some of the work we are doing and it would be very nice if this feature was officially supported. |
This works great for local mac M1 development purposes, but when using Docker on M1 it does not work well since Docker containers on M1 are (ideally) Linux aarch64/arm64. (You can have Docker on M1 emulate x86_64 / amd64, but it's painful...) |
|
Hi, my team has been attempting to retrieve a connector-x python wheel on aarch64 for an EC2 instance, and we've succeeded in building a # Retrieving the base image from manylinux 2014 because that's what connectorx
# 0.3.1 was built on.
FROM quay.io/pypa/manylinux2014_aarch64
# Installing devel dependencies
RUN yum install -y epel-release
RUN yum install -y mysql-devel freetds-devel postgresql-devel
# Creating and changing to a new directory
RUN mkdir /wheeler
WORKDIR /wheeler
# Installing and setting up rust
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="$PATH:/root/.cargo/bin"
# Installing just through cargo
#RUN "$HOME/.cargo/bin/cargo" install just
RUN cargo install just
# Installing python3.9.6 from source
RUN yum install -y wget
RUN wget https://www.python.org/ftp/python/3.9.6/Python-3.9.6.tgz
RUN tar -xvf Python-3.9.6.tgz
RUN cd Python-3.9.6 && ./configure --enable-optimizations
RUN cd Python-3.9.6 && make install
RUN pip3.9 install poetry
# Cloning the connectorx repo and switching to the 0.3.1 tag
RUN git clone https://github.com/sfu-db/connector-x.git
WORKDIR /wheeler/connector-x
RUN git checkout tags/v0.3.1
# Installing the nightly version of rust upon which connectorx was built
RUN cargo install nightly-2022-09-15
RUN cargo override set nightly-2022-09-15
# Installing maturin
RUN pip3.9 install maturin==0.12.1
# Building the python wheel through maturin
RUN maturin build -m connectorx-python/Cargo.toml -i python3.9 --no-sdist --release --manylinux 2014
# Copying the wheel into the host system
COPY /wheeler/connector-x/connectorx-python/target/wheels/connectorx-0.3.1-*.whl .
# The wheel should be on your system at this point.Hope this helps someone out with the build. |
|
Lovely stuff @dat-adi, thanks for sharing! Note that if you're looking for "pip install connectorx" in an arm-based Dockerfile, you can use adi's Dockerfile as a build stage in a multi-stage build to accomplish this. When naming the stage FROM quay.io/pypa/manylinux2014_aarch64 as connectorx-builderYou will be able to copy and pip install COPY --from=connectorx-builder /wheeler/connector-x/connectorx-python/target/wheels/connectorx-0.3.1-*.whl ./
RUN pip install connectorx-0.3.1-*.whl(Small side note: I had to install and configure the correct nightly version of rust the following way, as opposed to the cargo commands mentioned above) RUN rustup install nightly-2022-09-15
RUN rustup override set nightly-2022-09-15 |
|
When referring to these docs here, I'm trying to find the right rust version as stated to "search for rust". Everything above references night builds but is it now just the "stable" version of rust? |
|
Yes, just recently built from source using the above method described. Only now you don't need to specify the nightly anymore! Stable works. So you can remove these lines: RUN rustup install nightly-2022-09-15
RUN rustup override set nightly-2022-09-15 |
|
@duvenagep and @dat-adi Thanks for putting out this information. I tried building wheels using the docker specks you provided but I'm running into the issue below. Any suggestions? Do you perhaps have an updated script for version 0.3.2? |
|
Hi @vnijs, It seemed to be necessary on my end, and did not work as expected without it. Also, check whether you can retrieve a logfile or a trace. |
|
When I add nightly stuff I get. So fully the script you initially provided I don't use rust myself, just python, so I'm not sure how to get additional log/trace information here |
|
Update: Replace the cargo commands with Unfortunately I'm not sure where the wheel is now because I get the below. Any suggestions, please let me know. Thanks |
|
I was able to extract the wheel but when I tried to install it I got the below. I had hoped this would work with Ubuntu 22.04 (aarch64). Any help on next steps would be much appreciated. Below the platform I was hoping to install this wheel into: |
|
Got this working with connectorx 0.3.2, ubuntu 22.04, and python 3.11.7. See the code for the docker image below. |
|
Thanks to everyone’s help, I successfully built connectorx v0.3.3 from source on the python:3.10-slim image and M1 MacOS. Here, I’d like to share the Dockerfile content below. FROM python:3.10-slim AS builder
RUN apt-get update
RUN apt-get install -y curl
RUN mkdir /wheeler
WORKDIR /wheeler
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="$PATH:/root/.cargo/bin"
RUN rustup install 1.78.0
RUN rustup override set 1.78.0
RUN apt-get install -y git
RUN git clone https://github.com/sfu-db/connector-x.git
WORKDIR /wheeler/connector-x
RUN git checkout tags/v0.3.3
RUN pip install maturin[patchelf]==0.14.15
# Install the dependencies
RUN apt-get install -y clang build-essential libkrb5-dev
RUN maturin build -m connectorx-python/Cargo.toml -i python3.10 --release
FROM builder AS base
COPY --from=builder /wheeler/connector-x/connectorx-python/target/wheels/connectorx-0.3.3-*.whl ./
RUN pip install connectorx-0.3.3-*.whl |
Hi!
I've been really struggling to install the package in an EC2 instance that happens to have an aarch64 processor and is running ubuntu. Are there any plans to add aarch64 support for linux machines in the future?
Regards
The text was updated successfully, but these errors were encountered: