[fix][build] Fix pulsar-client-python installation on ARM arch

[nodece]

Motivation

The [email protected] is installed by apk, and [email protected] requires grpcio>=1.60.0, which causes the grpcio to be reinstalled by pip, but the grpcio doesn't provide the arm64 wheel, and then pip3 will take a long time to compile the grpcio locally, please see job for detailed compilation information.

Contexts:

• [improve][build] pip with --prefer-binary to reduce image build time #22613
• Set grpcio minimum version to 1.59.3 so that Alpine py3-grpcio can be used pulsar-client-python#211
• https://lists.apache.org/thread/cknro58cty8xjpf8zsb2p02flw6wqs2h

Modifications

• pip install with --only-binary option avoids compiling the grpcio locally, so the python-deps stage is unnecessary
• Use independent installation instead of pulsar-client[all]

Verifying this change

Verified locally and works fine. After this PR has been merged, I will add a job to verify the multi-arch image.

Collecting pulsar-client==3.4.0 (from -r /requirements.txt (line 2))
  Downloading pulsar_client-3.4.0-cp311-cp311-musllinux_1_1_aarch64.whl.metadata (1.0 kB)
Collecting protobuf<=3.20.3,>=3.6.1 (from -r /requirements.txt (line 4))
  Downloading protobuf-3.20.3-py2.py3-none-any.whl.metadata (720 bytes)
Requirement already satisfied: grpcio>=1.59.3 in /usr/lib/python3.11/site-packages (from -r /requirements.txt (line 5)) (1.59.3)
Collecting apache-bookkeeper-client>=4.16.1 (from -r /requirements.txt (line 6))
  Downloading apache_bookkeeper_client-4.16.5-py2.py3-none-any.whl.metadata (2.1 kB)
Collecting prometheus_client (from -r /requirements.txt (line 7))
  Downloading prometheus_client-0.20.0-py3-none-any.whl.metadata (1.8 kB)
Collecting ratelimit (from -r /requirements.txt (line 8))
  Downloading ratelimit-2.2.1.tar.gz (5.3 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting fastavro>=1.9.2 (from -r /requirements.txt (line 10))
  Downloading fastavro-1.9.4-cp311-cp311-musllinux_1_1_aarch64.whl.metadata (5.5 kB)
Collecting certifi (from pulsar-client==3.4.0->-r /requirements.txt (line 2))
  Downloading certifi-2024.2.2-py3-none-any.whl.metadata (2.2 kB)
Requirement already satisfied: setuptools>=34.0.0 in /usr/lib/python3.11/site-packages (from apache-bookkeeper-client>=4.16.1->-r /requirements.txt (line 6)) (68.2.2)
Requirement already satisfied: six>=1.10.0 in /usr/lib/python3.11/site-packages (from apache-bookkeeper-client>=4.16.1->-r /requirements.txt (line 6)) (1.16.0)
Collecting pytz (from apache-bookkeeper-client>=4.16.1->-r /requirements.txt (line 6))
  Downloading pytz-2024.1-py2.py3-none-any.whl.metadata (22 kB)
Collecting pymmh3>=0.0.5 (from apache-bookkeeper-client>=4.16.1->-r /requirements.txt (line 6))
  Downloading pymmh3-0.0.5-py2.py3-none-any.whl.metadata (822 bytes)
Downloading pulsar_client-3.4.0-cp311-cp311-musllinux_1_1_aarch64.whl (5.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 7.9 MB/s eta 0:00:00
Downloading protobuf-3.20.3-py2.py3-none-any.whl (162 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.1/162.1 kB 6.8 MB/s eta 0:00:00
Downloading apache_bookkeeper_client-4.16.5-py2.py3-none-any.whl (62 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.4/62.4 kB 4.0 MB/s eta 0:00:00
Downloading prometheus_client-0.20.0-py3-none-any.whl (54 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.5/54.5 kB 3.6 MB/s eta 0:00:00
Downloading fastavro-1.9.4-cp311-cp311-musllinux_1_1_aarch64.whl (3.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 9.6 MB/s eta 0:00:00
Downloading pymmh3-0.0.5-py2.py3-none-any.whl (7.4 kB)
Downloading certifi-2024.2.2-py3-none-any.whl (163 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 163.8/163.8 kB 7.0 MB/s eta 0:00:00
Downloading pytz-2024.1-py2.py3-none-any.whl (505 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 505.5/505.5 kB 8.1 MB/s eta 0:00:00
Building wheels for collected packages: ratelimit
  Building wheel for ratelimit (pyproject.toml): started
  Building wheel for ratelimit (pyproject.toml): finished with status 'done'
  Created wheel for ratelimit: filename=ratelimit-2.2.1-py3-none-any.whl size=5894 sha256=a1a1765d8022acfe8ae03c938830b6f7832df995875cb5f57ad81b53e8ff8d9b
  Stored in directory: /root/.cache/pip/wheels/ee/d5/e5/8fbffe089140fb498987b7709becf861086daace105d243475
Successfully built ratelimit
Installing collected packages: ratelimit, pytz, pymmh3, protobuf, prometheus_client, fastavro, certifi, pulsar-client, apache-bookkeeper-client
Successfully installed apache-bookkeeper-client-4.16.5 certifi-2024.2.2 fastavro-1.9.4 prometheus_client-0.20.0 protobuf-3.20.3 pulsar-client-3.4.0 pymmh3-0.0.5 pytz-2024.1 ratelimit-2.2.1

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[lhotari]

LGTM

[lhotari]

Error: Failed to execute goal io.fabric8:docker-maven-plugin:0.43.3:build (default) on project pulsar-docker-image: Unable to build image [apachepulsar/pulsar] : "COPY failed: no source files were specified" -> [Help 1]

Looks like the HEREDOC syntax isn't supported in the GitHub Actions VM.

[lhotari]

It might require setting DOCKER_BUILDKIT=1 environment variable in GitHub Actions.

[lhotari]

Another advice is to prefix the Dockerfile with # syntax=docker/dockerfile:1. (reference, another reference)

[coderzc]

@nodece Can you take a look as soon as possible, because this PR is marked as release/blocker

[nodece]

/pulsarbot rerun-failure-checks

[nodece]

I'm using podman to build the Pulsar image, it works fine. The docker may not support HEREDOC in the GitHub Actions.

Use echo -e instead of HEREDOC.

[nodece]

2024-05-17T12:01:21.386067417Z 2024-05-17T12:01:21,385+0000 [function-timer-thread-102-1] INFO  org.apache.pulsar.functions.runtime.process.ProcessRuntime - Started process successfully
2024-05-17T12:01:21.503839370Z Traceback (most recent call last):
2024-05-17T12:01:21.503867532Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 40, in <module>
2024-05-17T12:01:21.503947785Z     import server
2024-05-17T12:01:21.503974154Z   File "/pulsar/instances/python-instance/server.py", line 26, in <module>
2024-05-17T12:01:21.504117252Z     import grpc
2024-05-17T12:01:21.504140016Z   File "/usr/lib/python3.11/site-packages/grpc/__init__.py", line 22, in <module>
2024-05-17T12:01:21.504266741Z     from grpc import _compression
2024-05-17T12:01:21.504291218Z   File "/usr/lib/python3.11/site-packages/grpc/_compression.py", line 20, in <module>
2024-05-17T12:01:21.504427333Z     from grpc._cython import cygrpc
2024-05-17T12:01:21.504466465Z ImportError: Error loading shared library libre2.so.11: No such file or directory (needed by /usr/lib/python3.11/site-packages/grpc/_cython/cygrpc.cpython-311-x86_64-linux-musl.so)

No libre2.so..., let me try to install the py3-grpcio in the last build stage, which is from py3-grpcio.

Remove the python-deps stage, it is unnecessary.

[lhotari]

I'm using podman to build the Pulsar image, it works fine. The docker may not support HEREDOC in the GitHub Actions.

Use echo -e instead of HEREDOC.

I provided the links that explained what needs to be done to support the syntax. It would be great to start using HEREDOC syntax. Adding # syntax=docker/dockerfile:1 as the first line in the Dockerfile and setting DOCKER_BUILDKIT=1 in the environment would most likely address the issue.

[nodece]

I'm using podman to build the Pulsar image, it works fine. The docker may not support HEREDOC in the GitHub Actions.
Use echo -e instead of HEREDOC.

I provided the links that explained what needs to be done to support the syntax. It would be great to start using HEREDOC syntax. Adding # syntax=docker/dockerfile:1 as the first line in the Dockerfile and setting DOCKER_BUILDKIT=1 in the environment would most likely address the issue.

It sounds like I should have enabled both configurations, and when I enabled one configuration it was invalid(this idea has been tried).

Use echo -e is simpler, and we also need to move the installation command of the pulsar-client-python to the finally stage to fix the libre.so issue.

[nodece]

@coderzc Could you cherry-pick this PR to branch-3.3?