Skip to content

Commit 767d583

Browse files
lhotarilhotari
authored
[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image (#24544)
1 parent efe9dc8 commit 767d583

File tree

9 files changed

+416
-2794
lines changed

9 files changed

+416
-2794
lines changed

‎bin/proto/MLDataFormats_pb2.py‎

Lines changed: 51 additions & 330 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docker/pulsar/Dockerfile‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,6 @@ RUN apk add --no-cache \
8888
bash \
8989
python3 \
9090
py3-pip \
91-
py3-grpcio \
9291
py3-yaml \
9392
gcompat \
9493
libgcc \
@@ -105,9 +104,15 @@ RUN apk add --no-cache \
105104
RUN apk upgrade --no-cache
106105

107106
# Python dependencies
107+
# The pinned grpcio and protobuf versions should be compatible with the generated Protobuf and gRPC stubs used
108+
# in Pulsar Functions Python runtime. You should also update the grpcio version in src/update_python_protobuf_stubs.sh
109+
# and regenerate the Python stubs if you change the grpcio version here. Please see
110+
# pulsar-functions/instance/src/main/python/README.md for more details.
108111
ARG PULSAR_CLIENT_PYTHON_VERSION
109112
RUN pip3 install --break-system-packages --no-cache-dir \
110-
--only-binary grpcio \
113+
--only-binary \
114+
grpcio==1.73.1 \
115+
protobuf==6.31.1 \
111116
pulsar-client[all]==${PULSAR_CLIENT_PYTHON_VERSION} \
112117
kazoo
113118

‎pom.xml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ flexible messaging model and an intuitive client API.</description>
8282
<pulsar.broker.compiler.release>${maven.compiler.target}</pulsar.broker.compiler.release>
8383
<pulsar.client.compiler.release>17</pulsar.client.compiler.release>
8484

85-
<pulsar.client.python.version>3.7.0</pulsar.client.python.version>
85+
<pulsar.client.python.version>3.8.0</pulsar.client.python.version>
8686

8787
<IMAGE_JDK_MAJOR_VERSION>21</IMAGE_JDK_MAJOR_VERSION>
8888

@@ -1996,6 +1996,8 @@ flexible messaging model and an intuitive client API.</description>
19961996
<exclude>**/.mvn/**</exclude>
19971997
<exclude>**/generated/**</exclude>
19981998
<exclude>**/zk-3.5-test-data/*</exclude>
1999+
<exclude>**/*_pb2.py</exclude>
2000+
<exclude>**/*_pb2_grpc.py</exclude>
19992001
</excludes>
20002002
</licenseSet>
20012003
</licenseSets>
@@ -2149,6 +2151,8 @@ flexible messaging model and an intuitive client API.</description>
21492151
<exclude>**/*.dylib</exclude>
21502152
<exclude>**/*.patch</exclude>
21512153
<exclude>src/test/resources/*.txt</exclude>
2154+
<exclude>**/*_pb2.py</exclude>
2155+
<exclude>**/*_pb2_grpc.py</exclude>
21522156
</excludes>
21532157
</configuration>
21542158
</plugin>

‎pulsar-functions/instance/src/main/python/Function_pb2.py‎

Lines changed: 95 additions & 1766 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pulsar-functions/instance/src/main/python/InstanceCommunication_pb2.py‎

Lines changed: 34 additions & 656 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pulsar-functions/instance/src/main/python/InstanceCommunication_pb2_grpc.py‎

Lines changed: 75 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,14 @@
1-
#
2-
# Licensed to the Apache Software Foundation (ASF) under one
3-
# or more contributor license agreements. See the NOTICE file
4-
# distributed with this work for additional information
5-
# regarding copyright ownership. The ASF licenses this file
6-
# to you under the Apache License, Version 2.0 (the
7-
# "License"); you may not use this file except in compliance
8-
# with the License. You may obtain a copy of the License at
9-
#
10-
# http://www.apache.org/licenses/LICENSE-2.0
11-
#
12-
# Unless required by applicable law or agreed to in writing,
13-
# software distributed under the License is distributed on an
14-
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15-
# KIND, either express or implied. See the License for the
16-
# specific language governing permissions and limitations
17-
# under the License.
18-
#
19-
201
# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
212
"""Client and server classes corresponding to protobuf-defined services."""
223
import grpc
4+
import warnings
235

246
import InstanceCommunication_pb2 as InstanceCommunication__pb2
257
from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
268

9+
GRPC_GENERATED_VERSION = '1.73.1'
10+
GRPC_VERSION = grpc.__version__
11+
2712

2813
class InstanceControlStub(object):
2914
"""Missing associated documentation comment in .proto file."""
@@ -38,27 +23,27 @@ def __init__(self, channel):
3823
'/proto.InstanceControl/GetFunctionStatus',
3924
request_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
4025
response_deserializer=InstanceCommunication__pb2.FunctionStatus.FromString,
41-
)
26+
_registered_method=True)
4227
self.GetAndResetMetrics = channel.unary_unary(
4328
'/proto.InstanceControl/GetAndResetMetrics',
4429
request_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
4530
response_deserializer=InstanceCommunication__pb2.MetricsData.FromString,
46-
)
31+
_registered_method=True)
4732
self.ResetMetrics = channel.unary_unary(
4833
'/proto.InstanceControl/ResetMetrics',
4934
request_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
5035
response_deserializer=google_dot_protobuf_dot_empty__pb2.Empty.FromString,
51-
)
36+
_registered_method=True)
5237
self.GetMetrics = channel.unary_unary(
5338
'/proto.InstanceControl/GetMetrics',
5439
request_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
5540
response_deserializer=InstanceCommunication__pb2.MetricsData.FromString,
56-
)
41+
_registered_method=True)
5742
self.HealthCheck = channel.unary_unary(
5843
'/proto.InstanceControl/HealthCheck',
5944
request_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
6045
response_deserializer=InstanceCommunication__pb2.HealthCheckResult.FromString,
61-
)
46+
_registered_method=True)
6247

6348

6449
class InstanceControlServicer(object):
@@ -126,6 +111,7 @@ def add_InstanceControlServicer_to_server(servicer, server):
126111
generic_handler = grpc.method_handlers_generic_handler(
127112
'proto.InstanceControl', rpc_method_handlers)
128113
server.add_generic_rpc_handlers((generic_handler,))
114+
server.add_registered_method_handlers('proto.InstanceControl', rpc_method_handlers)
129115

130116

131117
# This class is part of an EXPERIMENTAL API.
@@ -143,11 +129,21 @@ def GetFunctionStatus(request,
143129
wait_for_ready=None,
144130
timeout=None,
145131
metadata=None):
146-
return grpc.experimental.unary_unary(request, target, '/proto.InstanceControl/GetFunctionStatus',
132+
return grpc.experimental.unary_unary(
133+
request,
134+
target,
135+
'/proto.InstanceControl/GetFunctionStatus',
147136
google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
148137
InstanceCommunication__pb2.FunctionStatus.FromString,
149-
options, channel_credentials,
150-
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
138+
options,
139+
channel_credentials,
140+
insecure,
141+
call_credentials,
142+
compression,
143+
wait_for_ready,
144+
timeout,
145+
metadata,
146+
_registered_method=True)
151147

152148
@staticmethod
153149
def GetAndResetMetrics(request,
@@ -160,11 +156,21 @@ def GetAndResetMetrics(request,
160156
wait_for_ready=None,
161157
timeout=None,
162158
metadata=None):
163-
return grpc.experimental.unary_unary(request, target, '/proto.InstanceControl/GetAndResetMetrics',
159+
return grpc.experimental.unary_unary(
160+
request,
161+
target,
162+
'/proto.InstanceControl/GetAndResetMetrics',
164163
google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
165164
InstanceCommunication__pb2.MetricsData.FromString,
166-
options, channel_credentials,
167-
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
165+
options,
166+
channel_credentials,
167+
insecure,
168+
call_credentials,
169+
compression,
170+
wait_for_ready,
171+
timeout,
172+
metadata,
173+
_registered_method=True)
168174

169175
@staticmethod
170176
def ResetMetrics(request,
@@ -177,11 +183,21 @@ def ResetMetrics(request,
177183
wait_for_ready=None,
178184
timeout=None,
179185
metadata=None):
180-
return grpc.experimental.unary_unary(request, target, '/proto.InstanceControl/ResetMetrics',
186+
return grpc.experimental.unary_unary(
187+
request,
188+
target,
189+
'/proto.InstanceControl/ResetMetrics',
181190
google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
182191
google_dot_protobuf_dot_empty__pb2.Empty.FromString,
183-
options, channel_credentials,
184-
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
192+
options,
193+
channel_credentials,
194+
insecure,
195+
call_credentials,
196+
compression,
197+
wait_for_ready,
198+
timeout,
199+
metadata,
200+
_registered_method=True)
185201

186202
@staticmethod
187203
def GetMetrics(request,
@@ -194,11 +210,21 @@ def GetMetrics(request,
194210
wait_for_ready=None,
195211
timeout=None,
196212
metadata=None):
197-
return grpc.experimental.unary_unary(request, target, '/proto.InstanceControl/GetMetrics',
213+
return grpc.experimental.unary_unary(
214+
request,
215+
target,
216+
'/proto.InstanceControl/GetMetrics',
198217
google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
199218
InstanceCommunication__pb2.MetricsData.FromString,
200-
options, channel_credentials,
201-
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
219+
options,
220+
channel_credentials,
221+
insecure,
222+
call_credentials,
223+
compression,
224+
wait_for_ready,
225+
timeout,
226+
metadata,
227+
_registered_method=True)
202228

203229
@staticmethod
204230
def HealthCheck(request,
@@ -211,8 +237,18 @@ def HealthCheck(request,
211237
wait_for_ready=None,
212238
timeout=None,
213239
metadata=None):
214-
return grpc.experimental.unary_unary(request, target, '/proto.InstanceControl/HealthCheck',
240+
return grpc.experimental.unary_unary(
241+
request,
242+
target,
243+
'/proto.InstanceControl/HealthCheck',
215244
google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
216245
InstanceCommunication__pb2.HealthCheckResult.FromString,
217-
options, channel_credentials,
218-
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
246+
options,
247+
channel_credentials,
248+
insecure,
249+
call_credentials,
250+
compression,
251+
wait_for_ready,
252+
timeout,
253+
metadata,
254+
_registered_method=True)

‎pulsar-functions/instance/src/main/python/README.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Pulsar Functions Python Runtime
2+
3+
### Updating Protobuf and gRPC generated stubs
4+
5+
When using generated Protobuf and gRPC stubs (`*_pb2.py`, `*_pb2_gprc.py`), the generated code should be
6+
updated when the grpcio and protobuf Python packages are updated. This is due to the fact that generated
7+
Protobuf and gRPC stubs are not necessarily compatible across different versions of these packages at runtime.
8+
The compatibility policy of Protobuf is documented in
9+
[Protobuf's "Cross-Version Runtime Guarantee"](https://protobuf.dev/support/cross-version-runtime-guarantee/),
10+
which states that cross-version runtime support isn't guaranteed. gRPC follows a similar policy.
11+
12+
In Pulsar's [Docker image](../../../../../docker/pulsar/Dockerfile), the `grpcio` and `protobuf` packages are
13+
pinned to specific versions. Whenever these versions are updated, the `PYTHON_GRPCIO_VERSION`
14+
in [src/update_python_protobuf_stubs.sh](../../../../../src/update_python_protobuf_stubs.sh) should also be updated
15+
and the generated stubs should be regenerated with this script to ensure compatibility.
16+
17+
To update the generated stubs, run the following command in the project root directory:
18+
19+
```bash
20+
# run this command from the project root directory
21+
src/update_python_protobuf_stubs.sh
22+
```
23+
24+
Alternatively, you can run this command to install the required tools in a docker container and update the stubs:
25+
26+
```bash
27+
# run this command from the project root directory
28+
src/update_python_protobuf_stubs_with_docker.sh
29+
```
30+
31+
When the script is run, it will also print such information to the console:
32+
33+
```
34+
libprotoc library included in grpcio-tools will be used:
35+
libprotoc 31.0
36+
The compatible matching protobuf package version in Python is prefixed with '6.'
37+
Ensure that you are using a compatible version of the protobuf package such as 6.31.0 (or a matching patch version).
38+
```
39+
40+
When pinning the `protobuf` package in your Python project follow this guidance to ensure compatibility of the generated stubs with the `protobuf` package version.

‎src/update_python_protobuf_stubs.sh‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
#
20+
21+
# This script generates Python gRPC and Protobuf stubs from the .proto files
22+
# Set the version of PYTHON_GRPCIO_VERSION to the version which matches the version of grpcio in the Dockerfile
23+
PYTHON_GRPCIO_VERSION=${PYTHON_GRPCIO_VERSION:-"1.73.1"}
24+
25+
set -euo pipefail
26+
27+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" &>/dev/null && pwd)"
28+
29+
# Create a temporary virtual environment to avoid polluting the global Python environment
30+
tempvenv=$(mktemp -d /tmp/pulsar-venv.XXXXXX)
31+
python3 -m venv $tempvenv
32+
source $tempvenv/bin/activate
33+
34+
# install the required packages for protobuf and grpc
35+
echo "Installing grpc-tools $PYTHON_GRPCIO_VERSION..."
36+
python3 -m pip install grpcio-tools==$PYTHON_GRPCIO_VERSION
37+
38+
echo "libprotoc library included in grpcio-tools will be used:"
39+
python3 -m grpc_tools.protoc --version
40+
echo "The compatible matching protobuf package version in Python is prefixed with '6.'"
41+
echo "Ensure that you are using a compatible version of the protobuf package such as 6.$(python3 -m grpc_tools.protoc --version | awk '{print $2}') (or a matching patch version)."
42+
43+
cd $SCRIPT_DIR/..
44+
echo "Generating Python gRPC and Protobuf stubs from the .proto files..."
45+
46+
# Generate Python gRPC and Protobuf stubs from the .proto files
47+
48+
# Generate stubs for Function.proto and InstanceCommunication.proto, used for Pulsar Functions Python runtime
49+
python3 -m grpc_tools.protoc \
50+
--proto_path=pulsar-functions/proto/src/main/proto \
51+
--python_out=pulsar-functions/instance/src/main/python \
52+
pulsar-functions/proto/src/main/proto/Function.proto
53+
54+
# Remove the strict version checking in the generated file
55+
sed -i '/^_runtime_version\.ValidateProtobufRuntimeVersion($/,/^)$/d' \
56+
pulsar-functions/instance/src/main/python/Function_pb2.py
57+
58+
python3 -m grpc_tools.protoc \
59+
--proto_path=pulsar-functions/proto/src/main/proto \
60+
--python_out=pulsar-functions/instance/src/main/python \
61+
--grpc_python_out=pulsar-functions/instance/src/main/python \
62+
pulsar-functions/proto/src/main/proto/InstanceCommunication.proto
63+
64+
# Remove the strict version checking in the generated file
65+
sed -i '/^_runtime_version\.ValidateProtobufRuntimeVersion($/,/^)$/d' \
66+
pulsar-functions/instance/src/main/python/InstanceCommunication_pb2.py
67+
sed -i '/^_version_not_supported = False$/,/^ )$/d' \
68+
pulsar-functions/instance/src/main/python/InstanceCommunication_pb2_grpc.py
69+
70+
# Generate stubs for MLDataFormats.proto, used for managed-ledger python scripts
71+
python3 -m grpc_tools.protoc \
72+
--proto_path=managed-ledger/src/main/proto \
73+
--python_out=bin/proto \
74+
managed-ledger/src/main/proto/MLDataFormats.proto
75+
76+
# Remove the strict version checking in the generated file
77+
sed -i '/^_runtime_version\.ValidateProtobufRuntimeVersion($/,/^)$/d' \
78+
bin/proto/MLDataFormats_pb2.py
79+
80+
echo "Python gRPC and Protobuf stubs generated successfully."
81+
82+
# Deactivate and remove the temporary virtual environment
83+
deactivate
84+
rm -rf $tempvenv

0 commit comments

Comments
 (0)