[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image #24544

lhotari · 2025-07-22T10:54:57Z

Motivation

pulsar-client-python 3.8.0 has been released with fixes and it would be useful to upgrade pulsar-client-python used in Pulsar docker image.

Modifications

upgrade pulsar-client-python version to 3.8.0
add script for re-generating python protobuf and grpc stubs used in Pulsar Functions
update python protobuf and grpc stubs
pin python pip protobuf to 6.31.1 and grpc to 1.73.1 since these were used to generate the stubs

Documentation

doc
doc-required
doc-not-needed
doc-complete

lhotari · 2025-07-22T13:56:16Z

test failure in python related test:

  Error:  org.apache.pulsar.tests.integration.functions.PulsarStateTest.testPythonWordCountFunction  Time elapsed: 300.007 s  <<< FAILURE!
  org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.tests.integration.functions.PulsarStateTest.testPythonWordCountFunction() didn't finish within the time-out 300000
  	at java.base/jdk.internal.misc.Unsafe.park(Native Method)
  	at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
  	at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  	at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712)
  	at org.apache.pulsar.common.util.collections.GrowableArrayBlockingQueue.take(GrowableArrayBlockingQueue.java:197)
  	at org.apache.pulsar.client.impl.ConsumerImpl.internalReceive(ConsumerImpl.java:517)
  	at org.apache.pulsar.client.impl.ConsumerBase.receive(ConsumerBase.java:264)
  	at org.apache.pulsar.tests.integration.functions.PulsarStateTest.publishAndConsumeMessages(PulsarStateTest.java:609)
  	at org.apache.pulsar.tests.integration.functions.PulsarStateTest.doTestPythonWordCountFunction(PulsarStateTest.java:95)
  	at org.apache.pulsar.tests.integration.functions.PulsarStateTest.testPythonWordCountFunction(PulsarStateTest.java:75)
  	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
  	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  	at java.base/java.lang.Thread.run(Thread.java:1583)

lhotari · 2025-07-22T14:01:34Z

Found the python related error in logs:

2025-07-22T13:50:15.843245326Z 2025-07-22T13:50:15,843+0000 [function-timer-thread-104-1] INFO  org.apache.pulsar.functions.runtime.process.ProcessRuntime - Creating function log directory logs//functions/public/default/test-wordcount-py-fn-dvjvzzau
2025-07-22T13:50:15.843381144Z 2025-07-22T13:50:15,843+0000 [function-timer-thread-104-1] INFO  org.apache.pulsar.functions.runtime.process.ProcessRuntime - Created or found function log directory logs//functions/public/default/test-wordcount-py-fn-dvjvzzau
2025-07-22T13:50:15.843488732Z 2025-07-22T13:50:15,843+0000 [function-timer-thread-104-1] INFO  org.apache.pulsar.functions.runtime.process.ProcessRuntime - ProcessBuilder starting the process with args python3 /pulsar/instances/python-instance/python_instance_main.py --py /pulsar/download/pulsar_functions/public/default/test-wordcount-py-fn-dvjvzzau/0/wordcount_function.py --logging_directory logs//functions --logging_file test-wordcount-py-fn-dvjvzzau --logging_config_file /pulsar/conf/functions-logging/logging_config.ini --instance_id 0 --function_id 9c4218dc-01be-4cc8-a752-be47853460ab --function_version 0c43a98b-8731-4a6e-9350-71c1567c2fa3 --function_details '{"tenant":"public","namespace":"default","name":"test-wordcount-py-fn-dvjvzzau","className":"wordcount_function.WordCountFunction","runtime":"PYTHON","autoAck":true,"parallelism":1,"source":{"inputSpecs":{"test-wordcount-py-input-xwrapdis":{}},"cleanupSubscription":true},"sink":{"topic":"test-wordcount-py-output-ovmhbmug","forwardSourceMessageProperty":true},"resources":{"cpu":1.0,"ram":"1073741824","disk":"10737418240"},"componentType":"FUNCTION"}' --pulsar_serviceurl pulsar://localhost:6650 --max_buffered_tuples 1024 --port 39523 --metrics_port 37195 --state_storage_serviceurl bk://127.0.0.1:4181 --expected_healthcheck_interval 30 --secrets_provider secretsprovider.ClearTextSecretsProvider --cluster_name standalone
2025-07-22T13:50:15.846003070Z 2025-07-22T13:50:15,845+0000 [function-timer-thread-104-1] INFO  org.apache.pulsar.functions.runtime.process.ProcessRuntime - Started process successfully
2025-07-22T13:50:16.097383975Z Traceback (most recent call last):
2025-07-22T13:50:16.097406687Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 636, in _ConvertFieldValuePair
2025-07-22T13:50:16.097586414Z     self._ConvertMapFieldValue(
2025-07-22T13:50:16.097592856Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 833, in _ConvertMapFieldValue
2025-07-22T13:50:16.097839397Z     getattr(message, field.name)[key_value],
2025-07-22T13:50:16.097907004Z     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
2025-07-22T13:50:16.097911182Z   File "/usr/lib/python3.12/site-packages/google/protobuf/internal/containers.py", line 70, in __getitem__
2025-07-22T13:50:16.098044201Z     return self._values[key]
2025-07-22T13:50:16.098142318Z            ~~~~~~~~~~~~^^^^^
2025-07-22T13:50:16.098150653Z TypeError: list indices must be integers or slices, not str
2025-07-22T13:50:16.098155092Z 
2025-07-22T13:50:16.098158698Z The above exception was the direct cause of the following exception:
2025-07-22T13:50:16.098162526Z 
2025-07-22T13:50:16.098166102Z Traceback (most recent call last):
2025-07-22T13:50:16.098169368Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 684, in _ConvertFieldValuePair
2025-07-22T13:50:16.098404749Z     self.ConvertMessage(value, sub_message, '{0}.{1}'.format(path, name))
2025-07-22T13:50:16.098480150Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 540, in ConvertMessage
2025-07-22T13:50:16.098699912Z     self._ConvertFieldValuePair(value, message, path)
2025-07-22T13:50:16.098759733Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 702, in _ConvertFieldValuePair
2025-07-22T13:50:16.098994023Z     raise ParseError(
2025-07-22T13:50:16.099001526Z google.protobuf.json_format.ParseError: Failed to parse inputSpecs field: list indices must be integers or slices, not str.
2025-07-22T13:50:16.099005444Z 
2025-07-22T13:50:16.099008860Z The above exception was the direct cause of the following exception:
2025-07-22T13:50:16.099012457Z 
2025-07-22T13:50:16.099015392Z Traceback (most recent call last):
2025-07-22T13:50:16.099018438Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 318, in <module>
2025-07-22T13:50:16.099214756Z     main()
2025-07-22T13:50:16.099265337Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 165, in main
2025-07-22T13:50:16.099395865Z     json_format.Parse(args.function_details, function_details)
2025-07-22T13:50:16.099449836Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 464, in Parse
2025-07-22T13:50:16.099647907Z     raise e
2025-07-22T13:50:16.099654700Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 460, in Parse
2025-07-22T13:50:16.099860645Z     return ParseDict(
2025-07-22T13:50:16.099867588Z            ^^^^^^^^^^
2025-07-22T13:50:16.099870924Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 494, in ParseDict
2025-07-22T13:50:16.100043718Z     parser.ConvertMessage(js_dict, message, '')
2025-07-22T13:50:16.100107978Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 540, in ConvertMessage
2025-07-22T13:50:16.100319200Z     self._ConvertFieldValuePair(value, message, path)
2025-07-22T13:50:16.100377910Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 692, in _ConvertFieldValuePair
2025-07-22T13:50:16.100588585Z     raise ParseError(
2025-07-22T13:50:16.100593835Z google.protobuf.json_format.ParseError: Failed to parse source field: Failed to parse inputSpecs field: list indices must be integers or slices, not str..

@BewareMyPower Any recommendation about how to fix this?

seems to originate to this line of code:

pulsar/pulsar-functions/instance/src/main/python/python_instance_main.py

Line 165 in de33f3b

json_format.Parse(args.function_details, function_details)

BewareMyPower · 2025-07-22T14:29:38Z

This might be caused by the upgrade of protobuf in apache/pulsar-client-python#260 for CVE. The python protobuf API might have changed. I guess that we need to regenerate the Function_pb2.py with latest protobuf. I will take a look tomorrow.

BewareMyPower · 2025-07-22T14:33:26Z

BTW, it also reminds me of the discussion I started long days ago: https://lists.apache.org/thread/5s2l9vgt5f973psb6xlttpmy4rpym7zh

The Python function related code and tests are maintained in the apache/pulsar repo, while the extra dependencies are still maintained in https://github.com/apache/pulsar-client-python/blob/main/setup.py. If we can separate the [function] dependencies, it could be easier to maintain and the core repo won't have to wait for a new Python client release.

lhotari · 2025-07-22T17:27:38Z

This might be caused by the upgrade of protobuf in apache/pulsar-client-python#260 for CVE. The python protobuf API might have changed. I guess that we need to regenerate the Function_pb2.py with latest protobuf. I will take a look tomorrow.

Yes, it seems to be caused by that change. There are multiple *_pb2.py files in the apache/pulsar repository:

bin/proto/MLDataFormats_pb2.py
pulsar-functions/instance/src/main/python/Function_pb2.py
pulsar-functions/instance/src/main/python/InstanceCommunication_pb2.py
pulsar-functions/instance/src/main/python/InstanceCommunication_pb2_grpc.py

I guess all of them would have to be re-generated.
I didn't find the instructions or a script in the code base other than this:

pulsar/bin/pulsar-managed-ledger-admin

Lines 33 to 35 in de53c29

    
           sys.exit("Incompatible proto/MLDataFormats_pb2.py. Regenerate using: "+  
        
           "protoc -I=${PULSAR_PATH}/managed-ledger/src/main/proto --python_out=${PULSAR_PATH}/bin/proto/ "+ 
        
           "${PULSAR_PATH}/managed-ledger/src/main/proto/MLDataFormats.proto")

…image

lhotari · 2025-07-22T19:15:40Z

Let's see if the BookKeeper stream storage protobuf and grpc stubs are compatible. The stubs at https://github.com/apache/bookkeeper/tree/master/stream/clients/python/bookkeeper/proto haven't been updated for ages.

…lable in pip

codecov-commenter · 2025-07-22T22:34:57Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.31%. Comparing base (bbc6224) to head (d9cda77).
⚠️ Report is 1393 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #24544      +/-   ##
============================================
+ Coverage     73.57%   74.31%   +0.74%     
+ Complexity    32624    32575      -49     
============================================
  Files          1877     1874       -3     
  Lines        139502   146231    +6729     
  Branches      15299    16772    +1473     
============================================
+ Hits         102638   108673    +6035     
- Misses        28908    28922      +14     
- Partials       7956     8636     +680

Flag	Coverage Δ
inttests	`26.70% <ø> (+2.12%)`	⬆️
systests	`23.30% <ø> (-1.03%)`	⬇️
unittests	`73.79% <ø> (+0.95%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1111 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

BewareMyPower

Are these new scripts documented or integrated into Maven build?

lhotari · 2025-07-23T07:57:33Z

Are these new scripts documented or integrated into Maven build?

@BewareMyPower I've added docs now. There's no intention to integrate into the Maven build since the stubs would only need to be updated when the grpc version changes since grpcio-tools is used to generate stubs. grpcio-tools includes a specific libprotoc library so the protobuf version should match this version. I've documented this too. PTAL

(cherry picked from commit 767d583)

…he#24544) (cherry picked from commit 767d583) (cherry picked from commit 51dd60e)

…he#24544) (cherry picked from commit 767d583) (cherry picked from commit c171ecc)

…he#24544) (cherry picked from commit 767d583) (cherry picked from commit 51dd60e)

…he#24544) (cherry picked from commit 767d583) (cherry picked from commit c171ecc)

…he#24544) (cherry picked from commit 767d583) Signed-off-by: Zixuan Liu <[email protected]>

…he#24544)

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image

2149f72

lhotari requested a review from BewareMyPower July 22, 2025 10:54

lhotari self-assigned this Jul 22, 2025

lhotari added the ready-to-test label Jul 22, 2025

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jul 22, 2025

lhotari added release/3.0.13 release/4.0.6 release/3.3.8 and removed doc-not-needed Your PR changes do not impact docs labels Jul 22, 2025

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jul 22, 2025

BewareMyPower approved these changes Jul 22, 2025

View reviewed changes

lhotari added 4 commits July 22, 2025 21:38

Update protobuf stubs for Python using protobuf 6.31.0 and grpcio 1.73.1

f2c91cc

Install grpcio 1.73.1 and protobuf 6.31.1 pip packages in the docker …

f1431d3

…image

Exclude generated *_pb2.py and *_pb2_grpc.py files from license checks

ef5a965

Delete temp venv in the script

9c86de6

lhotari added 4 commits July 22, 2025 23:30

Remove version checks from generated stubs for less strict usage

fc2c23b

Don't install py3-grpcio package since the version is older than avai…

77e144d

…lable in pip

pin version for grpcio-tools in generating stubs

af865b2

Fix license header

b672f25

BewareMyPower reviewed Jul 23, 2025

View reviewed changes

lhotari added 3 commits July 23, 2025 10:35

Add docs

ae85815

Rename the docker container script

5cad4cc

Add notes about protobuf package compatibility

c7f0d1a

Reorder script and add some comments

b9fdb66

lhotari requested a review from BewareMyPower July 23, 2025 07:57

Cleanup temp venv in script

d9cda77

merlimat approved these changes Jul 23, 2025

View reviewed changes

lhotari merged commit 767d583 into apache:master Jul 23, 2025
93 of 97 checks passed

lhotari added a commit that referenced this pull request Jul 23, 2025

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image (#24544)

d56d647

(cherry picked from commit 767d583)

lhotari added a commit that referenced this pull request Jul 23, 2025

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image (#24544)

c171ecc

(cherry picked from commit 767d583)

lhotari added cherry-picked/branch-4.0 cherry-picked/branch-3.3 labels Jul 23, 2025

lhotari added a commit that referenced this pull request Jul 23, 2025

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image (#24544)

51dd60e

(cherry picked from commit 767d583)

lhotari added the cherry-picked/branch-3.0 label Jul 23, 2025

jiangpengcheng mentioned this pull request Jul 24, 2025

Upgrade protobuf to 6.31.1 in python runner image streamnative/function-mesh#814

Merged

4 tasks

priyanshu-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 24, 2025

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image (apac…

0593810

…he#24544) (cherry picked from commit 767d583) (cherry picked from commit 51dd60e)

priyanshu-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 24, 2025

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image (apac…

6e74d09

…he#24544) (cherry picked from commit 767d583) (cherry picked from commit 51dd60e)

priyanshu-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 24, 2025

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image (apac…

c0968ee

…he#24544) (cherry picked from commit 767d583) (cherry picked from commit c171ecc)

srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 24, 2025

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image (apac…

1724692

…he#24544) (cherry picked from commit 767d583) (cherry picked from commit 51dd60e)

srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 24, 2025

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image (apac…

9f00ce3

…he#24544) (cherry picked from commit 767d583) (cherry picked from commit c171ecc)

nodece pushed a commit to ascentstream/pulsar that referenced this pull request Jul 28, 2025

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image (apac…

372bcfa

…he#24544) (cherry picked from commit 767d583) Signed-off-by: Zixuan Liu <[email protected]>

nodece pushed a commit to ascentstream/pulsar that referenced this pull request Jul 28, 2025

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image (apac…

c48e172

…he#24544) (cherry picked from commit 767d583) Signed-off-by: Zixuan Liu <[email protected]>

KannarFr pushed a commit to CleverCloud/pulsar that referenced this pull request Sep 22, 2025

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image (apac…

b06f5f6

…he#24544)

walkinggo pushed a commit to walkinggo/pulsar that referenced this pull request Oct 8, 2025

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image (apac…

a091270

…he#24544)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image #24544

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image #24544

Uh oh!

lhotari commented Jul 22, 2025 •

edited

Loading

Uh oh!

lhotari commented Jul 22, 2025

Uh oh!

lhotari commented Jul 22, 2025 •

edited

Loading

Uh oh!

BewareMyPower commented Jul 22, 2025

Uh oh!

BewareMyPower commented Jul 22, 2025

Uh oh!

lhotari commented Jul 22, 2025

Uh oh!

lhotari commented Jul 22, 2025

Uh oh!

codecov-commenter commented Jul 22, 2025 •

edited

Loading

Uh oh!

BewareMyPower left a comment

Uh oh!

lhotari commented Jul 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image #24544

[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image #24544

Uh oh!

Conversation

lhotari commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Documentation

Uh oh!

lhotari commented Jul 22, 2025

Uh oh!

lhotari commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BewareMyPower commented Jul 22, 2025

Uh oh!

BewareMyPower commented Jul 22, 2025

Uh oh!

lhotari commented Jul 22, 2025

Uh oh!

lhotari commented Jul 22, 2025

Uh oh!

codecov-commenter commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

BewareMyPower left a comment

Choose a reason for hiding this comment

Uh oh!

lhotari commented Jul 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lhotari commented Jul 22, 2025 •

edited

Loading

lhotari commented Jul 22, 2025 •

edited

Loading

codecov-commenter commented Jul 22, 2025 •

edited

Loading