Implementation of the asyncio consumer #228
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BewareMyPower
requested changes
Sep 27, 2024
Could you also make the tests pass for Python 3.8? Though Python 3.8 will reach EOL on the next month (2024-10) |
|
I will try to fix the issues this week. |
|
I found another issue with the schema system in my asyncio implementation, i will fix this too before the next commit. |
|
Let me fix the broken CI first |
|
Could you rebase to master to resolve the conflicts and have the CI fixed? |
BewareMyPower
requested changes
Nov 27, 2024
|
@BewareMyPower I removed the binary files |
merlimat
reviewed
Dec 2, 2024
There was a problem hiding this comment.
When you wrap an xxxAsync API from pulsar-client-cpp, the correct way is:
// release the GIL because xxxAsync does not access any Python objects
py::gil_scoped_release release;
// Pass the callback directly because pybind11 acquires the GIL automatically when the callback is executed
xxx.xxxAsync(yyy, callback);|
|
||
| namespace py = pybind11; | ||
|
|
||
| void Consumer_unsubscribe(Consumer& consumer) { | ||
| waitForAsyncResult([&consumer](ResultCallback callback) { consumer.unsubscribeAsync(callback); }); | ||
| } | ||
|
|
||
| void Consumer_unsubscribeAsync(Consumer& consumer, ResultCallback callback) { | ||
| consumer.unsubscribeAsync([callback] (Result result) { | ||
| py::gil_scoped_acquire acquire; |
There was a problem hiding this comment.
This GIL acquire is not necessary
| Message Consumer_receive(Consumer& consumer) { | ||
| return waitForAsyncValue<Message>([&](ReceiveCallback callback) { consumer.receiveAsync(callback); }); | ||
| } | ||
|
|
||
| void Consumer_receiveAsync(Consumer& consumer, ReceiveCallback callback) { | ||
| py::gil_scoped_acquire acquire; |
There was a problem hiding this comment.
It should release the GIL rather than acquire the GIL
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi,
I implemented the asyncio consumer in a similar way as the producer. All missing async functions provided by the C++ library are now implemented in the pybind11 classes and I use futures in python to convert the functions with callbacks to async functions. The reason I started with this implementation is because I need the async consumer in a project with FastAPI myself.
Tests could be a little bit patchy, for that reason i would appreciate help if i missed something