Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HighLevelProducer with KeyedPartitioner fails on first send #354

Open
itamarwe opened this issue Apr 19, 2016 · 9 comments · May be fixed by #1360
Open

HighLevelProducer with KeyedPartitioner fails on first send #354

itamarwe opened this issue Apr 19, 2016 · 9 comments · May be fixed by #1360

Comments

@itamarwe
Copy link

When using KeyedParitioner with the HighLevelProducer the first send fails with

BrokerNotAvailableError: Could not find the leader

Consecutive sends work perfectly.

@eugeneware
Copy link

Yeah. I'm getting the same error no matter what I do.

@eugeneware
Copy link

The random partitioner and default parttitioners work. random and keyed both fail with the BrokerNotAvailableError.

@itamarwe
Copy link
Author

itamarwe commented Apr 26, 2016

Only for the first message, though. (or until the topic metadata is updated?)

@itamarwe
Copy link
Author

I've created a pull-request to fix it:
#378

As suspected, the metadata isn't updated when the producer is initialized, so the partitions array is empty, which causes the keyed partitioner to return NaN as the chosen partitions, which fails the producer.

@griffinmichl
Copy link

Yup, same error. I have also found the documentation for working with keyed partitions very poor (probably because they don't work). I think this is something that could be improved upon as well.

@askhogan
Copy link

I just ended up rolling my own and manually setting partition number in payloads.

@bfmnorman
Copy link

I ran into this problem today, and I was able to get around it by calling client.loadMetadataForTopics([topic]) and then calling client.updateMetadatas(metadata) in the callback function. After calling that, I called back my own callback saying that the producer is ready to use.

I did all this with several promises, so it might be more difficult to do with just using callbacks. Basically, I didn't resolve the "producer" promise until after updating the metadata for the topic on the client.

@badeball
Copy link

badeball commented Feb 4, 2019

I've attemped to create a minimal example to reproduce this issue in this gist. I say attempted, because there's no error thrown with v4.0.1. I wasn't able to install v0.3.2, which seems to be what was available back then (and I'm not super keen on degrading my Arch system back to dinosaur). Is anyone able to use the example to confirm that this used to be an issue and isn't anymore?

@alex3165
Copy link

alex3165 commented Dec 3, 2019

Hey, I opened a PR to fix this issue: #1360

@alex3165 alex3165 linked a pull request Dec 3, 2019 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants