Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recommend ZK for Pulsar standalone #871

Merged
merged 4 commits into from
Apr 2, 2024
Merged

Recommend ZK for Pulsar standalone #871

merged 4 commits into from
Apr 2, 2024

Conversation

visortelle
Copy link
Member

@visortelle visortelle commented Mar 29, 2024

✅ Contribution Checklist


@visortelle visortelle requested a review from lhotari March 29, 2024 13:25
@merlimat
Copy link
Contributor

According to the issue it doesn't look to be a RocksDB metadata issue, rather a BK bug


You may encounter issues with the default RocksDB metadata store after restarting Pulsar standalone.

We recommend you consider using the following environment variable to use ZooKeeper as the metadata store:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's a good idea to recommend not to use a default setting. If we have a bug in default configuration, we must fix it. If it cannot be fixed, we will have to change the defaults.

Copy link
Member Author

@visortelle visortelle Mar 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be the best solution. @lhotari 's concern was that:

it would have to be documented since many might be using old versions and wondering about the resolution.
The problem in changing the default is that it would break existing "deployments" with data

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's a good idea to recommend not to use a default setting. If we have a bug in default configuration, we must fix it. If it cannot be fixed, we will have to change the defaults.

@merlimat I agree. Since we cannot change already released versions and users are hitting problems, I think it's worth documenting the workaround until there's a better solution. It seems that there isn't a nice way to solve this problem.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll merge this PR with the current content. We can come back to this later once there's a better resolution.

@visortelle
Copy link
Member Author

visortelle commented Mar 29, 2024

@merlimat you're right. I removed the mention of the wrong tracking issue.

I'm a little confused about the different problems I encountered with Pulsar standalone. The issue with RocksDB I have:
apache/pulsar#15896 (comment)

Error generating ledger id: org.apache.pulsar.metadata.api.MetadataStoreException: org.rocksdb.RocksDBException: TimedOut(LockTimeout)

I had an app that concurrently created 2 tenants, a few namespaces, and about 15 topics with roughly 50 partitions in total and created a single producer and consumer for each topic.

The problem was fixed after I started to create resources sequentially.

I can share a docker image with the app where the issue can be reproduced if it makes sense.
Note: Now it creates about 60 topics with roughly 210 partitions. It has quite a small load for each topic:

Screenshot 2024-03-29 at 9 04 05 PM

@lhotari
Copy link
Member

lhotari commented Apr 2, 2024

I made a separate commit 96cdbf2 for fixing the chown

Copy link
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, although as discussed there isn't a good solution and it's better to have the workaround documented.

@lhotari lhotari merged commit 19a9862 into main Apr 2, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants