[improve][doc] Updated concepts-broker-load-balancing-types #655
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@heesung-sn Please add the following content to your PR description and select a checkbox: |
github-actions
bot
added
doc
sharing a preview from my local ide.
|
Anonymitaet
reviewed
Jul 27, 2023
| | Dimension | Modular broker load balancer |Extensible broker load balancer | ||
| |--------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--- | ||
| Available version | Pulsar 1.7.0 <br/> <br/>2017 |Pulsar 3.0.0 <br/> <br/>2023 | ||
| Load Balance Metadata Store and Replication | Dependent on ZooKeeper for load balance metadata store and replication.<br/> <br/> - All broker load data and all bundle load data are stored in ZooKeeper and replicated to all brokers via ZooKeeper watchers. <br/><br/>- All bundle ownerships are stored in Zookeeper as ephemeral locks. | Dependent on system topics and table views for load balance metadata store and replication. <br/> <br/> - All broker load data are stored in a non-persistent system topic and replicated to all brokers via table views. <br/> <br/>- Each broker publishes only top k bundles' load data in a non-persistent system topic, and only the leader broker consumes it via a table view. <br/> <br/> - Bundle ownerships are stored in a persistent system topic and replicated to all brokers via table views.<br/> <br/> * The absolute size of the replicated load data and the complexity of the replication decreased due to passing the data through memory-only, non-persistent topics. |
There was a problem hiding this comment.
Suggested change
| Load Balance Metadata Store and Replication | Dependent on ZooKeeper for load balance metadata store and replication.<br/> <br/> - All broker load data and all bundle load data are stored in ZooKeeper and replicated to all brokers via ZooKeeper watchers. <br/><br/>- All bundle ownerships are stored in Zookeeper as ephemeral locks. | Dependent on system topics and table views for load balance metadata store and replication. <br/> <br/> - All broker load data are stored in a non-persistent system topic and replicated to all brokers via table views. <br/> <br/>- Each broker publishes only top k bundles' load data in a non-persistent system topic, and only the leader broker consumes it via a table view. <br/> <br/> - Bundle ownerships are stored in a persistent system topic and replicated to all brokers via table views.<br/> <br/> * The absolute size of the replicated load data and the complexity of the replication decreased due to passing the data through memory-only, non-persistent topics. | |
| Load Balance Metadata Store and Replication | Dependent on ZooKeeper for load balance metadata store and replication.<br/> <br/> - All broker load data and all bundle load data are stored in ZooKeeper and replicated to all brokers via ZooKeeper watchers. <br/><br/>- All bundle ownerships are stored in ZooKeeper as ephemeral locks. | Dependent on system topics and table views for load balance metadata store and replication. <br/> <br/> - All broker load data are stored in non-persistent system topics and replicated to all brokers via table views. <br/> <br/>- Each broker publishes only top k bundles' load data in non-persistent system topics, and only the leader broker consumes it via table views. <br/> <br/> - Bundle ownerships are stored in persistent system topics and replicated to all brokers via table views.<br/> <br/> - The absolute size of the replicated load data and the complexity of the replication decreased due to passing the data through memory-only, non-persistent topics. |
There was a problem hiding this comment.
The broker load data is stored in a single non-persistent topic. (1st topic)
The topk bundle load data is stored in a separate non-persistent topic.(2nd topic)
Bundle ownerships are stored in a separate persistent topic.(3rd topic)
I intentionally used * to note that the following is an annotation.
the absolute size of the replicated load data and the complexity of the replication decreased due to passing the data through memory-only, non-persistent topics
There was a problem hiding this comment.
Suggest using another format (like Note: xxxx), otherwise, users might take it as an item of the unordered list.
There was a problem hiding this comment.
Replaced * with Note.
| |--------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--- | ||
| Available version | Pulsar 1.7.0 <br/> <br/>2017 |Pulsar 3.0.0 <br/> <br/>2023 | ||
| Load Balance Metadata Store and Replication | Dependent on ZooKeeper for load balance metadata store and replication.<br/> <br/> - All broker load data and all bundle load data are stored in ZooKeeper and replicated to all brokers via ZooKeeper watchers. <br/><br/>- All bundle ownerships are stored in Zookeeper as ephemeral locks. | Dependent on system topics and table views for load balance metadata store and replication. <br/> <br/> - All broker load data are stored in a non-persistent system topic and replicated to all brokers via table views. <br/> <br/>- Each broker publishes only top k bundles' load data in a non-persistent system topic, and only the leader broker consumes it via a table view. <br/> <br/> - Bundle ownerships are stored in a persistent system topic and replicated to all brokers via table views.<br/> <br/> * The absolute size of the replicated load data and the complexity of the replication decreased due to passing the data through memory-only, non-persistent topics. | ||
| Load Balance Decision<br/>(Load Balance leader dependency)| Single leader broker decides on bundle-broker assignment, bundle splitting, and bundle unloading. <br/> <br/> * The leader broker requires global load information | Each broker decides and runs bundle-broker assignment(lookup) and bundle splitting based on the local(replicated) information. <br/><br/> The leader broker still decides on bundle unloading globally. |
There was a problem hiding this comment.
Suggested change
| Load Balance Decision<br/>(Load Balance leader dependency)| Single leader broker decides on bundle-broker assignment, bundle splitting, and bundle unloading. <br/> <br/> * The leader broker requires global load information | Each broker decides and runs bundle-broker assignment(lookup) and bundle splitting based on the local(replicated) information. <br/><br/> The leader broker still decides on bundle unloading globally. | |
| Load Balance Decision<br/>(Load Balance leader dependency)| Single leader broker decides on bundle-broker assignment, bundle splitting, and bundle unloading. <br/> <br/> - The leader broker requires global load information | Each broker decides and runs bundle-broker assignment (lookup) and bundle splitting based on the local (replicated) information. <br/><br/> The leader broker still decides on bundle unloading globally. |
There was a problem hiding this comment.
I intentionally used * to note that the following is an annotation.
The leader broker requires global load information
| Available version | Pulsar 1.7.0 <br/> <br/>2017 |Pulsar 3.0.0 <br/> <br/>2023 | ||
| Load Balance Metadata Store and Replication | Dependent on ZooKeeper for load balance metadata store and replication.<br/> <br/> - All broker load data and all bundle load data are stored in ZooKeeper and replicated to all brokers via ZooKeeper watchers. <br/><br/>- All bundle ownerships are stored in Zookeeper as ephemeral locks. | Dependent on system topics and table views for load balance metadata store and replication. <br/> <br/> - All broker load data are stored in a non-persistent system topic and replicated to all brokers via table views. <br/> <br/>- Each broker publishes only top k bundles' load data in a non-persistent system topic, and only the leader broker consumes it via a table view. <br/> <br/> - Bundle ownerships are stored in a persistent system topic and replicated to all brokers via table views.<br/> <br/> * The absolute size of the replicated load data and the complexity of the replication decreased due to passing the data through memory-only, non-persistent topics. | ||
| Load Balance Decision<br/>(Load Balance leader dependency)| Single leader broker decides on bundle-broker assignment, bundle splitting, and bundle unloading. <br/> <br/> * The leader broker requires global load information | Each broker decides and runs bundle-broker assignment(lookup) and bundle splitting based on the local(replicated) information. <br/><br/> The leader broker still decides on bundle unloading globally. | ||
| Pub/Sub Reconnection upon Bundle Unloading | When bundles are unloaded, the topics in the bundles will be temporarily closed, and producers and consumers connect to new owner brokers. <br/> <br/> - Connecting to the new owner brokers involves redirecting lookup requests via the leader broker. | Any broker can respond to the lookup requests to locate the new owner brokers without asking the leader broker. The reconnection does not go through the leader broker. |
There was a problem hiding this comment.
Suggested change
| Pub/Sub Reconnection upon Bundle Unloading | When bundles are unloaded, the topics in the bundles will be temporarily closed, and producers and consumers connect to new owner brokers. <br/> <br/> - Connecting to the new owner brokers involves redirecting lookup requests via the leader broker. | Any broker can respond to the lookup requests to locate the new owner brokers without asking the leader broker. The reconnection does not go through the leader broker. | |
| Pub/sub reconnection upon bundle unloading | When bundles are unloaded, the topics in the bundles are temporarily closed, and producers and consumers connect to new owner brokers. <br/> <br/> - Connecting to the new owner brokers involves redirecting lookup requests via the leader broker. | Any broker can respond to the lookup requests to locate the new owner brokers without asking the leader broker. The reconnection does not go through the leader broker. |
heesung-sn
force-pushed
the
concepts-broker-load-balancing-types
branch
from
f2a054e
to
341254f
Compare
Anonymitaet
approved these changes
Jul 27, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updated concepts-broker-load-balancing-types.
We will have a separate blog for the performance test part. I think we need to keep the tone of this comparison natural on the website.
docdoc-requireddoc-not-neededdoc-complete