Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kafka Provider Comparision Report #1

Open
KaimingWan opened this issue Jun 3, 2024 · 8 comments
Open

Kafka Provider Comparision Report #1

KaimingWan opened this issue Jun 3, 2024 · 8 comments

Comments

@KaimingWan
Copy link
Contributor

No description provided.

@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 6, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 6, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 6, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 6, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 12, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 12, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 12, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 12, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 12, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 12, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 12, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 12, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 13, 2024 •
edited by KaimingWan
Loading

AutoMQ Benchmark VS. Result 🚀

Benchmark Info

Report Generated: 2024-06-13 11:16:42

Workload Configuration

name: 1-topic-1000-partitions-4kb-4p4c-500m
topics: 1
partitionsPerTopic: 1000
partitionsPerTopicList: null
randomTopicNames: true
keyDistributor: NO_KEY
messageSize: 4096
useRandomizedPayloads: false
randomBytesRatio: 0
randomizedPayloadPoolSize: 0
payloadFile: payload/payload-4Kb.data
subscriptionsPerTopic: 1
producersPerTopic: 8
producersPerTopicList: null
consumerPerSubscription: 4
producerRate: 128000
producerRateList: null
consumerBacklogSizeGB: 0
backlogDrainRatio: 1
testDurationMinutes: 15
warmupDurationMinutes: 10
logIntervalMillis: 1500

Producer Configuration

value.serializer: org.apache.kafka.common.serialization.ByteArraySerializer
acks: all
batch.size: 65536
bootstrap.servers: 10.0.0.37:9092,10.0.1.168:9092,10.0.0.160:9092
key.serializer: org.apache.kafka.common.serialization.StringSerializer
linger.ms: 1

Consumer Configuration

key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
value.deserializer: org.apache.kafka.common.serialization.ByteArrayDeserializer
enable.auto.commit: true
bootstrap.servers: 10.0.0.37:9092,10.0.1.168:9092,10.0.0.160:9092
auto.offset.reset: earliest

Topic Configuration

min.insync.replicas: 1
retention.ms: 86400000
flush.messages: 1

replicationFactor

[AutoMQ] 1
[Kafka] 3

Average Throughput

Average Throughput [AutoMQ]: 500.30 MB/s
Average Throughput [Kafka]: 500.23 MB/s

Cost Estimate Rule: Check explanation under cost-explanation directory of this repository

Kafka Provider Pub Latency (ms) avg Pub Latency (ms) P99 E2E LatencyAvg(ms) E2E P95 Latency(ms) E2E P99 Latency(ms) Baseline Cost Usage Cost Total Cost
AutoMQ 3.4 15.19 6.54 15.91 34.16 1108.35 1265.0 2373.35
Apache Kafka 4.2 30.21 6.31 12.39 42.31 10647.35 0.0 10647.35

@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 13, 2024
@github-actions GitHub Actions
Copy link

AutoMQ Benchmark VS. Result 🚀

Benchmark Info

Report Generated: 2024-06-17 01:20:56

Workload Configuration

name: 1-topic-1000-partitions-4kb-4p4c-500m
topics: 1
partitionsPerTopic: 1000
partitionsPerTopicList: null
randomTopicNames: true
keyDistributor: NO_KEY
messageSize: 4096
useRandomizedPayloads: false
randomBytesRatio: 0
randomizedPayloadPoolSize: 0
payloadFile: payload/payload-4Kb.data
subscriptionsPerTopic: 1
producersPerTopic: 8
producersPerTopicList: null
consumerPerSubscription: 4
producerRate: 128000
producerRateList: null
consumerBacklogSizeGB: 0
backlogDrainRatio: 1
testDurationMinutes: 15
warmupDurationMinutes: 10
logIntervalMillis: 1500

Producer Configuration

value.serializer: org.apache.kafka.common.serialization.ByteArraySerializer
acks: all
batch.size: 65536
bootstrap.servers: 10.0.0.60:9092,10.0.1.146:9092,10.0.0.158:9092
key.serializer: org.apache.kafka.common.serialization.StringSerializer
linger.ms: 1

Consumer Configuration

key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
value.deserializer: org.apache.kafka.common.serialization.ByteArrayDeserializer
enable.auto.commit: true
bootstrap.servers: 10.0.0.60:9092,10.0.1.146:9092,10.0.0.158:9092
auto.offset.reset: earliest

Topic Configuration

min.insync.replicas: 1
retention.ms: 86400000
flush.messages: 1

replicationFactor

[AutoMQ] 1
[Kafka] 3

Average Throughput

Average Throughput [AutoMQ]: 500.33 MB/s
Average Throughput [Kafka]: 500.25 MB/s

Cost Estimate Rule: Check explanation under cost-explanation directory of this repository

Kafka Provider Pub Latency (ms) avg Pub Latency (ms) P99 E2E LatencyAvg(ms) E2E P95 Latency(ms) E2E P99 Latency(ms) Baseline Cost Usage Cost Total Cost
AutoMQ 3.4 15.69 6.64 15.54 34.96 1107.35 1264.0 2371.35
Apache Kafka 4.92 37.65 6.88 16.40 50.34 10639.35 0.0 10639.35

@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 20, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 20, 2024
@github-actions GitHub Actions
Copy link

AutoMQ Benchmark VS. Result 🚀

Benchmark Info

Report Generated: 2024-06-20 07:38:52

Workload Configuration

name: 1-topic-1000-partitions-4kb-4p4c-500m
topics: 1
partitionsPerTopic: 1000
partitionsPerTopicList: null
randomTopicNames: true
keyDistributor: NO_KEY
messageSize: 4096
useRandomizedPayloads: false
randomBytesRatio: 0
randomizedPayloadPoolSize: 0
payloadFile: payload/payload-4Kb.data
subscriptionsPerTopic: 1
producersPerTopic: 8
producersPerTopicList: null
consumerPerSubscription: 4
producerRate: 128000
producerRateList: null
consumerBacklogSizeGB: 0
backlogDrainRatio: 1
testDurationMinutes: 15
warmupDurationMinutes: 10
logIntervalMillis: 1500

Producer Configuration

value.serializer: org.apache.kafka.common.serialization.ByteArraySerializer
acks: all
batch.size: 65536
bootstrap.servers: 10.0.0.80:9092,10.0.1.244:9092,10.0.0.6:9092
key.serializer: org.apache.kafka.common.serialization.StringSerializer
linger.ms: 1

Consumer Configuration

key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
value.deserializer: org.apache.kafka.common.serialization.ByteArrayDeserializer
enable.auto.commit: true
bootstrap.servers: 10.0.0.80:9092,10.0.1.244:9092,10.0.0.6:9092
auto.offset.reset: earliest

Topic Configuration

min.insync.replicas: 1
retention.ms: 86400000
flush.messages: 1

replicationFactor

[AutoMQ] 1
[Kafka] 3

Average Throughput

Average Throughput [AutoMQ]: 500.18 MB/s
Average Throughput [Kafka]: 500.18 MB/s
Average Throughput [AWS MSK]: 500.12 MB/s

Cost Estimate Rule: Check explanation under cost-explanation directory of this repository

Kafka Provider Pub Latency (ms) avg Pub Latency (ms) P99 E2E LatencyAvg(ms) E2E P95 Latency(ms) E2E P99 Latency(ms) Baseline Cost($) Usage Cost($) Total Cost($)
AutoMQ 3.36 14.6 6.65 15.53 34.93 1107.35 1264.00 2371.35
Apache Kafka 4.8 40.48 7.22 16.21 50.52 10639.35 0.00 10639.35
AWS MSK 164.16 794.8 151.26 439.94 757.91 2312.35 12657.00 14969.35

@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 20, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 20, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 21, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 21, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 24, 2024
@github-actions GitHub Actions
Copy link

AutoMQ Benchmark VS. Result 🚀

Benchmark Info

Report Generated: 2024-06-24 04:22:22

Workload Configuration

name: 1-topic-1000-partitions-4kb-4p4c-500m
topics: 1
partitionsPerTopic: 1000
partitionsPerTopicList: null
randomTopicNames: true
keyDistributor: NO_KEY
messageSize: 4096
useRandomizedPayloads: false
randomBytesRatio: 0
randomizedPayloadPoolSize: 0
payloadFile: payload/payload-4Kb.data
subscriptionsPerTopic: 1
producersPerTopic: 8
producersPerTopicList: null
consumerPerSubscription: 4
producerRate: 100
producerRateList: null
consumerBacklogSizeGB: 0
backlogDrainRatio: 1
testDurationMinutes: 1
warmupDurationMinutes: 1
logIntervalMillis: 100

Producer Configuration

value.serializer: org.apache.kafka.common.serialization.ByteArraySerializer
acks: all
batch.size: 65536
bootstrap.servers: 10.0.0.91:9092
key.serializer: org.apache.kafka.common.serialization.StringSerializer
linger.ms: 1

Consumer Configuration

key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
value.deserializer: org.apache.kafka.common.serialization.ByteArrayDeserializer
enable.auto.commit: true
bootstrap.servers: 10.0.0.91:9092
auto.offset.reset: earliest

Topic Configuration

min.insync.replicas: 1
retention.ms: 86400000
flush.messages: 1

replicationFactor

[AutoMQ] 1
[Kafka] 3

Average Throughput

Average Throughput [AutoMQ]: 0.00 MB/s
Average Throughput [Kafka]: 0.00 MB/s
Average Throughput [AWS MSK]: 0.00 MB/s

Cost Estimate Rule: Check explanation under cost-explanation directory of this repository

Elasticity

Take how many seconds to move 30 partitions from one broker to another broker after write 750GB data
[AutoMQ]: 3 seconds
[Kafka]: 3 seconds
[Amazon MSK]: seconds

Kafka Provider Pub Latency (ms) avg Pub Latency (ms) P99 E2E LatencyAvg(ms) E2E P95 Latency(ms) E2E P99 Latency(ms) Baseline Cost($) Usage Cost($) Total Cost($)
AutoMQ 3.05 272.21 3.83 4.29 5.72 -94.65 1263.00 1168.35
Apache Kafka 2.54 314.71 3.05 3.57 7.23 1754.35 0.00 1754.35
AWS MSK 15.34 69.36 68.50 126.65 160.68 551.35 15.00 566.35

@github-actions GitHub Actions
Copy link

AutoMQ Benchmark VS. Result 🚀

Benchmark Info

Report Generated: 2024-06-24 06:31:21

Workload Configuration

name: 1-topic-1000-partitions-4kb-4p4c-500m
topics: 1
partitionsPerTopic: 1000
partitionsPerTopicList: null
randomTopicNames: true
keyDistributor: NO_KEY
messageSize: 4096
useRandomizedPayloads: false
randomBytesRatio: 0
randomizedPayloadPoolSize: 0
payloadFile: payload/payload-4Kb.data
subscriptionsPerTopic: 1
producersPerTopic: 8
producersPerTopicList: null
consumerPerSubscription: 4
producerRate: 100
producerRateList: null
consumerBacklogSizeGB: 0
backlogDrainRatio: 1
testDurationMinutes: 1
warmupDurationMinutes: 1
logIntervalMillis: 100

Producer Configuration

value.serializer: org.apache.kafka.common.serialization.ByteArraySerializer
acks: all
batch.size: 65536
bootstrap.servers: 10.0.0.91:9092
key.serializer: org.apache.kafka.common.serialization.StringSerializer
linger.ms: 1

Consumer Configuration

key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
value.deserializer: org.apache.kafka.common.serialization.ByteArrayDeserializer
enable.auto.commit: true
bootstrap.servers: 10.0.0.91:9092
auto.offset.reset: earliest

Topic Configuration

min.insync.replicas: 1
retention.ms: 86400000
flush.messages: 1

replicationFactor

[AutoMQ] 1
[Kafka] 3

Average Throughput

Average Throughput [AutoMQ]: 0.00 MB/s
Average Throughput [Kafka]: 0.00 MB/s
Average Throughput [AWS MSK]: 0.00 MB/s

Cost Estimate Rule: Check explanation under cost-explanation directory of this repository

Elasticity

Take how many seconds to move 30 partitions from one broker to another broker after write 750GB data
[AutoMQ]: 3 seconds
[Kafka]: 3 seconds
[Amazon MSK]: 3 seconds

Kafka Provider Pub Latency (ms) avg Pub Latency (ms) P99 E2E LatencyAvg(ms) E2E P95 Latency(ms) E2E P99 Latency(ms) Baseline Cost($) Usage Cost($) Total Cost($)
AutoMQ 2.71 752.37 3.14 3.84 4.88 -94.65 1263.00 1168.35
Apache Kafka 2.28 297.82 2.37 2.86 6.32 1754.35 0.00 1754.35
AWS MSK 16.31 68 70.27 124.31 155.37 551.35 15.00 566.35

@github-actions GitHub Actions
Copy link

AutoMQ Benchmark VS. Result 🚀

Benchmark Info

Report Generated: 2024-06-24 10:21:20

Workload Configuration

name: 1-topic-1000-partitions-4kb-4p4c-500m
topics: 1
partitionsPerTopic: 1000
partitionsPerTopicList: null
randomTopicNames: true
keyDistributor: NO_KEY
messageSize: 4096
useRandomizedPayloads: false
randomBytesRatio: 0
randomizedPayloadPoolSize: 0
payloadFile: payload/payload-4Kb.data
subscriptionsPerTopic: 1
producersPerTopic: 8
producersPerTopicList: null
consumerPerSubscription: 4
producerRate: 25600
producerRateList: null
consumerBacklogSizeGB: 0
backlogDrainRatio: 1
testDurationMinutes: 5
warmupDurationMinutes: 1
logIntervalMillis: 100

Producer Configuration

value.serializer: org.apache.kafka.common.serialization.ByteArraySerializer
acks: all
batch.size: 65536
bootstrap.servers: 10.0.0.179:9092
key.serializer: org.apache.kafka.common.serialization.StringSerializer
linger.ms: 1

Consumer Configuration

key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
value.deserializer: org.apache.kafka.common.serialization.ByteArrayDeserializer
enable.auto.commit: true
bootstrap.servers: 10.0.0.179:9092
auto.offset.reset: earliest

Topic Configuration

min.insync.replicas: 1
retention.ms: 86400000
flush.messages: 1

replicationFactor

[AutoMQ] 1
[Kafka] 3

Average Throughput

Average Throughput [AutoMQ]: 101.00 MB/s
Average Throughput [Kafka]: 100.91 MB/s
Average Throughput [AWS MSK]: 101.35 MB/s

Cost Estimate Rule: Check explanation under cost-explanation directory of this repository

Elasticity

Take how many seconds to move 30 partitions from one broker to another broker after write 36GB data
[AutoMQ]: 4 seconds
[Kafka]: 4 seconds
[Amazon MSK]: 3 seconds

Kafka Provider Pub Latency (ms) avg Pub Latency (ms) P99 E2E LatencyAvg(ms) E2E P95 Latency(ms) E2E P99 Latency(ms) Baseline Cost($) Usage Cost($) Total Cost($)
AutoMQ 8.08 30.62 12.08 26.96 45.22 277.07 1263.00 1540.07
Apache Kafka 17.98 75.13 16.30 39.97 95.52 2126.07 0.00 2126.07
AWS MSK 93.31 227.51 211.29 356.83 460.66 923.07 2531.00 3455.07

@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 24, 2024
@AutoMQ AutoMQ deleted a comment from github-actions bot Jun 24, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 1, 2024

AutoMQ Benchmark VS. Result 🚀

Benchmark Info

Report Generated: 2024-07-01 00:56:32

Workload Configuration

name: 1-topic-1000-partitions-4kb-4p4c-500m
topics: 1
partitionsPerTopic: 1000
partitionsPerTopicList: null
randomTopicNames: true
keyDistributor: NO_KEY
messageSize: 4096
useRandomizedPayloads: false
randomBytesRatio: 0
randomizedPayloadPoolSize: 0
payloadFile: payload/payload-4Kb.data
subscriptionsPerTopic: 1
producersPerTopic: 8
producersPerTopicList: null
consumerPerSubscription: 4
producerRate: 25600
producerRateList: null
consumerBacklogSizeGB: 0
backlogDrainRatio: 1
testDurationMinutes: 5
warmupDurationMinutes: 1
logIntervalMillis: 100

Producer Configuration

value.serializer: org.apache.kafka.common.serialization.ByteArraySerializer
acks: all
batch.size: 65536
bootstrap.servers: 10.0.0.161:9092
key.serializer: org.apache.kafka.common.serialization.StringSerializer
linger.ms: 1

Consumer Configuration

key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
value.deserializer: org.apache.kafka.common.serialization.ByteArrayDeserializer
enable.auto.commit: true
bootstrap.servers: 10.0.0.161:9092
auto.offset.reset: earliest

Topic Configuration

min.insync.replicas: 1
retention.ms: 86400000
flush.messages: 1

replicationFactor

[AutoMQ] 1
[Kafka] 3

Average Throughput

Average Throughput [AutoMQ]: 100.93 MB/s
Average Throughput [Kafka]: 101.03 MB/s
Average Throughput [AWS MSK]: 101.29 MB/s

Cost Estimate Rule: Check explanation under cost-explanation directory of this repository

Elasticity

Take how many seconds to move 30 partitions from one broker to another broker after write 36GB data
[AutoMQ]: 4 seconds
[Kafka]: 3 seconds
[Amazon MSK]: 3 seconds

Kafka Provider Pub Latency (ms) avg Pub Latency (ms) P99 E2E LatencyAvg(ms) E2E P95 Latency(ms) E2E P99 Latency(ms) Baseline Cost($) Usage Cost($) Total Cost($)
AutoMQ 8.51 30.92 12.13 26.54 44.28 277.07 1263.00 1540.07
Apache Kafka 17.31 69.83 17.04 42.30 96.59 2126.07 0.00 2126.07
AWS MSK 78.05 216.78 220.35 397.10 654.36 923.07 2531.00 3455.07

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 8, 2024

AutoMQ Benchmark VS. Result 🚀

Benchmark Info

Report Generated: 2024-07-08 01:27:42

Workload Configuration

name: 1-topic-1000-partitions-4kb-4p4c-500m
topics: 1
partitionsPerTopic: 1000
partitionsPerTopicList: null
randomTopicNames: true
keyDistributor: NO_KEY
messageSize: 4096
useRandomizedPayloads: false
randomBytesRatio: 0
randomizedPayloadPoolSize: 0
payloadFile: payload/payload-4Kb.data
subscriptionsPerTopic: 1
producersPerTopic: 8
producersPerTopicList: null
consumerPerSubscription: 4
producerRate: 25600
producerRateList: null
consumerBacklogSizeGB: 0
backlogDrainRatio: 1
testDurationMinutes: 30
warmupDurationMinutes: 5
logIntervalMillis: 500

Producer Configuration

value.serializer: org.apache.kafka.common.serialization.ByteArraySerializer
acks: all
batch.size: 65536
bootstrap.servers: 10.0.0.128:9092
key.serializer: org.apache.kafka.common.serialization.StringSerializer
linger.ms: 1

Consumer Configuration

key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
value.deserializer: org.apache.kafka.common.serialization.ByteArrayDeserializer
enable.auto.commit: true
bootstrap.servers: 10.0.0.128:9092
auto.offset.reset: earliest

Topic Configuration

min.insync.replicas: 1
retention.ms: 86400000
flush.messages: 1

replicationFactor

[AutoMQ] 1
[Kafka] 3
[MSK] 3

Average Throughput

Average Throughput [AutoMQ]: 100.08 MB/s
Average Throughput [Kafka]: 100.10 MB/s
Average Throughput [AWS MSK]: 100.10 MB/s

Cost Estimate Rule: Check explanation under cost-explanation directory of this repository

Elasticity

Take how many seconds to move 30 partitions from one broker to another broker after write 210 GB data (100 MB/s for 35 minutes)
[AutoMQ]: 3 seconds
[Kafka]: 4 seconds
[Amazon MSK]: 3 seconds

Kafka Provider Pub Latency (ms) avg Pub Latency (ms) P99 E2E LatencyAvg(ms) E2E P95 Latency(ms) E2E P99 Latency(ms) Baseline Cost($) Usage Cost($) Total Cost($)
AutoMQ 6.81 25.36 10.22 19.37 35.26 276.07 1262.00 1539.07
Apache Kafka 12.3 72.54 14.82 37.17 99.12 2124.07 0.00 2124.07
AWS MSK 104.53 361.96 211.21 412.89 590.23 923.07 2531.00 3455.07

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@KaimingWan