nightly

Automated nightly release, will be updated every day at 20:00 UTC

1.5.0

New Feature

• Support produce / consume zone alignment to save inter zone traffic cost.
• Support table topic to seamless ingest Kafka records to Iceberg table.
• Support deploy on Kubernetes.

What's Changed

• feat(zerozone): support zero cross az traffic cost by @superhx in #2506
• fix(s3stream): fix backward compatibility of default aws credentials by @SCNieh in #2510
• feat(table_topic): add table topic support (#2511) by @superhx in #2513
• feat(failover): add wal failover support (#2516) by @superhx in #2517
• fix(test): fix e2e dependencies conflict (#2518) by @superhx in #2519
• feat(cli): add default s3 wal config by @SCNieh in #2512
• feat(table): validate table config by @superhx in #2523
• chore(test): increase timeout for write completion in AbstractObjectSto… by @Gezi-lzq in #2536
• feat(zerozone): fast upload when snapshot-read enable by @superhx in #2534
• fix(storage): fix upload wal rate missing by @superhx in #2528
• fix(s3stream): add delayed deletion for S3 WAL by @SCNieh in #2525
• fix(s3stream/wal): fix incorrect offset return value during recovery by @Chillax-0v0 in #2543
• fix(github-release): correct IS_LATEST variable usage in upload condi… by @Gezi-lzq in #2538
• refactor(s3stream/object-wal): complete appends sequentially by @Chillax-0v0 in #2549
• fix(perf): unify the time in different nodes by @superhx in #2554
• feat(config): remove client logs from stdout by @superhx in #2565
• feat(docker): add Spark Iceberg Docker setup and demonstration notebook by @Gezi-lzq in #2567
• chore(log): log readiness check result to stdout by @superhx in #2568
• feat: cherry-pick 2560 by @1sonofqiu in #2571
• feat(container): cp_2561_2570_2562_2574 by @1sonofqiu in #2575
• feat(docker): cherry pick #2468 #2469 by @superhx in #2576
• feat(release): release 1.5.0 by @superhx in #2577

Full Changelog: 1.4.0...1.5.0

NOTE: 1.5.0 and later versions offer full S3 support, while EBS and Regional EBS support have been moved to the Enterprise Edition.

1.4.0

What's Changed

• feat(table): enhance command utils by @superhx in #2217
• fix(tools/perf): fix the perf tools await count by @superhx in #2220
• feat(tools/perf): support schema message perf by @superhx in #2227
• feat(telemetry): support gzip compression on uploading metrics & logs… by @SCNieh in #2222
• fix(core): fix potential infinite recursion on reading empty segment by @SCNieh in #2229
• feat(version): bump to 1.3.1-rc0 by @superhx in #2234
• feat: support backpressure by @Chillax-0v0 in #2236
• feat(table): cherry table topic aspect by @superhx in #2237
• feat(stream): exponential backoff retry by @superhx in #2239
• feat(config): add table topic conversion type configuration (#2203) by @Gezi-lzq in #2240
• fix(s3stream): report compaction delay after two compaction period by @SCNieh in #2242
• feat(tools/perf): user experience optimization by @Chillax-0v0 in #2244
• feat(table): support partition & upsert config by @superhx in #2248
• chore(config): change s3.stream.object.compaction.max.size.bytes default value from 1GiB to 10GiB by @superhx in #2250
• fix(cli/deploy): override "controller.quorum.bootstrap.servers" by @Chillax-0v0 in #2252
• chore(version): bump version to 1.3.1 by @Chillax-0v0 in #2254
• perf(produce): fix validate compressed records alloc too many memory by @superhx in #2257
• chore(version): bump version to 1.3.2-rc0 by @superhx in #2260
• fix(s3stream): fix compaction block on upload exception by @SCNieh in #2264
• feat(config): fix exporter uri type by @SCNieh in #2267
• feat(tools/perf): support to limit the max poll rate of consumers by @Chillax-0v0 in #2271
• fix(s3stream/wal): check isBlockDev by prefix in some rare cases by @Chillax-0v0 in #2278
• fix(tools/perf): fix option name "--max-consume-record-rate" by @Chillax-0v0 in #2273
• refactor(controller): consider brokers that has recently CONTROLLED_SHUTDOWN as SHUTTING_DOWN by @Chillax-0v0 in #2262
• fix(core): add missing setting method by @SCNieh in #2279
• feat(s3stream): fast fail s3 request by @superhx in #2281
• chore(s3stream): replace eventloop with executor in async semaphore by @superhx in #2283
• fix(tools/perf): only delete test topics on reset by @Chillax-0v0 in #2285
• fix(s3stream): halt the process when node is fenced by @superhx in #2292
• fix(network): adjust number of permits if the request is huge by @Chillax-0v0 in #2295
• fix(s3stream/wal): increase max record size from 16MiB to 64MiB by @Chillax-0v0 in #2297
• fix(s3stream/wal): increase max record size from 64MiB to 128MiB by @Chillax-0v0 in #2299
• feat(s3stream/wal): add constraints in recovery mode by @Chillax-0v0 in #2304
• feat(s3stream/failover): check request epoch >= wal epoch by @Chillax-0v0 in #2305
• feat(tool/perf): add option "--common-config-file" by @Chillax-0v0 in #2309
• fix(tool/perf): add admin properties in ConsumerService#admin by @Chillax-0v0 in #2311
• fix(action): change upload bucket by @superhx in #2313
• fix(s3stream): skip waiting for pending part on release (#2316) by @SCNieh in #2319
• fix(action): fix release action by @superhx in #2324
• feat(release): bump version to 1.3.2 by @superhx in #2330
• feat(core): add kafka linking interface by @SCNieh in #2336
• perf(s3stream): limit write traffic to object storage by @Chillax-0v0 in #2335
• fix(s3storage): release Bytebuf allocated by WAL earlier to prevent memory fragmentation by @Chillax-0v0 in #2342
• refactor(controller): add method ControllerServer#reconfigurables by @Chillax-0v0 in #2346
• refactor(core): refine config names for kafka linking (#2348) by @SCNieh in #2349
• refactor(automq): rename producerouter to traffic interceptor by @superhx in #2353
• feat(linking): use linkId for update group API by @SCNieh in #2352
• feat(release): bump version to 1.3.3-rc0 by @superhx in #2355
• feat(s3stream): support s3 write timeout (#2356) by @superhx in #2361
• perf(s3stream/objectstorage): unify the throttle criteria by @Chillax-0v0 in #2396
• chore(test): add test timeout by @superhx in #2411
• feat(release): release 1.3.3 by @superhx in #2412
• feat(multi_read): add get partitions api (#2338) by @superhx in #2345
• feat(circuit): support node circuit breaker (#2409) by @superhx in #2413
• perf(s3stream/wal): add append timeout in block WAL by @Chillax-0v0 in #2399
• feat(circuit): add LocalFileObjectStorage storage limit (#2415) by @superhx in #2417
• feat(circuit): prevent unregister locked node (#2418) by @superhx in #2419
• feat(interceptor): extend client id by @superhx in #2422
• feat(s3stream): add trigger wal upload interval (#2347) by @superhx in #2423
• fix(snapshot_read): prevent append by @superhx in #2425
• fix(s3stream): fix potential index leak on stream deletion by @SCNieh in #2429
• chore(license): use the apache license for the next major version by @daniel-y in #2434
• feat(snapshot_read): support preferred node (#2436) by @superhx in #2437
• feat(metrics): cherry pick add cert metrics 1.4 by @woshigaopp in #2439
• fix(gradle): fix kafka-client conflict #2445 by @superhx in #2446
• feat(snapshot_read): snapshot-read cache by @superhx in #2453
• fix(snapshot_read): fix snapshot-read cache tryload trigger by @superhx in #2460
• fix(storage): ensure proper reference counting for ByteBuf in write o… by @Gezi-lzq in #2499
• fix(s3stream): add delayed deletion for S3 WAL by @SCNieh in #2527
• fix(storage): fix upload wal rate missing by @superhx in #2529
• fix(s3stream/wal): fix incorrect offset return value during recovery by @Chillax-0v0 in #2544
• fix(github-release): correct IS_LATEST variable usage in upload condi… by @Gezi-lzq in #2540
• refactor(s3stream/object-wal): complete appends sequentially by @Chillax-0v0 in #2426
• feat(release): release 1.4.0 by @superhx in #2559

Full Changelog: 1.3.0...1.4.0

1.3.4

What's Changed

• fix(s3stream): fix potential index leak on stream deletion by @SCNieh in #2428
• fix(gradle): fix kafka-client conflict by @superhx in #2444
• feat(cmp): cherry pick add cert metrics 1.3 by @woshigaopp in #2451
• fix(storage): ensure proper reference counting for ByteBuf in write operations(#2452) by @Gezi-lzq in #2500
• feat(release): release 1.3.4-rc0 by @superhx in #2508
• fix(storage): fix upload wal rate missing by @superhx in #2530
• fix(s3stream): add delayed deletion for S3 WAL by @SCNieh in #2531
• fix(github-release): correct IS_LATEST variable usage in upload condi… by @Gezi-lzq in #2539
• fix(s3stream/wal): fix incorrect offset return value during recovery by @Chillax-0v0 in #2545
• refactor(s3stream/object-wal): complete appends sequentially by @Chillax-0v0 in #2550
• feat(release): release 1.3.4 by @superhx in #2558

Full Changelog: 1.3.3...1.3.4

1.3.4-rc0

What's Changed

• fix(s3stream): fix potential index leak on stream deletion by @SCNieh in #2428
• fix(gradle): fix kafka-client conflict by @superhx in #2444
• feat(cmp): cherry pick add cert metrics 1.3 by @woshigaopp in #2451
• fix(storage): ensure proper reference counting for ByteBuf in write operations(#2452) by @Gezi-lzq in #2500
• feat(release): release 1.3.4-rc0 by @superhx in #2508

Full Changelog: 1.3.3...1.3.4-rc0

1.3.3

What's Changed

• feat(core): add kafka linking interface by @SCNieh in #2336
• perf(s3stream): limit write traffic to object storage by @Chillax-0v0 in #2335
• fix(s3storage): release Bytebuf allocated by WAL earlier to prevent memory fragmentation by @Chillax-0v0 in #2342
• refactor(controller): add method ControllerServer#reconfigurables by @Chillax-0v0 in #2346
• refactor(core): refine config names for kafka linking (#2348) by @SCNieh in #2349
• refactor(automq): rename producerouter to traffic interceptor by @superhx in #2353
• feat(linking): use linkId for update group API by @SCNieh in #2352
• feat(release): bump version to 1.3.3-rc0 by @superhx in #2355
• feat(s3stream): support s3 write timeout (#2356) by @superhx in #2361
• perf(s3stream/objectstorage): unify the throttle criteria by @Chillax-0v0 in #2396
• chore(test): add test timeout by @superhx in #2411
• feat(release): release 1.3.3 by @superhx in #2412

Full Changelog: 1.3.2...1.3.3

1.3.3-rc0

What's Changed

• feat(core): add kafka linking interface by @SCNieh in #2336
• perf(s3stream): limit write traffic to object storage by @Chillax-0v0 in #2335
• fix(s3storage): release Bytebuf allocated by WAL earlier to prevent memory fragmentation by @Chillax-0v0 in #2342
• refactor(controller): add method ControllerServer#reconfigurables by @Chillax-0v0 in #2346
• refactor(core): refine config names for kafka linking (#2348) by @SCNieh in #2349
• refactor(automq): rename producerouter to traffic interceptor by @superhx in #2353
• feat(linking): use linkId for update group API by @SCNieh in #2352
• feat(release): bump version to 1.3.3-rc0 by @superhx in #2355

Full Changelog: 1.3.2...1.3.3-rc0

1.3.2

What's Changed

• perf(produce): fix validate compressed records alloc too many memory by @superhx in #2257
• chore(version): bump version to 1.3.2-rc0 by @superhx in #2260
• fix(s3stream): fix compaction block on upload exception by @SCNieh in #2264
• feat(config): fix exporter uri type by @SCNieh in #2267
• feat(tools/perf): support to limit the max poll rate of consumers by @Chillax-0v0 in #2271
• fix(s3stream/wal): check isBlockDev by prefix in some rare cases by @Chillax-0v0 in #2278
• fix(tools/perf): fix option name "--max-consume-record-rate" by @Chillax-0v0 in #2273
• refactor(controller): consider brokers that has recently CONTROLLED_SHUTDOWN as SHUTTING_DOWN by @Chillax-0v0 in #2262
• fix(core): add missing setting method by @SCNieh in #2279
• feat(s3stream): fast fail s3 request by @superhx in #2281
• chore(s3stream): replace eventloop with executor in async semaphore by @superhx in #2283
• fix(tools/perf): only delete test topics on reset by @Chillax-0v0 in #2285
• fix(s3stream): halt the process when node is fenced by @superhx in #2292
• fix(network): adjust number of permits if the request is huge by @Chillax-0v0 in #2295
• fix(s3stream/wal): increase max record size from 16MiB to 64MiB by @Chillax-0v0 in #2297
• fix(s3stream/wal): increase max record size from 64MiB to 128MiB by @Chillax-0v0 in #2299
• feat(s3stream/wal): add constraints in recovery mode by @Chillax-0v0 in #2304
• feat(s3stream/failover): check request epoch >= wal epoch by @Chillax-0v0 in #2305
• feat(tool/perf): add option "--common-config-file" by @Chillax-0v0 in #2309
• fix(tool/perf): add admin properties in ConsumerService#admin by @Chillax-0v0 in #2311
• fix(action): change upload bucket by @superhx in #2313
• fix(s3stream): skip waiting for pending part on release (#2316) by @SCNieh in #2319
• fix(action): fix release action by @superhx in #2324
• feat(release): bump version to 1.3.2 by @superhx in #2330

Full Changelog: 1.3.1...1.3.2

1.2.2

What's Changed

• chore(github): update code owners by @Chillax-0v0 in #2157
• fix(stream): release FetchResults if the subsequent fetch fails by @Chillax-0v0 in #2175
• fix: release PooledMemoryRecords if it's dropped in the fetch session by @Chillax-0v0 in #2188
• fix: use the "adjusted" maxSize in ElasticLogSegment#readAsync by @Chillax-0v0 in #2191
• fix(issues2193): retry 2 times to cover most of BlockNotContinuousException by @superhx in #2196
• fix(docker): use bucket url by @superhx in #2214
• feat(telemetry): support gzip compression on uploading metrics & logs… by @SCNieh in #2224
• fix(core): fix potential infinite recursion on reading empty segment by @SCNieh in #2231
• feat(gradle): bump version to 1.2.2-rc0 by @superhx in #2245
• chore(config): change s3.stream.object.compaction.max.size.bytes default value from 1GiB to 10GiB by @superhx in #2251
• fix(s3stream): fix compaction block on upload exception by @SCNieh in #2265
• feat(release): bump version to 1.2.2 by @superhx in #2293
• fix(action): change upload bucket by @superhx in #2314
• fix(action): add upload endpoint by @superhx in #2317
• fix(action): try fix docker build with update platform by @superhx in #2318
• fix(action): downgrade qemu by @superhx in #2320

Full Changelog: 1.2.1...1.2.2

1.3.2-rc0

What's Changed

• perf(produce): fix validate compressed records alloc too many memory by @superhx in #2257
• chore(version): bump version to 1.3.2-rc0 by @superhx in #2260

Full Changelog: 1.3.1...1.3.2-rc0