v23.3.5

Features

• #16411 You can print a schema now using rpk registry schema get --print-schema. by @r-vasquez in #16412
• PR #16292 [v23.3.x] "enable by default spillover manifest" testing followups by @andijcr

Bug Fixes

• Aggregates partitions in some cloud storage metrics when the aggregate_metrics cluster config is set to true. by @ballard26 in #16344
• #16350 rpk: fixed a bug where the --password flag could not be used along with the new configuration flag -X pass in clusters where basic authentication was enabled. by @r-vasquez in #16351
• #16389 Prevent oversized allocation with large amounts of controller metadata by @rockwotj in #16390
• #16391 Fixes a bug that may prevent redpanda from shutting down cleanly when auditing is enabled by @graphcareful in #16392
• #16393 Fix graceful shutdown of the TS archive area retention procedure. by @Lazin in #16394
• #16404 Fixes issue that causes the connection to hang when an unsupported compression type is passed via an incremental_alter_configs request by @graphcareful in #16406
• #16450 #16451 Fix an issue where create topics responses would show incorrect partition count and replication factor by @oleiman in #16452
• #16500 Fix assertion triggered by interleaving of log flush and log truncation followed by append by @Lazin in #16501
• #16517 Fix timequery error that triggered full partition scan by @Lazin in #16518

Improvements

• Adds a new cluster configuration property fetch_read_strategy. This property determines which fetch execution strategy Redpanda will use to fulfill a fetch request. The newly introduced non_polling execution strategy is the default for this property with the polling strategy being included to make backporting possible. by @ballard26 in #16484

• Improved handling of follower fetching offset validation when used with relaxed consistency by @mmaslankaprv in #16522

• Improves observability by allowing Redpanda to detect that some internal processes are stuck. by @Lazin in #16476

• Introduces a new non-polling fetch execution strategy that decreases CPU utilization of fetch requests and fetch request latency. by @ballard26 in #16484

• Publish total reclaimable space to avoid stuck decommission scenario. by @dotnwat in #16422

• SIMD instructions are generated by default for WebAssembly binaries when building with rpk. by @rockwotj in #16403

• • #16362 rpk cluster health: now --exit-when-healthy enables --watch when provided. by @r-vasquez in #16363

• PR #16294 [v23.3.x] transform-sdk/rust: borrow output record by @rockwotj

• PR #16300 [v23.3.x] Transform logging data model by @oleiman

• PR #16316 [v23.3.x] transform-sdk/rust: update rustdocs by @rockwotj

• PR #16345 [v23.3.x] Wrapped logging with vlog macro in places that missed it by @mmaslankaprv

• PR #16401 [v23.3.x] cloud_storage_clients/client_pool: handle broken _self_config_barrier by @andijcr

• PR #16432 [v23.3.x] Fixed large allocation in kafka::wait_for_leaders by @mmaslankaprv

• PR #16436 [v23.3.x] Introduce transform::logging::manager by @oleiman

• PR #16439 [v23.3.x] rptest: Fix s3.copy_object when running on GCP by @savex

• PR #16492 [v23.3.x] k/metadata: guesstimate leader when information is not yet present by @mmaslankaprv

• PR #16528 Use the original fetch impl by default in backports by @ballard26

• PR #16531 [v23.3.x] Increase default value of rpc_client_connections_per_peer to 32 by @ballard26

• Full Changelog: v23.3.4...v23.3.5

v23.2.25

Bug Fixes

• #16348 rpk: fixed a bug where the --password flag could not be used along with the new configuration flag -X pass in clusters where basic authentication was enabled. by @r-vasquez in #16349
• #16387 Prevent oversized allocation with large amounts of controller metadata by @rockwotj in #16388
• #16405 Fixes issue that causes the connection to hang when an unsupported compression type is passed via an incremental_alter_configs request by @graphcareful in #16407
• #16447 #16448 Fix an issue where create topics responses would show incorrect partition count and replication factor by @oleiman in #16449
• #16479 Fix timequery error that triggered full partition scan by @Lazin in #16520
• PR #16435 [v23.2.x] Fixed large allocation in kafka::wait_for_leaders by @mmaslankaprv

Improvements

• Improved handling of follower fetching offset validation when used with relaxed consistency by @mmaslankaprv in #16530
• Improves observability by allowing Redpanda to detect that some internal processes are stuck. by @Lazin in #16477
• #16423 Publish total reclaimable space to avoid stuck decommission scenario. by @dotnwat in #16442
• PR #16219 [v23.2.x] Introduced partition shutdown watchdog timer by @mmaslankaprv
• PR #16494 [v23.2.x] Wrapped logging with vlog macro in places that missed it by @mmaslankaprv

Full Changelog: v23.2.24...v23.2.25

v23.3.4

Bug Fixes

• Fix the starter code for Rust projects in rpk transform init by @rockwotj in #16194
• Report runtime public metrics by task queue for all cores, not just core 0 by @rockwotj in #16203
• #16271 Fixes a bug that would previously cause read replicas to report the wrong value for the redpand_kafka_max_offset metric. by @andrwng in #16272

Improvements

• smaller memory footprint when using with large number of topics with small partition count by @mmaslankaprv in #16266
• PR #16182 [v23.3.x] Rename max_client_count to max_connection_count by @travisdowns
• PR #16195 [v23.3.x] Introduce wasm::logger by @oleiman
• PR #16216 [v23.3.x] Introduced partition shutdown watchdog timer by @mmaslankaprv
• PR #16236 [v23.3.x] c/topics_dispatcher: do not guesstimate leader ids by @mmaslankaprv
• PR #16255 [v23.3.x] admin api: skip partition info in /brokers end point by @bharathv

Full Changelog: v23.3.3...v23.3.4

v23.2.24

Bug Fixes

• #16274 Fixes a bug that would previously cause read replicas to report the wrong value for the redpand_kafka_max_offset metric. by @andrwng in #16275
• PR #16221 [v23.2.x] Fixed skipping application of raft snapshot by @mmaslankaprv
• PR #16239 [v23.2.x] c/topics_dispatcher: do not guesstimate leader ids by @mmaslankaprv
• PR #16282 [v23.2.x] cloud_storage: hold gate in hydration by @andrwng

Improvements

• smaller memory footprint when using with large number of topics with small partition count by @mmaslankaprv in #16267
• PR #16254 [v23.2.x] admin api: skip partition info in /brokers end point by @bharathv

Full Changelog: v23.2.23...v23.2.24

v23.3.3

Features

• spillover manifests are enabled by default for clusters that did not explicit set a value or null by @andijcr in #16174

Bug Fixes

• PR #16178 [v23.3.x] c/log_eviction_stm: do not request snapshot if already progressed by @mmaslankaprv
• Fix internal RPC client connection stall after more than 2^32 requests are sent. by @ztlpn in #16176
• Fix large allocation in partition manifest. by @dotnwat in #16188
• Fix tiered-storage housekeeping problem that may cause replaced segments to pile up if the spillover is enabled. by @Lazin in #16167
• Fix tiered-storage housekeeping problem that may cause replaced segments to pile up if the spillover is enabled. by @Lazin in #16170
• Protect against a very rare scenario where after node restart, some of the partition replicas hosted on that node could not take part in leader elections. by @ztlpn in #16080
• #15811 Several additional metrics will have their "partition" label aggregated away (i.e., into a single series per remaining label set with no partition label, whose value is the sum of all input series with the same label set and different partition labels). This is already the default behavior for most metrics, but this change extends it to almost all remaining metrics. by @travisdowns in #16094
• #16093 Several additional metrics will have their "partition" label aggregated away (i.e., into a single series per remaining label set with no partition label, whose value is the sum of all input series with the same label set and different partition labels). This is already the default behavior for most metrics, but this change extends it to almost all remaining metrics. by @travisdowns in #16100
• fixed incorrect fetch offset validation by @mmaslankaprv in #16169

Improvements

• Add a dedicated CPU scheduling policy for Data Transforms by @rockwotj in #16139
• Reduces the number of allocations performed by the auditing subsystem by @graphcareful in #16147
• rpk transform deploy --file now supports https:// URLs by @rockwotj in #16063
• PR #16111 [v23.3.x] archival: Start housekeeping jobs after STM sync by @Lazin
• PR #16103 [v23.3.x] Add at: in Top-N alloc site output by @travisdowns

Full Changelog: v23.3.2...v23.3.3

v23.2.23

Bug Fixes

• Fix internal RPC client connection stall after more than 2^32 requests are sent. by @ztlpn in #16175
• Fix large allocation in partition manifest. by @dotnwat in #16191
• Fix tiered-storage housekeeping problem that may cause replaced segments to pile up if the spillover is enabled. by @Lazin in #16166
• Fix tiered-storage housekeeping problem that may cause replaced segments to pile up if the spillover is enabled. by @Lazin in #16171
• Have fetch handler ensure rack awareness is enabled before performing follower fetching by @michael-redpanda in #15914
• Protect against a very rare scenario where after node restart, some of the partition replicas hosted on that node could not take part in leader elections. by @ztlpn in #16082
• #15839 safer handle unknown properties in local state by @andijcr in #15874
• #15925 Prevent oversized allocs when group fetching from many partitions. by @rockwotj in #15926
• ext4 is no longer incorrectly detected as ext2 (all of ext2, 3 and 4 are assumed to be ext4). by @travisdowns in #15812
• fixed incorrect fetch offset validation by @mmaslankaprv in #16168
• PR #15816 [v23.2.x] c/archival_stm: do not reset _last_replicate on timeout by @nvartolomei
• PR #15989 [v23.2.x] r/offset_translator: remove unsafe bootstrap code by @ztlpn

Improvements

• #15829 Added new metric to provide Follower Fetching feature observability by @mmaslankaprv in #15831
• PR #16099 [v23.2.x] archival: Use explicit types to encode upload candidate creation result by @abhijat
• PR #16104 [v23.2.x] Add at: in Top-N alloc site output by @travisdowns

Full Changelog: v23.2.22...v23.2.23

v23.3.2

Bug Fixes

• Fix an issue with Cargo.toml when initializing a Rust Data Transform project via rpk transform init by @rockwotj in #15947
• Fixes a crash if a WebAssembly function is deployed that immediately crashes. by @rockwotj in #15943
• Fixes an an improper initialization of metrics related to controller snapshot uploads. by @andrwng in #16074
• Have fetch handler ensure rack awareness is enabled before performing follower fetching by @michael-redpanda in #15915
• Prevent an assertion from being triggered when Wasm VMs fail immediately. by @rockwotj in #15941
• Redpanda used to accept an empty string in redpanda.rack in node config. This would cause issues in Kafka operations. Redpanda will now error on startup if redpanda.rack is set to an empty string. by @michael-redpanda in #15849
• Redpanda will now correctly handle an empty rack ID provided in a fetch request by @michael-redpanda in #15860
• #15928 Prevent oversized allocs when group fetching from many partitions. by @rockwotj in #15929
• ext4 is no longer incorrectly detected as ext2 (all of ext2, 3 and 4 are assumed to be ext4). by @travisdowns in #15855
• safer handle unknown properties in local state by @andijcr in #15838

Improvements

• Caches the connections local address preventing the need to make a system calls to grab this value when auditing events. by @graphcareful in #15958
• Data Transforms written in Golang now use a non-buffered write mechanism. Transforms that used to be written as by @rockwotj in #15936
• Support changing the timeout for WebAssembly functions by @rockwotj in #15984
• Support dynamically changing the limit for WebAssembly binary size by @rockwotj in #15984
• This PR partially reverts the change such that strict retention remains enabled after upgrade unless it had been explicitly disabled before the upgrade. by @dotnwat in #16084
• #15974 Internal kafka client now uses asynchronous compression (when possible) to reduce possibility of oversized allocations and reactor stalls by @michael-redpanda in #15976
• [rpk] more informative error message display on create topic failure by @michael-redpanda in #15847
• rpk transform deploy takes a --file flag to deploy a compiled WebAssembly binary. by @rockwotj in #15954
• PR #15843 [v23.3.x] Increase audit buffer sizes for audit scale test by @graphcareful
• PR #15866 [v23.3.x] rptest: log error on failure to delete bucket by @andrwng
• PR #15867 [v23.3.x] rptest: allow the new version of xfs/ext4 fs error msg by @nvartolomei
• PR #15871 [v23.3.x] c/s/leader_balancer: prevent oversized alloc by @rockwotj
• PR #15876 [v23.3.x] t/kgo: upgrade kgo to do a full run after /last_pass by @nvartolomei
• PR #15885 [v23.3.x] gh/workflow: add build message in promote trigger by @gousteris
• PR #15888 [v23.3.x] gha: s/git.ref_name/github.ref_name by @rockwotj
• PR #15930 [v23.3.x] Skip assertion in audit log tests if results beat the baseline by @graphcareful
• PR #15970 [v23.3.x] dt: Fixed flaky schemas test by @michael-redpanda
• PR #15980 [v23.3.x] tx_migration: avoid ping pong of requests between brokers by @bharathv
• PR #15988 [v23.3.x] r/offset_translator: remove unsafe bootstrap code by @ztlpn
• PR #15998 [v23.3.x] c/partition_balancer: use full partition move when disk is full by @mmaslankaprv
• PR #15999 [v23.3.x] securit/:OIDC: Enable licence check and telemetry by @BenPope
• PR #16065 [v23.3.x] cloud_storage: Improve scrubber by @Lazin
• PR #16073 [v23.3.x] archival: avoid division by 0 when computing slow down rate by @nvartolomei
• PR #16052 [v23.3.x] storage: enable space management by default by @dotnwat

Full Changelog: v23.3.1...v23.3.2

v23.2.22

Bug Fixes

• Redpanda used to accept an empty string in redpanda.rack in node config. This would cause issues in Kafka operations. Redpanda will now error on startup if redpanda.rack is set to an empty string. by @michael-redpanda in #15848
• Redpanda will now correctly handle an empty rack ID provided in a fetch request by @michael-redpanda in #15861
• #15785 Fix a rare bug where http client connections would vanish from the connection pool leading to various operations hanging while waiting for an http client. by @nvartolomei in #15834
• PR #15873 [v23.2.x] c/s/leader_balancer: prevent oversized alloc by @rockwotj

Full Changelog: v23.2.21...v23.2.22

v23.3.1

Features

• Add recovery mode - an option to start redpanda in "metadata-only" mode, skipping loading user partitions and allowing only metadata operations to enable recovery from fatal misconfiguration or resource exhaustion situations. Enabled by the recovery_mode_enabled node config property. by @ztlpn in #14236
• Add broker support for SASL reauthentication. To enable, config connections_max_reauth_ms > 0. by @oleiman in #13822
• Adds cloud storage scrubbing capabilities to Redpanda. In brief, the scrubber runs in the background and verifies the integrity of the cloud storage metadata and the existence of data referenced by it. When an issue is discovered, the redpanda_cloud_storage_anomalies will increment its counters based on the anomaly type. Per partition anomalies can be queried via the v1/cloud_storage/anomalies/ admin API endpoint. by @VladLazar in #13253
• Add per-shard download throughput limit to tiered-storage by @Lazin in #13552
• Added a Golang SDK for Data Transforms. by @rockwotj in #12322
• Added the ability do disable whole topics or specific topic partitions. Disabled partitions are ignored by Redpanda and producing/consuming to them is impossible. This is useful when only a handful of partitions prevent the cluster from becoming healthy (e.g. if the data in them is corrupted in a way that causes Redpanda to crash). by @ztlpn in #15141
• Adds an admin API to reset the crash loop prevention counter. Additionally the tracking metadata is reset every time the broker boots up in recovery mode. by @bharathv in #15064
• Adds node wise partition recovery functionality. Given an input set of node ids, force reconfigures all partitions that would lose majority if the specified set of nodes are dead. Intended to bulk recover partitions that are stuck when majority of brokers hosting their replicas are dead and irrecoverable. by @bharathv in #13943 and
  #15394
• Admin API: Support OpenID connect Authentication @BenPope in #14378
• Allow template prefix in advertised addresses by @RafalKorepta in #14177
• K8s: Allow to move pods into new nodes if needed, and allow down scaling Redpanda custom resource. by @alejandroEsc in #12847
• Force reconfiguration now supports recovering from an ‘all replicas lost’ scenario. The replicas are reset with empty logs (no data) on the (broker, shard) destinations passed in the reconfiguration command. by @bharathv in #13661
• HTTP Proxy: Support OpenID connect authentication by @BenPope in #14378
• Adds fast partition movement to speed up cluster scaling operations by @mmaslankaprv in #14305
• Implements audit logging capabilities for the admin server. To configure set audit_enabled to True and ensure one of the auditing enabled type in audit_enabled_event_types is set to management by @graphcareful in #14158
• Metadata backup: In addition to topic data, Redpanda will now upload cluster-related metadata, such as cluster configs, security settings, and consumer offsets into cloud storage. This metadata can be restored alongside topic data, allowing for the functional recovery of the entire cluster. by @andrwng in #15188
• Introduce Prometheus metrics for tracking TLS cert/CA expiration by @oleiman in #13477
• Kafka API: Adds SASL/OAUTHBEARER authentication to the Kafka API by @BenPope in #14378
• Redpanda is now compatible with Azure Storage Accounts that enable Hierarchical Namespaces. by @VladLazar in #12632
• K8s: Redpanda Node ID will be reported as label and annotation in Pod metadata by @RafalKorepta in #12524
• Schema Registry: Support OpenID connect by @BenPope in #14378
• You can decode now schema registry encoded messages (AVRO or Protobuf) using rpk topic consume --use-schema-registry by @r-vasquez in #14498
• #12912 Adds rpc_client_connections_per_shard cluster property that allows for the number of clients a broker opens to a given peer to be user configurable. by @ballard26 in #12906
• #12934 #13617 Time-based retention now uses broker-based timestamps for determining when to purge new data. This reduces the risk of retention not removing segments when a misbehaving client produces messages with incorrect timestamps (e.g. a timestamp in the future). by @andijcr in #12991
  • The configuration option storage_ignore_timestamps_in_future_secs is retained to deal with bad segments produced before v23.3
  • This changes the behavior for messages with a timestamp in the past. Before, retention would use this timestamp to delete data. now, the retention window starts when the message arrives in the broker.
• #12989 Schema Registry: Support GET /schemas/ids/{id}/subjects by @oleiman in #13020
• #13229 Schema Registry: Add DELETE /config/{subject} endpoint by @oleiman in #13557
• #14191 Audit Kafka API authentication and authorization events by @michael-redpanda in #14452
• #14394 rpk: adds rpk cluster partitions enable/disable: now users can disable or enable partitions of a topic using rpk. by @r-vasquez in #14909
• #14483 rpk debug bundle: now you can use the --partition flag to request additional debugging information from the admin API for the provided partitions. by @r-vasquez in #15136
• #9128 K8s: New Topic Custom Resource that can manage single topic by @RafalKorepta in #11208
• #9205 rpk: new 'rpk cluster partitions move' command to reassign replicas by @daisukebe in #13684
• #9205 rpk: new ‘rpk cluster partitions move-status’ to show ongoing partition movements by @daisukebe in #13258
• adds rpk redpanda mode recovery command to put redpanda into recovery mode. by @r-vasquez in #14431
• ability to change number of partitions in tx manager topic by @mmaslankaprv in #15121
• adds support to the Cluster API (deprecated) for a user-specified ServiceAccount by @joejulian in #12864
• adds support to the Console API (deprecated) for a user-specified ServiceAccount by @joejulian in #13120
• crash_loop_limit now defaults to 5. If a broker uncleanly shutdowns for 5 times back to back, it is considered to be in a crash loop mode and Redpanda refuses to start up and may need manual intervention. This enforcement is disabled in developer mode and rpk's dev-container mode. by @bharathv in #13431
• rpk introduces rpk cluster partitions list which lets users query cluster-level metadata of all partitions in the cluster. by @r-vasquez in #14862
• rpk now has a cluster txn command space by @twmb in #7557
• rpk now supports encoding messages with a given schema stored in the schema registry when producing via rpk topic produce. by @r-vasquez in #13543
• rpk: Introduce a new flag --label-selector to rpk debug bundle and have a way to filter the bundled resources by label. by @r-vasquez in #15185
• rpk: new rpk registry command to manage schema registry with rpk by @r-vasquez in #12669
• PR #14283 Compaction lossy hash by @dotnwat...

v23.2.21

Bug Fixes

• Fix a memory leak when transactions are used with many different producer IDs. by @rockwotj in #15796

• #15688 topic remote recovery via redpanda.remote.recovery handle correctly disabled retention by @andijcr in #15689

• #15786 Fix an issue where new configs would continually revert to legacy defaults after an upgrade. by @oleiman in #15787

• rpk: Redpanda log collection now is possible in pods where you have multiple containers, by default, rpk will gather logs from the 'redpanda' container. by @r-vasquez in #15684

• PR #15597 [v23.2.x] r/state_machine: ignore broken semaphore and condition var exceptions by @mmaslankaprv

• PR #15599 [v23.2.x] tests/e2e_shadow_indexing_test: wait until segments has one entry by @andijcr

• PR #15600 [v23.2.x] archival: Disable cross term merging of adjacent segments by @Lazin

• PR #15602 [v23.2.x] storage/disk_log_impl: defensive reverse iteration of _segs by @andijcr

• PR #15772 [v23.2.x] c/members_member: don't ignore update_broker_client future by @rockwotj

• PR #15780 [v23.2.x] Fix 128K iobuf zero-copy by @travisdowns

• PR #15800 [v23.2.x] c/topic_table: do not log duplicated lifecycle marker command by @mmaslankaprv

Full Changelog: v23.2.20...v23.2.21