v1.26.0

[romange]

Dragonfly v1.26.0 - Chrismukkah edition 🔯 🎄🔯 🎄🔯

This marks our final release for the year, featuring numerous stability fixes and performance enhancements.

Important features and changes:

1. Numerous Search and Json fixes fix(search_family): Process wrong field types in indexes for the FT.SEARCH and FT.AGGREGATE commands #4070 fix(search_family): Fix LOAD fields parsing in the FT.AGGREGATE and FT.SEARCH commands #4012 fix(search_family): Fix crash when no SEPARATOR is specified in the FT.CREATE command #4205 fix(rax_tree): Fix crash caused by destructor in RaxTreeMap #4228 fix(search_family): Remove the output of extra fields in the FT.AGGREGATE command #4231 fix(search_family): Fix FT.AGGREGATE output #4311 fix(search_family): Fix logging in ParseFieldWithAtSign #4343
2. Account for list type memory, fixes List types memory tracking is inaccurate #3800
3. Native support of GCS storage! Just point your --dir to gs::\mybucket\df_snapshots\
4. Fix metrics compatibility with prometheus 3.0 scraping Prometheus metrics broken on prom 3.0 #4277
5. Series of STREAM improvements bringing Dragonfly to be compatible with redis/valkey 7.2 streams: feat: add ability reading stream_listpacks_2/3 rdb types #4192, chore: fix bugs in stream_family #4237, fix: bugs in stream code #4239 fix: stream bugs #4240 fix: support loading of 7.x streams correctly #4281 chore: add active time to stream consumers #4285

What's Changed

• chore: remove old io by @dranikpg in chore: remove old io #3953
• chore: RdbSerializer::SaveListObject supports QList by @romange in chore: RdbSerializer::SaveListObject supports QList #4101
• chore: Implement list Pop/Erase functionality with QList by @romange in chore: Implement list Pop/Erase functionality with QList #4099
• chore: change Namespaces to be a global pointer by @romange in chore: change Namespaces to be a global pointer #4032
• fix: update instruction how to build from the source for ubuntu by @BorysTheDev in fix: update instruction how to build from the source for ubuntu #4103
• fix: test_noreply_pipeline flakiness by @romange in fix: test_noreply_pipeline flakiness #4102
• chore: Implement LMOVE over QList by @romange in chore: Implement LMOVE over QList #4104
• fix(search_family): Process wrong field types in indexes for the FT.SEARCH and FT.AGGREGATE commands by @BagritsevichStepan in fix(search_family): Process wrong field types in indexes for the FT.SEARCH and FT.AGGREGATE commands #4070
• chore: implement Erase for a range by @romange in chore: implement Erase for a range #4106
• chore: decouple reply_builder from ConnectionContext by @romange in chore: decouple reply_builder from ConnectionContext #4069
• chore: support rdb loading and container utils with QList. by @romange in chore: support rdb loading and container utils with QList. #4109
• feat(server) : snapshot traverse physical buckets by @adiholden in feat(server) : snapshot traverse physical buckets #4084
• fix: skip Send() in SinkReplyBuilder::Flush() when vec is empty by @kostasrim in fix: skip Send() in SinkReplyBuilder::Flush() when vec is empty #4114
• fix: regression in squashing code when determining eval commands by @romange in fix: regression in squashing code when determining eval commands #4116
• fix: slow regression tests tests by @kostasrim in fix: slow regression tests tests #4117
• test: add test for snapshoting during migration by @BorysTheDev in test: add test for snapshoting during migration #4108
• feat server: experimental_new_io flag add as deprecated by @adiholden in feat server: experimental_new_io flag add as deprecated #4127
• chore: simplify BumpUps deduplication by @romange in chore: simplify BumpUps deduplication #4098
• fix: Do not use cc_ in connection if it's null by @chakaz in fix: Do not use cc_ in connection if it's null #4131
• fix: Use MOVED error type for moved replies by @chakaz in fix: Use MOVED error type for moved replies #4125
• chore(deps): bump github/codeql-action from 3.27.0 to 3.27.1 in the actions group by @dependabot in chore(deps): bump github/codeql-action from 3.27.0 to 3.27.1 in the actions group #4115
• fix server: fix stats of pipeline squashed commands by @adiholden in fix server: fix stats of pipeline squashed commands #4132
• chore: fix plain node insertion by @romange in chore: fix plain node insertion #4134
• chore: improve the state machine of RedisParser by @romange in chore: improve the state machine of RedisParser #4085
• chore: run tests with list_experimental_v2 enabled by @romange in chore: run tests with list_experimental_v2 enabled #4112
• chore: optimize info command by @romange in chore: optimize info command #4137
• fix server: fix write to slowlog on squashing flow by @adiholden in fix server: fix write to slowlog on squashing flow #4138
• chore:update fakeredis, remove irrelevant tests by @cunla in chore:update fakeredis, remove irrelevant tests #4014
• chore: fix a potential crash during client list by @romange in chore: fix a potential crash during client list #4141
• fix: migration ACK response processing by @BorysTheDev in fix: migration ACK response processing #4140
• feat: add INFO memory section for squashing replies memory consuming by @BorysTheDev in feat: add INFO memory section for squashing replies memory consuming #4147
• fix: allow SELECT in multi/exec if it's a noop by @romange in fix: allow SELECT in multi/exec if it's a noop #4146
• chore(deps): bump github/codeql-action from 3.27.1 to 3.27.4 in the actions group by @dependabot in chore(deps): bump github/codeql-action from 3.27.1 to 3.27.4 in the actions group #4148
• chore: tune logs and improve restrict denied error by @romange in chore: tune logs and improve restrict denied error #4145
• fix: UB during cmd squashing reply size calculation by @BorysTheDev in fix: UB during cmd squashing reply size calculation #4149
• fix: enforce load limits when loading snapshot by @romange in fix: enforce load limits when loading snapshot #4136
• fix: test_replication_all failure by @romange in fix: test_replication_all failure #4155
• fix: Huge entries fail to load outside RDB / replication by @chakaz in fix: Huge entries fail to load outside RDB / replication #4154
• chore: allow slow and precise memory measurement of an object by @romange in chore: allow slow and precise memory measurement of an object #4160
• chore: fix a regression build break by @romange in chore: fix a regression build break #4162
• chore: log exit code of failing dragonfly in tests by @romange in chore: log exit code of failing dragonfly in tests #4166
• fix: remove DenseSet::IteratorBase::TraverseApply by @kostasrim in fix: remove DenseSet::IteratorBase::TraverseApply #4170
• adjusted grafana dashboard to be more user friendly by @applike-ss in adjusted grafana dashboard to be more user friendly #4165
• chore: fix shutdown sequence in Dragonfly server by @romange in chore: fix shutdown sequence in Dragonfly server #4168
• feat(server): add eval_ro and evalsha_ro by @andydunstall in feat(server): add eval_ro and evalsha_ro #4091
• chore: Hide replicas from CLUSTER subcmds in managed mode by @chakaz in chore: Hide replicas from CLUSTER subcmds in managed mode #4174
• fix(tools): Prevent dragonfly.logrotate from stopping logrotate service by @s0xDk in fix(tools): Prevent dragonfly.logrotate from stopping logrotate service #4176
• chore: more parser improvements by @romange in chore: more parser improvements #4177
• fix(release/helm): allow empty commits for rerun by @Pothulapati in fix(release/helm): allow empty commits for rerun #4163
• fix(search_family): Fix LOAD fields parsing in the FT.AGGREGATE and FT.SEARCH commands by @BagritsevichStepan in fix(search_family): Fix LOAD fields parsing in the FT.AGGREGATE and FT.SEARCH commands #4012
• feat: Huge values breakdown in cluster migration by @chakaz in feat: Huge values breakdown in cluster migration #4144
• feat: cluster migrations restarts if timeout happens by @BorysTheDev in feat: cluster migrations restarts if timeout happens #4081
• fix: deduplicate mget response by @romange in fix: deduplicate mget response #4175
• fix: stop using openssl for container healthchecks by @romange in fix: stop using openssl for container healthchecks #4181
• chore: produce core files in regtests by @romange in chore: produce core files in regtests #4185
• chore: stop periodic task earlier during the shutdown process by @romange in chore: stop periodic task earlier during the shutdown process #4187
• chore(deps): bump github/codeql-action from 3.27.4 to 3.27.5 in the actions group by @dependabot in chore(deps): bump github/codeql-action from 3.27.4 to 3.27.5 in the actions group #4186
• test: skip test_cluster_mgr because of unclosed instance by @BorysTheDev in test: skip test_cluster_mgr because of unclosed instance #4191
• chore: fix false positives sanitizers by @kostasrim in chore: fix false positives sanitizers #4190
• fix: debug object encoding names by @romange in fix: debug object encoding names #4188
• feat: add ability reading stream_listpacks_2/3 rdb types by @BorysTheDev in feat: add ability reading stream_listpacks_2/3 rdb types #4192
• chore: get back on the decision to put a hard limit on command interface by @romange in chore: get back on the decision to put a hard limit on command interface #4203
• chore: qlist improvements by @romange in chore: qlist improvements #4194
• fix: stream memory tracking by @kostasrim in fix: stream memory tracking #4067
• chore: fully support qlist in all list_family commands by @romange in chore: fully support qlist in all list_family commands #4201
• feat(contrib/helm): evaluate the provided passwordSecretName value as a template by @oOraph in feat(contrib/helm): evaluate the provided passwordSecretName value as a template #4046
• fix(search_family): Fix crash when no SEPARATOR is specified in the FT.CREATE command by @BagritsevichStepan in fix(search_family): Fix crash when no SEPARATOR is specified in the FT.CREATE command #4205
• chore: update command interface for hset/set families by @romange in chore: update command interface for hset/set families #4209
• fix: cluster_mgr script by @BorysTheDev in fix: cluster_mgr script #4210
• bug(replication): snapshot cleanup fix in transition to stable sync by @adiholden in bug(replication): snapshot cleanup fix in transition to stable sync #4211
• test: fix and unskip test_migration_timeout_on_sync by @BorysTheDev in test: fix and unskip test_migration_timeout_on_sync #4216
• chore: Add more qlist tests by @romange in chore: Add more qlist tests #4217
• chore: change the interface of stream and server commands by @romange in chore: change the interface of stream and server commands #4219
• chore: add noop docs subcmd by @kostasrim in chore: add noop docs subcmd #4214
• chore: remove tail field from qlist by @romange in chore: remove tail field from qlist #4220
• chore: qlist code clean ups by @romange in chore: qlist code clean ups #4223
• chore: Track QList memory by @romange in chore: Track QList memory #4226
• chore: change the interface of hll, generic and list commands by @romange in chore: change the interface of hll, generic and list commands #4227
• fix: command docs by @kostasrim in fix: command docs #4229
• chore: support qlist compression when accounting for memory by @romange in chore: support qlist compression when accounting for memory #4233
• chore: update command interface for bitops, bloom, zset families by @romange in chore: update command interface for bitops, bloom, zset families #4235
• Feat(Helm chart): Add env vars into pods by @andreibe24 in Feat(Helm chart): Add env vars into pods #4196
• chore: fix bugs in stream_family by @romange in chore: fix bugs in stream_family #4237
• fix: bugs in stream code by @romange in fix: bugs in stream code #4239
• fix: Fix test_network_disconnect_during_migration test by @chakaz in fix: Fix test_network_disconnect_during_migration test #4224
• fix: RdbTest.LoadStream3 incorrect file usage by @BorysTheDev in fix: RdbTest.LoadStream3 incorrect file usage #4242
• chore: Hide replica info in real cluster if --managed_service_info by @chakaz in chore: Hide replica info in real cluster if --managed_service_info #4241
• chore: Hide managed service info in INFO by @chakaz in chore: Hide managed service info in INFO #4248
• fix: stream bugs by @romange in fix: stream bugs #4240
• fix: command interface for json_family by @romange in fix: command interface for json_family #4243
• fix(server): fix memory leak on lua error by @adiholden in fix(server): fix memory leak on lua error #4236
• chore: update command interface for acl, hll families by @romange in chore: update command interface for acl, hll families #4253
• chore: split RecordExpiry preemptive and non-preemptive flows by @kostasrim in chore: split RecordExpiry preemptive and non-preemptive flows #4252
• fix(rax_tree): Fix crash caused by destructor in RaxTreeMap by @BagritsevichStepan in fix(rax_tree): Fix crash caused by destructor in RaxTreeMap #4228
• chore: update command interface for cluster and search families by @romange in chore: update command interface for cluster and search families #4258
• refactor: remove redundant allocations for streamer by @BorysTheDev in refactor: remove redundant allocations for streamer #4225
• fix(save): don't create dirs for cloud dir by @andydunstall in fix(save): don't create dirs for cloud dir #4259
• feat: add support for big values in SeederV2 by @kostasrim in feat: add support for big values in SeederV2 #4222
• chore: remove DbSlice mutex and add ConditionFlag in SliceSnapshot by @kostasrim in chore: remove DbSlice mutex and add ConditionFlag in SliceSnapshot #4073
• fix(eviction): Tune eviction threshold in cache mode by @BagritsevichStepan in fix(eviction): Tune eviction threshold in cache mode #4142
• fix: test_network_disconnect_during_migration data size was too big f… by @BorysTheDev in fix: test_network_disconnect_during_migration data size was too big f… #4260
• fix(geo): support case insensitive units by @andydunstall in fix(geo): support case insensitive units #4264
• chore: monotonically increasing ports for cluster tests by @kostasrim in chore: monotonically increasing ports for cluster tests #4268
• fix: close socket to prevent onCompletion call after the journal stre… by @BorysTheDev in fix: close socket to prevent onCompletion call after the journal stre… #4270
• chore: Fix test_rss_used_mem_gap for all types by @chakaz in chore: Fix test_rss_used_mem_gap for all types #4254
• feat server: support config set serialization_max_chunk_size by @adiholden in feat server: support config set serialization_max_chunk_size #4274
• fix: prohibit read commands during takeover by @BorysTheDev in fix: prohibit read commands during takeover #4267
• fix cluster: migration traverse bug by @adiholden in fix cluster: migration traverse bug #4279
• fix(server) : dont apply eviction on rss over limit by @adiholden in fix(server) : dont apply eviction on rss over limit #4276
• chore: let resp parser provide more useful logs by @romange in chore: let resp parser provide more useful logs #4273
• chore: update command interface for main_service commands by @romange in chore: update command interface for main_service commands #4265
• fix: support loading of 7.x streams correctly by @romange in fix: support loading of 7.x streams correctly #4281
• chore: fixes the parse error for xread/xreadgroup with unbalanced ids by @romange in chore: fixes the parse error for xread/xreadgroup with unbalanced ids #4266
• fix: Do not attempt to defrag StringSet as a StringMap by @chakaz in fix: Do not attempt to defrag StringSet as a StringMap #4283
• chore: add active time to stream consumers by @romange in chore: add active time to stream consumers #4285
• fix: mismatch new-delete in unit test by @chakaz in fix: mismatch new-delete in unit test #4288
• fix(search_family): Remove the output of extra fields in the FT.AGGREGATE command by @BagritsevichStepan in fix(search_family): Remove the output of extra fields in the FT.AGGREGATE command #4231
• fix: increase cluster migration default timeout by @BorysTheDev in fix: increase cluster migration default timeout #4293
• chore: test metrics for huge value serialization by @kostasrim in chore: test metrics for huge value serialization #4262
• chore: update reflex version and fix its build on alpine by @romange in chore: update reflex version and fix its build on alpine #4304
• fix: circular dependency in qlist by @romange in fix: circular dependency in qlist #4302
• feat: add migration_finalization_timeout_ms flag by @BorysTheDev in feat: add migration_finalization_timeout_ms flag #4301
• fix: Protect BumpUp() from running in parallel to serialization by @chakaz in fix: Protect BumpUp() from running in parallel to serialization #4307
• fix(search_family): Fix FT.AGGREGATE output by @BagritsevichStepan in fix(search_family): Fix FT.AGGREGATE output #4311
• chore: improve parser state machine by @romange in chore: improve parser state machine #4313
• test: update logs and test for debug purpose by @BorysTheDev in test: update logs and test for debug purpose #4309
• server: report redis version 7.2.0 to support Sidekiq by @adiholden in server: report redis version 7.2.0 to support Sidekiq #4286
• chore: factor out rdb_load utilities into separate files by @romange in chore: factor out rdb_load utilities into separate files #4315
• fix: Remove hardcoded @assert_eventually 100 times retry by @chakaz in fix: Remove hardcoded @assert_eventually 100 times retry #4318
• refactor: remove serialization_max_chunk_size for cluster tests by @BorysTheDev in refactor: remove serialization_max_chunk_size for cluster tests #4316
• chore: factor out CompressorImpl into separate files by @romange in chore: factor out CompressorImpl into separate files #4319
• fix: Do not preempt on dispatcher fiber by @chakaz in fix: Do not preempt on dispatcher fiber #4323
• fix(test): seeder test remove check by @adiholden in fix(test): seeder test remove check #4320
• fix: potential OOM when first request sent in small bits by @romange in fix: potential OOM when first request sent in small bits #4325
• fix: crash during getting info about replication by @BorysTheDev in fix: crash during getting info about replication #4328
• chore: add ability to track connections stuck at send by @romange in chore: add ability to track connections stuck at send #4330
• fix: do not check-fail in OpRestore by @romange in fix: do not check-fail in OpRestore #4332
• test: skip test_network_disconnect_during_migration because it is uns… by @BorysTheDev in test: skip test_network_disconnect_during_migration because it is uns… #4334
• chore: fix wording around the dispatch fiber in dragonfly_connection by @romange in chore: fix wording around the dispatch fiber in dragonfly_connection #4333
• fix (regression tests): skip flaky test by @adiholden in fix (regression tests): skip flaky test #4336
• chore: Disable failing test by @chakaz in chore: Disable failing test #4337
• fix: cluster tests stability by @BorysTheDev in fix: cluster tests stability #4338
• fix(pytest): call stop for all instances even if stop raise exception by @adiholden in fix(pytest): call stop for all instances even if stop raise exception #4339
• fix: add content-type for metrics response by @romange in fix: add content-type for metrics response #4340
• fix(server): use compression for non big values by @adiholden in fix(server): use compression for non big values #4331
• fix: Stack overflow in DFLYCLUSTER CONFIG by @chakaz in fix: Stack overflow in DFLYCLUSTER CONFIG #4342
• fix(search_family): Fix logging in ParseFieldWithAtSign by @BagritsevichStepan in fix(search_family): Fix logging in ParseFieldWithAtSign #4343

Huge thanks to all the contributors! ❤️

New Contributors

• @s0xDk made their first contribution in fix(tools): Prevent dragonfly.logrotate from stopping logrotate service #4176
• @oOraph made their first contribution in feat(contrib/helm): evaluate the provided passwordSecretName value as a template #4046
• @andreibe24 made their first contribution in Feat(Helm chart): Add env vars into pods #4196

Full Changelog: v1.25.0...v1.26.0

---

This discussion was created from the release v1.26.0.