Skip to content

DOC-1378: Compaction robustness #1153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion antora.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
name: ROOT
title: Self-Managed
version: 25.1
version: 25.2
display_version: '25.2 Beta'
prerelease: true
start_page: home:index.adoc
nav:
- modules/ROOT/nav.adoc
Expand Down
4 changes: 2 additions & 2 deletions local-antora-playbook.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
site:
title: Redpanda Docs
start_page: 25.1@ROOT:get-started:intro-to-events.adoc
start_page: 25.2@ROOT:get-started:intro-to-events.adoc
url: http://localhost:5002
robots: disallow
keys:
Expand All @@ -15,7 +15,7 @@ content:
- url: .
branches: HEAD
- url: https://github.com/redpanda-data/docs
branches: [v/*, api, shared, site-search,'!v-end-of-life/*']
branches: [main, v/*, api, shared, site-search,'!v-end-of-life/*']
- url: https://github.com/redpanda-data/cloud-docs
branches: 'main'
- url: https://github.com/redpanda-data/redpanda-labs
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,14 @@ Where:
| xref:reference:cluster-properties.adoc#log_compaction_interval_ms[`log_compaction_interval_ms`]
| Cluster
| Compaction frequency in milliseconds.

| xref:reference:properties/topic-properties.adoc#max.compaction.lag.ms[`max.compaction.lag.ms`]
| Topic
| The maximum amount of time in milliseconds that a message remains ineligible for compaction.

| xref:reference:properties/topic-properties.adoc#min.compaction.lag.ms[`min.compaction.lag.ms`]
| Topic
| The minimum time in milliseconds that a message remains uncompacted in the log.
|===

Redpanda runs a scan every `log_compaction_interval_ms`. During each scan:
Expand Down
37 changes: 37 additions & 0 deletions modules/reference/pages/properties/cluster-properties.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3674,6 +3674,25 @@ Maximum compacted segment size after consolidation.

---

=== max_compaction_lag_ms

For a compacted topic, the maximum time a message remains ineligible for compaction.
The topic property (xref:reference:properties/topic-properties.adoc#max.compaction.lag.ms)[max.compaction.lag.ms] overrides this property.

*Unit:* milliseconds

*Requires restart:* No

*Visibility:* `user`

*Type:* integer

*Accepted values:* [`1, 9223372036854`]

*Default:* `9223372036854`

---

=== max_concurrent_producer_ids

Maximum number of active producer sessions. When the threshold is passed, Redpanda terminates old sessions. When an idle producer corresponding to the terminated session wakes up and produces, its message batches are rejected, and an out of order sequence error is emitted. Consumers don't affect this setting.
Expand Down Expand Up @@ -3919,6 +3938,24 @@ The minimum ratio between the number of bytes in dirty segments and the total nu

---

=== min_compaction_lag_ms

For a compacted topic, the minimum time a message remains uncompacted in the log. The topic property (xref:reference:properties/topic-properties.adoc#min.compaction.lag.ms)[min.compaction.lag.ms] overrides this property.

*Unit:* milliseconds

*Requires restart:* No

*Visibility:* `user`

*Type:* integer

*Accepted values:* [`0,9223372036854`]

*Default:* `9223372036854`

---

// tag::minimum_topic_replications[]
=== minimum_topic_replications

Expand Down
6 changes: 6 additions & 0 deletions modules/reference/pages/properties/topic-properties.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,12 @@ NOTE: All topic properties take effect immediately after being set.
| <<compressiontype,`compression.type`>>
| xref:./cluster-properties.adoc#log_compression_type[`log_compression_type`]

| <<max.compaction.lag.ms,`max.compaction.lag.ms`>>
| xref:./cluster-properties.adoc#max_compaction_lag_ms[`max_compaction_lag_ms`]

| <<min.compaction.lag.ms,`min.compaction.lag.ms`>>
| xref:./cluster-properties.adoc#min_compaction_lag_ms[`min_compaction_lag_ms`]

| <<messagetimestamptype,`message.timestamp.type`>>
| xref:./cluster-properties.adoc#log_message_timestamp_type[`log_message_timestamp_type`]

Expand Down