4.0.5: questions about disk and memory footprint #13225
-
Environment Details
Issue ObservedOne of the RabbitMQ pods in the cluster is consuming unexpectedly high memory and storage, while other pods in the same cluster and across different environments do not exhibit this issue. Potential Storage Leak
Potential Memory Leak
Additional Observations
Reproduction stepsExpected Behavior
|
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
|
@ronmegini I don't see any evidence of a leak. This should have been a discussion, not an issue. Messages are stored on disk with a fair amount of metadata. When messages are consumed and confirmed, they are not deleted immediately, neither by CQs nor by QQs and certainly not streams (where consumptions is non-destructive). Quorum queues delete entire segment files, not individual messages and CQs compact segment files when more than half of two files have "holes" (messages marked for deletion). The explanation for one pod having an outsized footprint is almost always as simple as: that pod hosts the majority of replicas, or, in your case, the majority of CQs which in 4.x are a non-replicated type. The runtime allocates memory without using it immediately, which is explained in the Reasoning About Memory Use guide. There is kernel page cache which Kubernetes before a certain version (1.25 or so) or certain cgroups version (v1) is considered to be a part of the process' footprint even thought that's completely incorrect, which our docs also mention. |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin Thanks for the explanation! Memory allocation - 27GiB cached: Storage allocation - 77GiB of messages stored in disk, while the total queues size stored on this pod is 25GiB: 
|
Beta Was this translation helpful? Give feedback.
-
|
So 92% of the memory footprint comes down to "reserved by the runtime", "allocated but unused" or belonging to the binary heap that is reference-counted and in practice, much lazier to be cleaned up. This is neither a leak nor a very common scenario to see. See It might help with the binary heap part since I cannot guess what specifically is referencing those messages. |
Beta Was this translation helpful? Give feedback.
-
|
You have accumulated a backlog of messages in CQs and kept it there without consumers. If this is your typical workload, perhaps use a stream or a quorum queue, in the latter case tuning the segment entry count according to the message size specifically to avoid gigantic segment files that might only have one alive messages in them each. |
Beta Was this translation helpful? Give feedback.
@ronmegini I don't see any evidence of a leak. This should have been a discussion, not an issue.
Messages are stored on disk with a fair amount of metadata. When messages are consumed and confirmed, they are not deleted immediately, neither by CQs nor by QQs and certainly not streams (where consumptions is non-destructive).
Quorum queues delete entire segment files, not individual messages and CQs compact segment files when more than half of two files have "holes" (messages marked for deletion).
The explanation for one pod having an outsized footprint is almost always as simple as: that pod hosts the majority of replicas, or, in your case, the majority of CQs which in 4.x are a non-replic…