Replies: 1 comment 10 replies
-
|
Hi, I think you mean to continue the discussion started in #13230 which was closed early. But please copy over relevant information to this discussion as well. I think it is a relevant information that your disk space is taken up almost entirely by the If you could provide the listing of the rdq files ( I am commenting because we've also seen cases on 4.0.x when the msg_store process does not delete rdq files which are not used any more (no queue references messages in them). The msg_store process should delete such file after the last message was dereferenced from it or after an unclean shutdown during a recovery. Obviously we couldn't reproduce it on a test env. |
Beta Was this translation helpful? Give feedback.
-
|
great, this one fully matches. Oldest two files are "full" (valid size = file size) Following 1586 files are "empty" (valid size = 0) then some new entries but they don't match because of the different time the two listings were created, but this is fine. There are gaps between the empty files' indices so some rdq files are cleaned up correctly, but there are empty files as recent as today, so some rdq files are continually left behind. The file sizes vary a lot but I'm not sure if it means that compaction already happened on them. That's all I can deduce right now (which is already a bit more than what I knew about the issue :) ) Hopefully a message store expert from Core Team can suggest how to investigate this issue further. |
Beta Was this translation helpful? Give feedback.
-
lists queues in a single virtual host. As explained in #13230, under If it's a virtual host that you don't care about, simply delete it and move on. Without a specific set of steps to reproduce we cannot suggest anything else. It is not a commonly The only thing that RabbitMQ itself could do that comes to mind is the migration from CQv1 (if you were on it) to CQv2 (the only option in 4.0.x) but we have close to a year of extensive experience with these, and they effectively always work as expected. |
Beta Was this translation helpful? Give feedback.
-
|
@icolombi how can we? We do not know what exactly happened to your cluster, what applications might have been doing during the upgrade, and so on. The If you are sure that those queues are gone or not needed, you can probably simply delete the files. |
Beta Was this translation helpful? Give feedback.
-
|
@icolombi as Michael suggested one workaround to avoid running out of disk is to delete older rdq files (or even better to truncate them to zero size) This way the rabbitmq node will stay in this state where the issue is reproducing @michaelklishin please don't close this discussion yet. It tracks an issue that both icolombi and we faced. Namely that on 4.0.5 the shared message store (not to be mixed with the queue index - CQv1 and CQv2 is only applicable to the queue index) from time to time does not delete rdq files that are not used any more (ie no queue index references the messages stored in it). On the long run this can lead to resource alarm or full disk. It is entirely not certain that this is related to the version upgrade at all (or what happened during the upgrade) and might happen on a new 4.0.5 version as well. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @gomoripeti , thanks. At the moment I'm deleting I've also tried to delete all |
Beta Was this translation helpful? Give feedback.
-
Community Support Policy
RabbitMQ version used
4.0.5
Erlang version used
26.2.x
Operating system (distribution) used
Rocky Linux release 9.5
How is RabbitMQ deployed?
RPM package
rabbitmq-diagnostics status output
See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics
Logs from node 1 (with sensitive values edited out)
See https://www.rabbitmq.com/docs/logging to learn how to collect logs
Logs from node 2 (if applicable, with sensitive values edited out)
See https://www.rabbitmq.com/docs/logging to learn how to collect logs
Logs from node 3 (if applicable, with sensitive values edited out)
See https://www.rabbitmq.com/docs/logging to learn how to collect logs
rabbitmq.conf
See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location
Steps to deploy RabbitMQ cluster
Download latest version from:
https://github.com/rabbitmq/rabbitmq-server/releases
RPM for Rocky / CentOS
Steps to reproduce the behavior in question
Upgrade with RPM from 3.13.x to 4.0.5
advanced.config
See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location
Application code
# PASTE CODE HERE, BETWEEN BACKTICKSKubernetes deployment file
What problem are you trying to solve?
Disk space consumed abnormally.
Beta Was this translation helpful? Give feedback.
All reactions