Wazuh indexer warning messages during uninstallation

[thony4uu]

Component	Installation	Type	OS
Indexer	Step by step	Multi node	Ubuntu 22.04 x86_64

Hello Team,

I observed that the uninstallation of the Wazuh Indexer produces warning messages, as shown below, during the testing for Release 4.9.1 - RC 1 - E2E UX tests - Wazuh Indexer #25836. The logs is shown below:

root@ip-172-31-32-65:~# apt-get remove --purge wazuh-indexer -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  wazuh-indexer*
0 upgraded, 0 newly installed, 1 to remove and 70 not upgraded.
After this operation, 1077 MB disk space will be freed.
(Reading database ... 192551 files and directories currently installed.)
Removing wazuh-indexer (4.9.1-1) ...
Running Wazuh Indexer Pre-Removal Script
Stop existing wazuh-indexer.service
(Reading database ... 191404 files and directories currently installed.)
Purging configuration files for wazuh-indexer (4.9.1-1) ...
dpkg: warning: while removing wazuh-indexer, directory '/var/log/wazuh-indexer' not empty so not removed
dpkg: warning: while removing wazuh-indexer, directory '/var/lib/wazuh-indexer' not empty so not removed
dpkg: warning: while removing wazuh-indexer, directory '/etc/wazuh-indexer' not empty so not removed
root@ip-172-31-32-65:~# rm -rf /var/lib/wazuh-indexer/
root@ip-172-31-32-65:~# rm -rf /usr/share/wazuh-indexer/
root@ip-172-31-32-65:~# rm -rf /etc/wazuh-indexer/
root@ip-172-31-32-65:~# systemctl daemon-reload
root@ip-172-31-32-65:~# systemctl status wazuh-indexer
Unit wazuh-indexer.service could not be found.

[rauldpm]

About this issue and why I asked @thony4uu to open it

When we remove a package from a system (YUM or APT) we use a remove command (yum remove / apt remove), both commands remove files installed with the package but leave configuration files or similar

In APT systems, there is another option called purge (the one we use in our documentation: apt remove --purge), this option is used to remove a package completely, including all the package files, without a trace, in a normal package, this will cover all files installed in the system and tracked by the system DDBB, but our package install files outside that DDBB, so when the purge option is used some files are still present in the system (except in the user's home that it is expected to be ignored based on the man description), for example in the following directories:

• /var/lib/wazuh-indexer
• /var/log/wazuh-indexer
• /etc/wazuh-indexer

After a quick test (service fail is expected, is to force the service start) we can check the reported behavior:

root@debian12agent:/home/vagrant# systemctl start wazuh-indexer
Job for wazuh-indexer.service failed because the control process exited with error code.
See "systemctl status wazuh-indexer.service" and "journalctl -xeu wazuh-indexer.service" for details.
root@debian12agent:/home/vagrant# apt remove --purge wazuh-indexer
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  wazuh-indexer*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 1077 MB disk space will be freed.
Do you want to continue? [Y/n] Y
(Reading database ... 42724 files and directories currently installed.)
Removing wazuh-indexer (4.9.1-1) ...
Running Wazuh Indexer Pre-Removal Script
(Reading database ... 41576 files and directories currently installed.)
Purging configuration files for wazuh-indexer (4.9.1-1) ...
dpkg: warning: while removing wazuh-indexer, directory '/var/log/wazuh-indexer' not empty so not removed
dpkg: warning: while removing wazuh-indexer, directory '/etc/wazuh-indexer' not empty so not removed
root@debian12agent:/home/vagrant# ls -l /var/log/wazuh-indexer/
total 60
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 15270 Sep 25 18:56 gc.log
-rw-r--r-- 1 wazuh-indexer wazuh-indexer  2113 Sep 25 18:56 gc.log.00
drwxr-xr-x 2 wazuh-indexer wazuh-indexer  4096 Sep 25 18:56 tmp
-rw-r----- 1 wazuh-indexer wazuh-indexer 15621 Sep 25 18:56 wazuh-cluster.log
-rw-r----- 1 wazuh-indexer wazuh-indexer     0 Sep 25 18:56 wazuh-cluster_deprecation.json
-rw-r----- 1 wazuh-indexer wazuh-indexer     0 Sep 25 18:56 wazuh-cluster_deprecation.log
-rw-r----- 1 wazuh-indexer wazuh-indexer     0 Sep 25 18:56 wazuh-cluster_index_indexing_slowlog.json
-rw-r----- 1 wazuh-indexer wazuh-indexer     0 Sep 25 18:56 wazuh-cluster_index_indexing_slowlog.log
-rw-r----- 1 wazuh-indexer wazuh-indexer     0 Sep 25 18:56 wazuh-cluster_index_search_slowlog.json
-rw-r----- 1 wazuh-indexer wazuh-indexer     0 Sep 25 18:56 wazuh-cluster_index_search_slowlog.log
-rw-r----- 1 wazuh-indexer wazuh-indexer 16942 Sep 25 18:56 wazuh-cluster_server.json
-rw-r----- 1 wazuh-indexer wazuh-indexer     0 Sep 25 18:56 wazuh-cluster_task_detailslog.json
-rw-r----- 1 wazuh-indexer wazuh-indexer     0 Sep 25 18:56 wazuh-cluster_task_detailslog.log
root@debian12agent:/home/vagrant# ls -l /etc/wazuh-indexer/
total 4
-rw-rw---- 1 wazuh-indexer wazuh-indexer 196 Sep 25 18:56 opensearch.keystore

We can see that the opensearch.keystore is still present

Usually, we use the remove option because we do not want to remove configuration or backup files, but we use the purge option when we do not want anything installed in the system, this is what a Linux user will expect, like the man page states (again, excluding the user's home files).

• https://manpages.ubuntu.com/manpages/jammy/man8/apt.8.html

           Removing a package removes all packaged data, but leaves usually small (modified) user
           configuration files behind, in case the remove was an accident. Just issuing an
           installation request for the accidentally removed package will restore its function as
           before in that case. On the other hand you can get rid of these leftovers by calling
           purge even on already removed packages. Note that this does not affect any data or
           configuration stored in your home directory.

A Reddit post about this: https://www.reddit.com/r/Ubuntu/comments/lecrlo/aptpurge_is_not_removing_configuration_files_why/

So, in a normal package, those files would have been removed when using purge.

As we are creating files outside the package DDBB when it is installed due to the package and deployment needs, we are responsible for those files too, there is a difference between how the purge is managed in 4.8.x and in 4.9.x, and that is why we were removing those files "manually"

• 4.8.2: https://github.com/wazuh/wazuh-packages/blob/v4.8.2/stack/indexer/deb/debian/postrm

    # Delete the config directory if we are purging
    if [ "$REMOVE_CONFIG_DIRECTORY" = "true" ]; then
        echo -n "Deleting configuration directory..."
        rm -rf "${CONFIG_DIR}" > /dev/null 2>&1
        echo " OK"
    fi

• 4.9.x: https://github.com/wazuh/wazuh-indexer/blob/4.9.1/distribution/packages/src/common/scripts/postrm#L67
  • The configuration directory is not present or marked to be removed manually

---

There are issues pre-4.9 that report this behavior, like wazuh/wazuh-packages#1643 (comment) that didn't fix the reported behavior about removing files because they were going to be fixed in the fork, but not only we are still having them but we have more files.

We need to review our Wazuh indexer (and probably the Wazuh dashboard) removal policies with respect to the current removal documentation and what the users will expect from those commands

If we want users to keep configuration files we should not use the purge removal in the documentation, because that is the nature of the remove command itself, and based on the output I provided, it does not make sense to keep the keystore file but not the other configuration files or backups.