Excessive Log File Accumulation in /var/log/kairos Due to Timestamp-Based Naming

[FelsiyaVasanthi]

We have observed that despite using maxage for log deletion, the issue persists because log files are generated every hour with unique timestamps. As a result, logrotate treats each file as a new log, and when a file is compressed and rotated, the original version remains. Hence, logrotate won't help much in rotating the old logs.
For ex: The log file which was part of 07th Feb log generation, is still available, as .log file will be the active file always. Refer the below screenshot.
Removing create or copytruncate may not be ideal, as it could impact the logging process. Given this, do we have any other recommendations to address this issue effectively?

Kairos version:
NAME="SLE Micro"
VERSION="5.4"
VERSION_ID="5.4"
PRETTY_NAME="SUSE Linux Enterprise Micro for Rancher 5.4"
ID="sle-micro-rancher"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sle-micro-rancher:5.4"
KAIROS_NAME="kairos-core-sles"
KAIROS_VERSION="v3.1.3"
KAIROS_ID="sles"
KAIROS_ID_LIKE="kairos-core-sles"
KAIROS_VERSION_ID="v3.1.3"
KAIROS_PRETTY_NAME="kairos-core-sles v3.1.3"
KAIROS_BUG_REPORT_URL="https://github.com/spectrocloud/CanvOS/issues"
KAIROS_HOME_URL="https://github.com/spectrocloud/CanvOS"
KAIROS_IMAGE_REPO="spectrocloud/CanvOS"
KAIROS_IMAGE_LABEL="latest"
KAIROS_GITHUB_REPO=""
KAIROS_VARIANT="sles"
KAIROS_FLAVOR="sles"
KAIROS_ARTIFACT="kairos-core-sles-v3.1.3"

CPU architecture, OS, and Version:
Linux edge-21f135429dda5453ed78724ee89aeedb 5.14.21-150500.55.59-default #1 SMP PREEMPT_DYNAMIC Thu Apr 18 12:59:33 UTC 2024 (e8ae24a) x86_64 x86_64 x86_64 GNU/Linux

Describe the bug
As the log file name used in the /var/log/kairos has the timestamps, hence the logrotate utility won't be helpful in rotating it. It would be better to have different way of naming it

To Reproduce
Take any old or 1 week old kairos based edge host. Monitor the /var/log/kairos folder

Expected behavior
Filenames should be such that it can be rotated

[jimmykarily]

The file timestamping is happening here: https://github.com/kairos-io/kairos-sdk/blob/c52bfb57089d55ea3666b94747522a89941b481c/types/logger.go#L26

While I try to understand why we do that (and not just let logrotate rotate the thing), a workaround could be to implement manual cleanup by setting up a cron job that parses those files and cleans up old ones (that's what timestamps are there for right?).

I'll play with this a bit to see if it could work.

[jimmykarily]

Actually, with the time format including milliseconds, it means we create a new file every time. We never re-use a file. Doesn't affect the above workaround, but I'm just pointing out that it's not a "rotation mechanism" implemented there.

[jimmykarily]

We could probably improve the situation if we didn't os.Create the file (which truncates/empties it if is exists) but rather open if for appending here: https://github.com/kairos-io/kairos-sdk/blob/c52bfb57089d55ea3666b94747522a89941b481c/types/logger.go#L33

That's only if we really want to take care of rotation and timestamping. Currently I leaning towards appending always to the same file and letting logrotate do its magic. Question, do we have logrotate in all flavors? (e.g. alpine)

[Itxaka]

kairos-io/kairos-sdk#566 from yesterday

IIRC the timestamp was due to several processes writing to the same file and making it a mess to understand

[jimmykarily]

In the meantime, as a workaround, one could rely either on the file metadata to cleanup old files, or on the filename. Chatgpt quickly came up with this (working) script for cleanup:

#!/bin/bash

# Directory where logs are stored
LOG_DIR="/var/log/kairos"

# Number of minutes (N) before logs should be deleted
N_MINUTES=1  # Change this to your desired threshold

# Get the current timestamp in the same format as the log filenames
CURRENT_TIME=$(date +"%Y%m%d%H%M%S")

# Convert N minutes ago to timestamp
THRESHOLD_TIME=$(date -d "$N_MINUTES minutes ago" +"%Y%m%d%H%M%S")

# Loop through log files
for file in "$LOG_DIR"/*.log; do
    # Extract the timestamp from the filename
    TIMESTAMP=$(echo "$file" | grep -oP '\d{14}\.\d{4}')

    # Remove the decimal part and keep only the first 14 digits
    TIMESTAMP=${TIMESTAMP%.*}

    # Check if the timestamp is valid and older than the threshold
    if [[ ! -z "$TIMESTAMP" && "$TIMESTAMP" -lt "$THRESHOLD_TIME" ]]; then
        echo "Deleting: $file"
        rm -f "$file"
    fi
done

echo "Cleanup complete: Removed logs older than $N_MINUTES minutes."

A crontab entry like this:

*/5 * * * * /path/to/cleanup_logs.sh

will make sure it gets triggered every 5 minutes.

[jimmykarily]

on an already installed system, adding this in /oem/90_custom.yaml puts it in the crontab:

stages:
    initramfs:
        - commands:
            - (crontab -l 2>/dev/null; echo "* * * * * /home/kairos/cleanup_logs.sh") | crontab -