journald archiving

[natefoo]

Log archiving is something that can be useful - for example, I tend to keep my Galaxy logs for a pretty long time so that I can go back and find old job errors, find the last time I saw particular error messages (or when they started), etc.

With supervisor you could configure log rotation and then use an external utility to archive old logs and prune the archive. With journald this is trickier - although journald rotates its storage, it's all one big journal file. Also, on EL (even EL8) the default journald configuration uses the tmpfs-mounted /run/log rather than /var/log, so logs are lost on reboot.

From a quick look I have two ideas of potential solutions:

1. Use journal namespaces. From systemd-journald.service(8):

   JOURNAL NAMESPACES
   Journal 'namespaces' are both a mechanism for logically isolating the log stream of projects consisting of one or more services from the rest of the system and a mechanism for improving performance. Multiple journal namespaces may exist simultaneously, each defining its own, independent log stream managed by its own instance of systemd-journald. Namespaces are independent of each other, both in the data store and in the IPC interface. By default only a single 'default' namespace exists, managed by systemd-journald.service (and its associated socket units). Additional namespaces are created by starting an instance of the [email protected] service template. The instance name is the namespace identifier, which is a short string used for referencing the journal namespace. Service units may be assigned to a specific journal namespace through the LogNamespace= unit file setting, see systemd.exec(5) for details. The --namespace= switch of journalctl(1) may be used to view the log stream of a specific namespace. If the switch is not used the log stream of the default namespace is shown, i.e. log data from other namespaces is not visible.

   If the Galaxy processes were in their own journald namespace then when you archive the rotated journals, you're only archiving the Galaxy logs.

   Unfortunately this is a relatively new feature, not even EL8 has it.

2. Periodically export logs to text using e.g. journalctl -u <whatever>.service --since yesterday --until today for all the previous day's logs, and then archive those exports.

This may be more of an Ansible/deployment issue than Gravity itself, but I figured I should capture it somewhere.

[hexylena]

Also, on EL (even EL8) the default journald configuration uses the tmpfs-mounted /run/log rather than /var/log, so logs are lost on reboot.

Wow, that's... a really terrible choice. There's so much useful information from previous boots.

You should definitely be overwriting that.

namespaces

Huh, didn't know about namespaces, very neat. We should definitely try and use that if possible.

log retention

I guess this is primarily a log retention issue right, we could open up some switches for that like in https://unix.stackexchange.com/questions/709570/how-to-keep-systemd-journal-for-one-year/712085#712085

Or we make it easier to do something like systemd-journal-remote to centralise logs and archive in one place?

[natefoo]

Also, on EL (even EL8) the default journald configuration uses the tmpfs-mounted /run/log rather than /var/log, so logs are lost on reboot.

Wow, that's... a really terrible choice. There's so much useful information from previous boots.

I was very surprised to discover it, to say the least.

namespaces

Huh, didn't know about namespaces, very neat. We should definitely try and use that if possible.

We can definitely work on it. Won't help for EL8, but newer Linuxes should eventually all have it.

log retention

I guess this is primarily a log retention issue right, we could open up some switches for that like in https://unix.stackexchange.com/questions/709570/how-to-keep-systemd-journal-for-one-year/712085#712085

Or we make it easier to do something like systemd-journal-remote to centralise logs and archive in one place?

Yeah. So I guess this (and namespaces) are more of a thing to address in the GTN?

[hexylena]

Yeah. So I guess this (and namespaces) are more of a thing to address in the GTN?

Wouldn't hurt to have a role for making the changes to journald.conf / setting up remote archiving.