Periodically "touch" the IPC socket files to prevent inadvertent cleanup by systemd-tmpfilesd

[achimnol]

We need to periodically refresh the atime/mtime timestamps so that IPC socket files do not get deleted.
Unlike normal files, the atime/mtime of IPC socket files are not updated by reading/writing through them without re-opening the file descriptors.

For example, we could update atime/mtime on every 1 hour.

How to use os.utime():

import os
import time

socket_file_path = '/path/to/your/socket.file'

# Get the current time
current_time = time.time()

# Update the access time and modification time to now
os.utime(socket_file_path, (current_time, current_time))

Places to update

• We could write a timer to scan and touch files in the IPC base path as a ai.backend.common module.
• Activate the timer in the manager, agent, storage-proxy, and webserver startup.

Workaround

Until we add this feature, we can workaround the issue by changing the ipc-base-path configuration to a directory outside /tmp or adding additoinal systemd-tmpfilesd rules to exempt the Backend.AI IPC directories.

How to update /etc/tmpfiles.d/tmp.conf:

cp /usr/lib/tmpfiles.d/tmp.conf /etc/tmpfiles.d/

x /tmp/backend.ai
x /tmp/backend.ai/*
x /tmp/backend.ai/ipc
x /tmp/backend.ai/ipc/*
x /tmp/backend.ai/ipc/container
x /tmp/backend.ai/ipc/container/*
x /tmp/backend.ai/manager
x /tmp/backend.ai/manager/ipc

systemctl start systemd-tmpfiles-clean && journalctl -u systemd-tmpfiles-clean -xe -f