improvement(artifacts): switch to syslog-ng logs transport

[dimakr]

Switch artifacts tests from ssh to syslog-ng logs transport.

For most of the artifacts test cases simply changing logs transport from ssh to syslog-ng is enough.
But there are also few scenarios where additional adjustments were needed:

• AWS backend: the SCT-sg-2 security group is added to the aws-sct-builders launch templates. This change allows syslog-ng communication to pass firewall and send logs
  from DB nodes to builder instances.
• nonroot installation of scylla on CentOS/Rocky: syslog-ng configuration is adjusted to read scylla-server logs from file. Also SELinux policies are adjusted to allow
  nonroot syslog-ng process to send logs to remote log destination.
• scylla installation with no-selinux-setup option: SELinux policy is added to allow syslog-ng to use the port of the remote log destination.

For artifacts-amazon2023-* configurations the ssh logs transport is still to be used due to amazonlinux/amazon-linux-2023#639.

Closes: https://github.com/scylladb/qa-tasks/issues/1848
Closes: #8532
Fixes: #9925

Testing

AWS jenkins builders were updated after this change in sdcm/utils/aws_builder.py (i.e. additional SG is used in launch template):

❯ ./sct.py configure-jenkins-builders -c aws -r us-east-1
logged in as arn:aws:sts::797456418907:assumed-role/DeveloperAccessRole/[email protected]
New directory created: /home/dmitriy/sct-results/20250127-223926-992009-configure-jenkins-builders
eu-west-1: create_launch_template
eu-west-1: checking if template needs update
eu-west-1: updating template
eu-west-1: create_auto_scaling_group
eu-west-1: add_scaling_group_to_jenkins
us-east-1: create_launch_template
us-east-1: checking if template needs update
us-east-1: updating template
us-east-1: create_auto_scaling_group
us-east-1: add_scaling_group_to_jenkins

Some artifacts tests on AWS backend:

• 🟢 artifacts-ami-test
• 🟢 artifacts-ubuntu2404-arm-test
• 🟢 artifacts-ubuntu2004-fips
• 🟢 artifacts-oel9-test

Some artifacts tests on GCE backend:

• 🟢 artifacts-gce-image-test
• 🟢 artifacts-debian12-test

Azure backend artifacts test:

• 🟢 artifacts-azure-image-test

Nonroot/selinux installation tests:

• 🟢 artifacts-offline-install centos9 nonroot
• 🟢 artifacts-offline-install rocky9 nonroot
• 🟢 artifacts-offline-install centos9 selinux

PR pre-checks (self review)

• I added the relevant backport labels
• I didn't leave commented-out/debugging code

Reminders

• Add New configuration option and document them (in sdcm/sct_config.py)
• Add unit tests to cover my changes (under unit-test/ folder)
• Update the Readme/doc folder relevant to this change (if needed)

[fruch]

@dimakr

lets test all of the affected artifact tests in the PR (since in previous PR, we missed a few)

[dimakr]

@dimakr

lets test all of the affected artifact tests in the PR (since in previous PR, we missed a few)

Yes , i see that problem popped up for nonroot user cases.
And only on GCE, the AWS based nonroot user artifacts tests are OK..

[fruch]

@dimakr
lets test all of the affected artifact tests in the PR (since in previous PR, we missed a few)

Yes , i see that problem popped up for nonroot user cases. And only on GCE, the AWS based nonroot user artifacts tests are OK..

my guess, maybe we pass down the user-data for cloud-init a bit differently, if scylla is preinstalled or not

[dimakr]

my guess, maybe we pass down the user-data for cloud-init a bit differently, if scylla is preinstalled or not

@fruch
Apparently the root cause is likely scylladb/scylladb#7131.
In the core the fix scylladb/scylladb#7326 introduced a workaround, for environments where writing to journal cannot be done under nonroot installation (e.g. on CentOS). So they introduced writing logs to a file.

For ssh logs transport we have some logic for this case, but not for syslog-ng:

scylla-cluster-tests/sdcm/utils/remote_logger.py

Line 587 in c0ac532

return SSHNonRootScyllaSystemdLogger(node, target_log_file)