[collect] Handle custom node and inherited configs with collections #3852

TrevorBenson · 2024-11-26T06:42:31Z

Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

Is the commit message split over multiple lines and hard-wrapped at 72 characters?
Is the subject and message clear and concise?
Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
Does the commit contain a Signed-off-by: First Lastname [email protected]?
Are any related Issues or existing PRs properly referenced via a Closes (Issue) or Resolved (PR) line?
Are all passwords or private data gathered by this PR obfuscated?

packit-as-a-service · 2024-11-26T06:44:49Z

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

sudo yum install -y dnf-plugins-core on RHEL 8
sudo dnf install -y dnf-plugins-core on Fedora
dnf copr enable packit/sosreport-sos-3852
And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

Signed-off-by: Trevor Benson <[email protected]>

sos/collector/transports/__init__.py

sos/collector/transports/control_persist.py

Signed-off-by: Trevor Benson <[email protected]>

arif-ali · 2024-11-26T08:59:00Z

I've restarted the 2 failed jobs, not sure why they are failing, and it's not the first time, I'll take that away as a separate issue

Signed-off-by: Trevor Benson <[email protected]>

TurboTurtle · 2024-11-26T16:12:18Z

sos/collector/__init__.py

+        collect_grp.add_argument('--inherit-config-file', type=str,
+                                 default=None,
+                                 help='Path to a local config file to'
+                                 ' copy to the remote node and use with'
+                                 ' sos report')


My initial thought was that this would be a toggle, whereby if True we implicitly take the config-file used locally, so sos.conf by default or whatever gets set.

Is there a use case you have where a user would have config_file_A locally that is used for the collect and local report collections, but would have config_file_B also local to the system that is running collect but should be used only for remote node collections?

I suspect any difference is mostly my misinterpretation of the discussion

A new --inherit-config-file (or whatever) option causes collect to transfer the supplied config file to remote nodes and implicitly sets the per-node sos command to use that transferred config file

Not intentional change to your thoughts on implementation.

Is there a use case you have where [...]

Possibly, but not one of my requirements. I plan to use the same config for the collector and the local/remote nodes.

I'll attempt to layout the resulting behavior from this change.

Overview

Three arguments

--config-file: str

--node-config-file: str

--inherit-config-file: bool

Logic

--inherit-config-file is False

--config-file & --node-config-file unset

Current behavior for both collector and report.

--config-file set to/etc/sos/custom1.conf

collector uses custom1.conf

report uses sos.conf

--node-config set to /etc/sos/custom2.conf

collector uses default /etc/sos/sos.conf

report uses (pre-existing) /etc/sos/custom2.conf for the local and remote nodes .

--inherit-config-file is True

--config-file & --node-config-file unset

collector uses /etc/sos/sos.conf

/etc/sos/sos.conf is copied to all nodes in a temp path.

report uses /tmp/sos.XXXXX.conf (local could use default path)

--config-file set to/etc/sos/custom3.conf

collector uses /etc/sos/custom3.conf

/etc/sos/custom3.conf is copied to all nodes in a temp path.

report uses /tmp/sos.XXXXX.conf (local ...)

--node-config-file & --inherit-config-file are an incompatible argument pair. Potentially also --config-file & --node-config-file.

Does this correctly describe what you were thinking?

I've got this logic and additional comments worked out. I will add the new commit for review, but can be dropped/reverted if this wasn't what you were thinking of.

sos/collector/sosnode.py

TurboTurtle · 2024-11-26T16:16:03Z

sos/collector/sosnode.py

+                self.run_command(f"rm {remote_config} ; rmdir {self.tmpdir}",
+                                 use_shell=True,
+                                 need_root=True)


Yeah, same here. Very much not a fan of doing rms like this - the local sos installation on the remote node should take care of any cleanup. If we place the inherited config files in /tmp, then they would be both out of the way and cleaned up separately by long-standing behavior.

I'll presume you mean leaving the inherited conf in the tmp directory and letting systemd-tmpfiles-clean.timer RHEL, or tmpfiles.d for Ubuntu, handle the cleanup on schedule?

It works for my use case, as long as we are comfortable that the file remains might (although probably uncommon) store plugin_options in the file for things like mysql.dbpass, postgresql.password, skydive.password?

Inherit-config-file as toggle No changes will be made Long-standing cleanup behavior Signed-off-by: Trevor Benson <[email protected]>

arif-ali · 2024-11-28T15:27:35Z

I've started building some packages for testing [1], so will do the testing soon

[1] https://launchpad.net/~arif-ali/+archive/ubuntu/sos-pr-3852

jcastill · 2024-11-29T09:00:09Z

@TrevorBenson did you test the oc transport already? I'm asking because I'm setting up an environment to test some other stuff and I can use it to test this PR as well.

arif-ali

tested both the of the new config options, and works as expected from the juju perspective. Thanks for your work on this

EDIT: don't forget to squash your commits, as ideally it is preferred to have one commit per PR

pmoravec · 2024-11-29T15:23:20Z

sos/collector/transports/__init__.py

+                if ret:
+                    return True
+                self.log_info(f"File copy attempt {attempts} failed")
+            self.log_info("File copy failed after 5 attempts")


while attempts < 3 vs. failed after 5 attempts :)

pmoravec · 2024-11-29T15:24:29Z

sos/collector/sosnode.py

+            self.sos_cmd = (
+                f"{self.sos_cmd} {config_file_arg}"
+                if config_file_arg else self.sos_cmd)


Isn't it easier just to have

if config_file_arg: self.sos_cmd = f"{self.sos_cmd} {config_file_arg}"

? But I am ok with the current code.

pmoravec · 2024-11-29T15:28:04Z

Just the 3 / 5 retry attempts are worth to fix, otherwise ACK.

TurboTurtle

Looks good to me overall. Just two super minor nits, and Pavel's comment on the 3 vs 5 attempts. FWIW, I think 3 attempts is sufficient.

TurboTurtle · 2024-11-30T17:17:28Z

sos/collector/transports/control_persist.py

+        cmd = (f"/usr/bin/scp -oControlPath={self.control_path} "
+               f"{fname} {self.opts.ssh_user}@{self.address}:{dest}")
+        res = sos_get_command_output(cmd,
+                                     timeout=10)


Nit: this can fit on one line

TurboTurtle · 2024-11-30T17:17:39Z

sos/collector/transports/juju.py

+        model_option = f"-m {model}" if model else ""
+        cmd = f"juju scp {model_option} -- {fname} {unit}:{dest}"
+        res = sos_get_command_output(cmd,
+                                     timeout=15)


Nit: same here

TurboTurtle · 2024-11-30T17:19:48Z

sos/collector/transports/local.py

@@ -41,6 +41,9 @@ def _retrieve_file(self, fname, dest):
    def _format_cmd_for_exec(self, cmd):
        return cmd

+    def _copy_file_to_remote(self, fname, dest):
+        pass


Hmm. I was initially going to say this should return True, but that might not track given that we may not be writing the the same location (and in this first implementation we already aren't).

This might need some attention later on depending on how much use this functionality sees, but for now I think it's fine.

TrevorBenson force-pushed the sos-trevorbenson-collector-node-inherit-config branch from d8a835a to 8641897 Compare November 26, 2024 06:59

TrevorBenson added 2 commits November 25, 2024 23:11

[collect] Node and Inherit config files

783ca2b

Signed-off-by: Trevor Benson <[email protected]>

[collect] Temporary directory and cleanup

d6bc604

Signed-off-by: Trevor Benson <[email protected]>

TrevorBenson force-pushed the sos-trevorbenson-collector-node-inherit-config branch from 8641897 to d6bc604 Compare November 26, 2024 07:12

jcastill suggested changes Nov 26, 2024

View reviewed changes

sos/collector/transports/__init__.py Show resolved Hide resolved

sos/collector/transports/control_persist.py Outdated Show resolved Hide resolved

[collect] Additional transports

4b56b74

Signed-off-by: Trevor Benson <[email protected]>

TrevorBenson force-pushed the sos-trevorbenson-collector-node-inherit-config branch from 24cdb35 to 4b56b74 Compare November 26, 2024 08:04

[collect] Transport timeouts and retries

b2d5a71

Signed-off-by: Trevor Benson <[email protected]>

TrevorBenson marked this pull request as ready for review November 26, 2024 16:12

TurboTurtle reviewed Nov 26, 2024

View reviewed changes

[collect] Review suggestions

2922e7b

Inherit-config-file as toggle No changes will be made Long-standing cleanup behavior Signed-off-by: Trevor Benson <[email protected]>

TrevorBenson requested review from jcastill, TurboTurtle and arif-ali November 26, 2024 18:24

arif-ali added the Needs Review Review required label Nov 28, 2024

arif-ali approved these changes Nov 29, 2024

View reviewed changes

pmoravec reviewed Nov 29, 2024

View reviewed changes

TurboTurtle reviewed Nov 30, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[collect] Handle custom node and inherited configs with collections #3852

[collect] Handle custom node and inherited configs with collections #3852

TrevorBenson commented Nov 26, 2024

packit-as-a-service bot commented Nov 26, 2024

arif-ali commented Nov 26, 2024

TurboTurtle Nov 26, 2024

TrevorBenson Nov 26, 2024 •

edited

Loading

TrevorBenson Nov 26, 2024

TurboTurtle Nov 26, 2024

TrevorBenson Nov 26, 2024

arif-ali commented Nov 28, 2024

jcastill commented Nov 29, 2024

arif-ali left a comment •

edited

Loading

pmoravec Nov 29, 2024

pmoravec Nov 29, 2024

pmoravec commented Nov 29, 2024

TurboTurtle left a comment

TurboTurtle Nov 30, 2024

TurboTurtle Nov 30, 2024

TurboTurtle Nov 30, 2024 •

edited

Loading

[collect] Handle custom node and inherited configs with collections #3852

Are you sure you want to change the base?

[collect] Handle custom node and inherited configs with collections #3852

Conversation

TrevorBenson commented Nov 26, 2024

packit-as-a-service bot commented Nov 26, 2024

arif-ali commented Nov 26, 2024

Choose a reason for hiding this comment

TrevorBenson Nov 26, 2024 • edited Loading

Choose a reason for hiding this comment

Overview

Logic

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arif-ali commented Nov 28, 2024

jcastill commented Nov 29, 2024

arif-ali left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmoravec commented Nov 29, 2024

TurboTurtle left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TurboTurtle Nov 30, 2024 • edited Loading

Choose a reason for hiding this comment

TrevorBenson Nov 26, 2024 •

edited

Loading

arif-ali left a comment •

edited

Loading

TurboTurtle Nov 30, 2024 •

edited

Loading