Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[utilities][ceph] wildcard support for path in files, fix ceph plugin paths #3312

Merged
merged 1 commit into from
Jul 20, 2023
Merged

Conversation

nkshirsagar
Copy link
Contributor

This lets files triggers use * and also updates the ceph plugins to trigger only if folders are non empty

Closes: #3311


Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

  • Is the commit message split over multiple lines and hard-wrapped at 72 characters?
  • Is the subject and message clear and concise?
  • Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
  • Does the commit contain a Signed-off-by: First Lastname [email protected]?
  • Are any related Issues or existing PRs properly referenced via a Closes (Issue) or Resolved (PR) line?

@packit-as-a-service
Copy link

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

  • sudo yum install -y dnf-plugins-core on RHEL 8
  • sudo dnf install -y dnf-plugins-core on Fedora
  • dnf copr enable packit/sosreport-sos-3312
  • And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

@TurboTurtle
Copy link
Member

My inclination here is to say we should instead target specific files within these subdirs, if an empty subdir can be present on a system where the plugin shouldn't run. At the moment, the ceph plugins here are the only ones that attempt to use wildcards in their files tuples, so this isn't a wide-spread design pattern.

The other thought I have is that the glob detection should go into path_exists() and fork from there, rather than dropping it into _os_wrapper() for efficiency reasons.

I'd prefer if we can identify reliable file(s) for these plugins, but if that isn't practically possible then I'd be ok with extending the functions to handle globs, provided that the logic is handled within path_exists() and not the wrapper function. E.G. (not tested, just theoretical here):

def path_exists(path, sysroot):
    if '*' in path:
        return _os_wrapper(path, sysroot, 'glob', module=glob)
    return _os_wrapper(path, sysroot, 'exists')

@nkshirsagar
Copy link
Contributor Author

My inclination here is to say we should instead target specific files within these subdirs, if an empty subdir can be present on a system where the plugin shouldn't run. At the moment, the ceph plugins here are the only ones that attempt to use wildcards in their files tuples, so this isn't a wide-spread design pattern.

The other thought I have is that the glob detection should go into path_exists() and fork from there, rather than dropping it into _os_wrapper() for efficiency reasons.

I'd prefer if we can identify reliable file(s) for these plugins, but if that isn't practically possible then I'd be ok with extending the functions to handle globs, provided that the logic is handled within path_exists() and not the wrapper function. E.G. (not tested, just theoretical here):

def path_exists(path, sysroot):
    if '*' in path:
        return _os_wrapper(path, sysroot, 'glob', module=glob)
    return _os_wrapper(path, sysroot, 'exists')

Thanks for the feedback Jake. I have seen in typical juju deployed ubuntu setups, the folder name inside the non empty folder is generally ceph-$hostname. But this may not always be the case, nor have I extensively seen other environments like standalone, or RH deployments, so I cannot be sure this will always be the case.

MON+MGR node:
root@juju-215b2d-quincysos-0:/var/lib/ceph/mon# ls
ceph-juju-215b2d-quincysos-0
root@juju-215b2d-quincysos-0:/var/lib/ceph/mon# hostname
juju-215b2d-quincysos-0
root@juju-215b2d-quincysos-0:/var/lib/ceph/mon# cd ..
root@juju-215b2d-quincysos-0:/var/lib/ceph# ls
bootstrap-mds  bootstrap-mgr  bootstrap-osd  bootstrap-rbd  bootstrap-rbd-mirror  bootstrap-rgw  crash  mds  mgr  mon  osd  radosgw  tmp
root@juju-215b2d-quincysos-0:/var/lib/ceph# cd osd/
root@juju-215b2d-quincysos-0:/var/lib/ceph/osd# ls
root@juju-215b2d-quincysos-0:/var/lib/ceph/osd# cd ..
root@juju-215b2d-quincysos-0:/var/lib/ceph# cd mgr/
root@juju-215b2d-quincysos-0:/var/lib/ceph/mgr# ls
ceph-juju-215b2d-quincysos-0

RGW:

root@juju-215b2d-quincysos-4:/var/lib/ceph/radosgw# ls
ceph-rgw.juju-215b2d-quincysos-4
root@juju-215b2d-quincysos-4:/var/lib/ceph/radosgw# hostname
juju-215b2d-quincysos-4


OSD node:
root@juju-215b2d-quincysos-1:/home/ubuntu# cd /var/lib/ceph/osd/
root@juju-215b2d-quincysos-1:/var/lib/ceph/osd# ls
ceph-2  ceph.client.crash.keyring  ceph.client.osd-removal.keyring  ceph.client.osd-upgrade.keyring
root@juju-215b2d-quincysos-1:/var/lib/ceph/osd# 

I think we gain more if we allow wildcards in "files" triggers, so I'll try to put the glob code in path_exists() and test it out.

This lets files triggers use * and also updates the
ceph plugins to trigger only if folders are non empty

Closes: #3311
Signed-off-by: Nikhil Kshirsagar <[email protected]>
@nkshirsagar
Copy link
Contributor Author

nkshirsagar commented Jul 19, 2023

I've tested the changes you suggested in path_exists() and they seem to work well, I've tested it on a mon and osd node.

MON+MGR

root@juju-215b2d-quincysos-0:/home/ubuntu/sos# bin/sos report

sosreport (version 4.5.5)

This command will collect system configuration and diagnostic
information from this Ubuntu system.

For more information on Canonical visit:

        Community Website  : https://www.ubuntu.com/
        Commercial Support : https://www.canonical.com

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.


Press ENTER to continue, or CTRL-C to quit.

Optionally, please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
[plugin:lxd] skipped command 'lxd.buginfo': required kmods missing: iptable_nat, bpfilter, iptable_raw, ip6table_raw, ip6table_mangle, ip6_tables, iptable_filter, ebtables, iptable_mangle, ebtable_filter, ip6table_filter, ip6table_nat.  
[plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.   Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ss -peaonmi': required kmods missing: inet_diag, unix_diag, udp_diag, netlink_diag, tcp_diag, xsk_diag, af_packet_diag.   Use '--allow-system-changes' to enable collection.
[plugin:ufw] skipped command 'ufw status numbered': required kmods missing: bpfilter, iptable_filter.  
[plugin:ufw] skipped command 'ufw app list': required kmods missing: bpfilter, iptable_filter.  
 Running plugins. Please wait ...

  Finishing plugins              [Running: ceph_mon]                                      
  Finished running plugins                                                               
Creating compressed archive...

Your sosreport has been generated and saved in:
	/tmp/sosreport-juju-215b2d-quincysos-0-2023-07-19-fibmbjq.tar.xz

 Size	4.04MiB
 Owner	root
 sha256	1cc712a9d31c678af20f9d13b8ce5c78a334b152d724a7abdbdfc955366affc7

Please send this file to your support representative.

root@juju-215b2d-quincysos-0:/home/ubuntu/sos# tar -xf /tmp/sosreport-juju-215b2d-quincysos-0-2023-07-19-fibmbjq.tar.xz
cd root@juju-215b2d-quincysos-0:/home/ubuntu/sos# sosreport-juju-215b2d-quincysos-0-2023-07-19-fibmbjqbjq
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-0-2023-07-19-fibmbjq# cd sos_logs/
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-0-2023-07-19-fibmbjq/sos_logs# ls
sos.log  ui.log
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-0-2023-07-19-fibmbjq/sos_logs# cat sos.log | grep -vi info
2023-07-19 16:27:44,552 WARNING: [plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.   Use '--allow-system-changes' to enable collection.
2023-07-19 16:27:44,553 WARNING: [plugin:networking] skipped command 'ss -peaonmi': required kmods missing: inet_diag, unix_diag, udp_diag, netlink_diag, tcp_diag, xsk_diag, af_packet_diag.   Use '--allow-system-changes' to enable collection.
2023-07-19 16:27:45,716 WARNING: [plugin:ufw] skipped command 'ufw status numbered': required kmods missing: bpfilter, iptable_filter.  
2023-07-19 16:27:45,717 WARNING: [plugin:ufw] skipped command 'ufw app list': required kmods missing: bpfilter, iptable_filter.  
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-0-2023-07-19-fibmbjq/sos_logs# cd ..
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-0-2023-07-19-fibmbjq# cd sos_commands/ceph_
ceph_common/ ceph_mgr/    ceph_mon/    
root@juju-215b2d-quincysos-0:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-0-2023-07-19-fibmbjq# cd sos_commands/ceph_
ceph_common/ ceph_mgr/    ceph_mon/    

OSD:

root@juju-215b2d-quincysos-1:/home/ubuntu/sos# bin/sos report --all-logs

sosreport (version 4.5.5)

This command will collect system configuration and diagnostic
information from this Ubuntu system.

For more information on Canonical visit:

        Community Website  : https://www.ubuntu.com/
        Commercial Support : https://www.canonical.com

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.


Press ENTER to continue, or CTRL-C to quit.

Optionally, please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
[plugin:lxd] skipped command 'lxd.buginfo': required kmods missing: iptable_nat, iptable_mangle, ip6_tables, bpfilter, iptable_filter, ip6table_filter, ebtables, ebtable_filter, ip6table_nat, ip6table_raw, iptable_raw, ip6table_mangle.  
[plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.   Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ss -peaonmi': required kmods missing: netlink_diag, af_packet_diag, udp_diag, inet_diag, unix_diag, tcp_diag, xsk_diag.   Use '--allow-system-changes' to enable collection.
[plugin:ufw] skipped command 'ufw status numbered': required kmods missing: bpfilter, iptable_filter.  
[plugin:ufw] skipped command 'ufw app list': required kmods missing: bpfilter, iptable_filter.  
 Running plugins. Please wait ...

  Finishing plugins              [Running: ubuntu]                                        
  Finished running plugins                                                               
Creating compressed archive...

Your sosreport has been generated and saved in:
	/tmp/sosreport-juju-215b2d-quincysos-1-2023-07-19-mmekrjt.tar.xz

 Size	33.73MiB
 Owner	root
 sha256	98edca57533d448e9abb3e6b75afa23c39075f838337c44280c0dab3a5dc2eab

Please send this file to your support representative.

root@juju-215b2d-quincysos-1:/home/ubuntu/sos# tar -xf /tmp/sosreport-juju-215b2d-quincysos-1-2023-07-19-mmekrjt.tar.xz
cd sosreport-juju-215b2d-quincysos-1-2023-07-19-mmekrjt
root@juju-215b2d-quincysos-1:/home/ubuntu/sos# cd sosreport-juju-215b2d-quincysos-1-2023-07-19-mmekrjt
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-mmekrjt# cat sos_logs/
sos.log  ui.log   
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-mmekrjt# cat sos_logs/
sos.log  ui.log   
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-mmekrjt# cat sos_logs/sos.log  | grep -vi info
2023-07-19 16:29:59,422 WARNING: [plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.   Use '--allow-system-changes' to enable collection.
2023-07-19 16:29:59,423 WARNING: [plugin:networking] skipped command 'ss -peaonmi': required kmods missing: netlink_diag, af_packet_diag, udp_diag, inet_diag, unix_diag, tcp_diag, xsk_diag.   Use '--allow-system-changes' to enable collection.
2023-07-19 16:29:59,813 WARNING: [plugin:ufw] skipped command 'ufw status numbered': required kmods missing: bpfilter, iptable_filter.  
2023-07-19 16:29:59,814 WARNING: [plugin:ufw] skipped command 'ufw app list': required kmods missing: bpfilter, iptable_filter.  
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-mmekrjt# cd sos_commands/ceph_
ceph_common/ ceph_osd/    
root@juju-215b2d-quincysos-1:/home/ubuntu/sos/sosreport-juju-215b2d-quincysos-1-2023-07-19-mmekrjt# cd sos_commands/ceph_
ceph_common/ ceph_osd/ 

@TurboTurtle TurboTurtle merged commit 221f587 into sosreport:main Jul 20, 2023
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ceph_osd is triggered on mon/mgr nodes , and mon/mgr plugins are triggered on osd nodes
3 participants