Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collect mgr and more osd data for microceph #3378

Merged
merged 2 commits into from
Oct 13, 2023
Merged

Collect mgr and more osd data for microceph #3378

merged 2 commits into from
Oct 13, 2023

Conversation

nkshirsagar
Copy link
Contributor


Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

  • Is the commit message split over multiple lines and hard-wrapped at 72 characters?
  • Is the subject and message clear and concise?
  • Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
  • Does the commit contain a Signed-off-by: First Lastname [email protected]?
  • Are any related Issues or existing PRs properly referenced via a Closes (Issue) or Resolved (PR) line?

Now that canonical/microceph#160 is fixed,
gather the ceph daemon command outputs for OSD nodes.

Signed-off-by: Nikhil Kshirsagar <[email protected]>
@nkshirsagar
Copy link
Contributor Author

nkshirsagar commented Oct 10, 2023

Some testing on microceph mgr and osd (colocated) nodes

root@juju-c97b0f-seg-test-2:/home/ubuntu/sosreport/sos# sos report -o ceph_mgr

sosreport (version 4.6.0)

This command will collect system configuration and diagnostic
information from this Ubuntu system.

For more information on Canonical visit:

        Community Website  : https://www.ubuntu.com/
        Commercial Support : https://www.canonical.com

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.


Press ENTER to continue, or CTRL-C to quit.

Optionally, please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Starting 1/1   ceph_mgr        [Running: ceph_mgr]                                      
  Finished running plugins                                                               
Creating compressed archive...

Your sosreport has been generated and saved in:
	/tmp/sosreport-juju-c97b0f-seg-test-2-2023-10-10-ffhljhm.tar.xz

 Size	521.14KiB
 Owner	root
 sha256	6b6fc6194a1e6d9cf7083743d5a35237ae06265a207d5d660eda6738297bc31b

Please send this file to your support representative.

root@juju-c97b0f-seg-test-2:/home/ubuntu/sosreport/sos# cd /tmp
root@juju-c97b0f-seg-test-2:/tmp# tar -xf /tmp/sosreport-juju-c97b0f-seg-test-2-2023-10-10-ffhljhm.tar.xz
root@juju-c97b0f-seg-test-2:/tmp# cd /tmp/sosreport-juju-c97b0f-seg-test-2-2023-10-10-ffhljhm
root@juju-c97b0f-seg-test-2:/tmp/sosreport-juju-c97b0f-seg-test-2-2023-10-10-ffhljhm# ls
sos_commands  sos_logs  sos_reports  var  version.txt
root@juju-c97b0f-seg-test-2:/tmp/sosreport-juju-c97b0f-seg-test-2-2023-10-10-ffhljhm# ls -R
.:
sos_commands  sos_logs  sos_reports  var  version.txt

./sos_commands:
ceph_mgr

./sos_commands/ceph_mgr:
ceph_balancer_status
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_config_diff
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_config_show
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_dump_cache
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_dump_mempools
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_dump_osd_network
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_mds_requests
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_mds_requests.1
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_mds_sessions
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_mds_sessions.1
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_objecter_requests
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_perf_dump
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_perf_histogram_dump
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_perf_histogram_schema
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_perf_schema
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_status
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_version
ceph_log_last_cephadm
ceph_orch_device_ls
ceph_orch_host_ls
ceph_orch_ls
ceph_orch_ls_--export
ceph_orch_ps
ceph_orch_status_--detail
ceph_orch_upgrade_status
json_output

./sos_commands/ceph_mgr/json_output:
ceph_balancer_status_--format_json-pretty   ceph_orch_ls_--format_json-pretty
ceph_log_last_cephadm_--format_json-pretty  ceph_orch_ps_--format_json-pretty
ceph_orch_device_ls_--format_json-pretty    ceph_orch_status_--detail_--format_json-pretty
ceph_orch_host_ls_--format_json-pretty      ceph_orch_upgrade_status_--format_json-pretty
ceph_orch_ls_--export_--format_json-pretty

./sos_logs:
sos.log  ui.log

./sos_reports:
manifest.json  sos.html  sos.json  sos.txt

./var:
snap

./var/snap:
microceph

./var/snap/microceph:
common

./var/snap/microceph/common:
logs

./var/snap/microceph/common/logs:
ceph-mgr.juju-c97b0f-seg-test-2.log
root@juju-c97b0f-seg-test-2:/tmp/sosreport-juju-c97b0f-seg-test-2-2023-10-10-ffhljhm# 


-----------

root@juju-c97b0f-seg-test-2:/home/ubuntu/sosreport/sos# bin/sos report -o ceph_osd 

sosreport (version 4.6.0)

This command will collect system configuration and diagnostic
information from this Ubuntu system.

For more information on Canonical visit:

        Community Website  : https://www.ubuntu.com/
        Commercial Support : https://www.canonical.com

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.


Press ENTER to continue, or CTRL-C to quit.

Optionally, please enter the case id that you are generating this report for []: 

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Starting 1/1   ceph_osd        [Running: ceph_osd]                                      
  Finished running plugins                                                               
Creating compressed archive...

Your sosreport has been generated and saved in:
	/tmp/sosreport-juju-c97b0f-seg-test-2-2023-10-10-msnymqh.tar.xz

 Size	360.89KiB
 Owner	root
 sha256	def311396da133adb9b2d34d5f5ae4fb6960b5d1fe39f4c754bab8a5c98c6134

Please send this file to your support representative.

root@juju-c97b0f-seg-test-2:/home/ubuntu/sosreport/sos# cd /tmp
root@juju-c97b0f-seg-test-2:/tmp# tar -xf /tmp/sosreport-juju-c97b0f-seg-test-2-2023-10-10-msnymqh.tar.xz
root@juju-c97b0f-seg-test-2:/tmp# cd /tmp/sosreport-juju-c97b0f-seg-test-2-2023-10-10-msnymqh
root@juju-c97b0f-seg-test-2:/tmp/sosreport-juju-c97b0f-seg-test-2-2023-10-10-msnymqh# ls -R
.:
dev  sos_commands  sos_logs  sos_reports  var  version.txt

./dev:
disk  dm-0  mapper  vdc

./dev/disk:
by-id

./dev/disk/by-id:
virtio-46c76c00-48fd-4f8d-9

./dev/mapper:
luksosd-2

./sos_commands:
ceph_osd

./sos_commands/ceph_osd:
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_bluefs_stats
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_bluestore_bluefs_available
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_bluestore_bluefs_device_info
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_config_diff
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_config_show
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_dump_blocked_ops
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_dump_blocklist
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_dump_historic_ops_by_duration
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_dump_historic_slow_ops
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_dump_mempools
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_dump_op_pq_state
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_dump_ops_in_flight
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_dump_osd_network
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_dump_pgstate_history
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_dump_recovery_reservations
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_dump_reservations
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_dump_scrubs
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_dump_watchers
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_get_mapped_pools
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_list_devices
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_list_unfound
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_log_dump
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_objecter_requests
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_ops
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_perf_dump
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_perf_histogram_dump
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_status
ceph_daemon_.var.snap.microceph.697.run.ceph-osd.2.asok_version

./sos_logs:
sos.log  ui.log

./sos_reports:
manifest.json  sos.html  sos.json  sos.txt

./var:
snap

./var/snap:
microceph

./var/snap/microceph:
common

./var/snap/microceph/common:
data  logs

./var/snap/microceph/common/data:
osd

./var/snap/microceph/common/data/osd:
ceph-2

./var/snap/microceph/common/data/osd/ceph-2:
bfm_blocks           bfm_size  ceph_fsid                  fsid        mkfs_done            type
bfm_blocks_per_key   block     ceph_version_when_created  kv_backend  ready                unencrypted
bfm_bytes_per_block  bluefs    created_at                 magic       require_osd_release  whoami

./var/snap/microceph/common/logs:
ceph-osd.2.log

@nkshirsagar nkshirsagar marked this pull request as draft October 10, 2023 05:25
@packit-as-a-service
Copy link

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

  • sudo yum install -y dnf-plugins-core on RHEL 8
  • sudo dnf install -y dnf-plugins-core on Fedora
  • dnf copr enable packit/sosreport-sos-3378
  • And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

@nkshirsagar nkshirsagar changed the title Collect mgr data for microceph Collect mgr and more osd data for microceph Oct 10, 2023
@nkshirsagar nkshirsagar marked this pull request as ready for review October 10, 2023 05:32
Copy link

@UtkarshBhatthere UtkarshBhatthere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one minor nit.

Copy link
Contributor

@pponnuvel pponnuvel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

sos/report/plugins/ceph_mgr.py Outdated Show resolved Hide resolved
sos/report/plugins/ceph_mgr.py Outdated Show resolved Hide resolved
@nkshirsagar
Copy link
Contributor Author

I just notice during testing, while the right files seem collected, this issue,

root@juju-c97b0f-seg-test-2:/tmp/sosreport-juju-c97b0f-seg-test-2-2023-10-10-kjftewt/sos_commands/ceph_mgr# ls
ceph_balancer_status                                                                                ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_perf_schema
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_config_diff            ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_status
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_config_show            ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_version
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_dump_cache             ceph_log_last_cephadm
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_dump_mempools          ceph_orch_device_ls
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_dump_osd_network       ceph_orch_host_ls
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_mds_requests           ceph_orch_ls
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_mds_sessions           ceph_orch_ls_--export
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_objecter_requests      ceph_orch_ps
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_perf_dump              ceph_orch_status_--detail
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_perf_histogram_dump    ceph_orch_upgrade_status
ceph_daemon_.var.snap.microceph.697.run.ceph-mgr.juju-c97b0f-seg-test-2.asok_perf_histogram_schema  json_output
root@juju-c97b0f-seg-test-2:/tmp/sosreport-juju-c97b0f-seg-test-2-2023-10-10-kjftewt/sos_commands/ceph_mgr# cat ceph_orch_ls
Error ENOENT: No orchestrator configured (try `ceph orch set backend`)

Is this expected, could I get your thoughts please @UtkarshBhatthere @pponnuvel ?

Copy link
Member

@arif-ali arif-ali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

made a few comments, they're minor nits

sos/report/plugins/ceph_mgr.py Show resolved Hide resolved
sos/report/plugins/ceph_mgr.py Show resolved Hide resolved
sos/report/plugins/ceph_mgr.py Outdated Show resolved Hide resolved
sos/report/plugins/ceph_osd.py Show resolved Hide resolved
sos/report/plugins/ceph_osd.py Show resolved Hide resolved
@UtkarshBhatthere
Copy link

Is this expected, could I get your thoughts please @UtkarshBhatthere @pponnuvel ?

Microceph does NOT use the ceph orchestrator backend. (Possible feature enhancement for future) So this looks okay to me.

@nkshirsagar
Copy link
Contributor Author

nkshirsagar commented Oct 10, 2023

Is this expected, could I get your thoughts please @UtkarshBhatthere @pponnuvel ?

Microceph does NOT use the ceph orchestrator backend. (Possible feature enhancement for future) So this looks okay to me.

Maybe I should skip those collections then for micro ceph case...

@UtkarshBhatthere
Copy link

UtkarshBhatthere commented Oct 10, 2023

Maybe I should skip those collections then for micro ceph case...

Yeah, orch backend implementation (although not researched too well) is a bit of a task I do not see it happening "soon". Another option would be to collect it only when orch backend is set (can use ceph orch status to check), so that it does not need re-enabling when we implement it. @nkshirsagar

Copy link
Member

@TurboTurtle TurboTurtle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, just some minor styling nits.

sos/report/plugins/ceph_osd.py Outdated Show resolved Hide resolved
sos/report/plugins/ceph_mgr.py Outdated Show resolved Hide resolved
sos/report/plugins/ceph_mgr.py Show resolved Hide resolved
@nkshirsagar
Copy link
Contributor Author

Maybe I should skip those collections then for micro ceph case...

Yeah, orch backend implementation (although not researched too well) is a bit of a task I do not see it happening "soon". Another option would be to collect it only when orch backend is set (can use ceph orch status to check), so that it does not need re-enabling when we implement it. @nkshirsagar

I've changed the code to check the ceph orch status and if ENOENT is seen in its output, we do not collect any of the orch commands. This change will apply to the ceph plugins as well, since its in common code, for ceph as well as microceph.

Signed-off-by: Nikhil Kshirsagar <[email protected]>
@nkshirsagar
Copy link
Contributor Author

Thank you everyone for the feedback, I think this one is in shape to merge now.

Copy link
Contributor

@pmoravec pmoravec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A minor nitpick, I am OK to merge the PR as is.

Nice collaboration!

@nkshirsagar
Copy link
Contributor Author

A minor nitpick, I am OK to merge the PR as is.

Nice collaboration!

Thank you. I won't make any more changes to this PR now, so please merge when possible.

@TurboTurtle TurboTurtle merged commit 1c6cf87 into sosreport:main Oct 13, 2023
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants