-
Notifications
You must be signed in to change notification settings - Fork 50
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
mon: add NVMe-oF gateway monitor and HA
- gateway submodule Fixes: https://tracker.ceph.com/issues/64777 This PR adds high availability support for the nvmeof Ceph service. High availability means that even in the case that a certain GW is down, there will be another available path for the initiator to be able to continue the IO through another GW. High availability is achieved by running nvmeof service consisting of at least 2 nvmeof GWs in the Ceph cluster. Every GW will be seen by the host (initiator) as a separate path to the nvme namespaces (volumes). The implementation consists of the following main modules: - NVMeofGWMon - a PaxosService. It is a monitor that tracks the status of the nvmeof running services, and take actions in case that services fail, and in case services restored. - NVMeofGwMonitorClient – It is an agent that is running as a part of each nvmeof GW. It is sending beacons to the monitor to signal that the GW is alive. As a part of the beacon, the client also sends information about the service. This information is used by the monitor to take decisions and perform some operations. - MNVMeofGwBeacon – It is a structure used by the client and the monitor to send/recv the beacons. - MNVMeofGwMap – The map is tracking the nvmeof GWs status. It also defines what should be the new role of every GW. So in the events of GWs go down or GWs restored, the map will reflect the new role of each GW resulted by these events. The map is distributed to the NVMeofGwMonitorClient on each GW, and it knows to update the GW with the required changes. It is also adding 3 new mon commands: - nvme-gw create - nvme-gw delete - nvme-gw show The commands are used by the ceph adm to update the monitor that a new GW is deployed. The monitor will update the map accordingly and will start tracking this GW until it is deleted. Signed-off-by: Leonid Chernin <[email protected]> Signed-off-by: Alexander Indenbaum <[email protected]>
- Loading branch information
Showing
37 changed files
with
3,885 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| // -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- | ||
| // vim: ts=8 sw=2 smarttab | ||
| /* | ||
| * Ceph - scalable distributed file system | ||
| * | ||
| * Copyright (C) 2023 IBM Inc | ||
| * | ||
| * Author: Alexander Indenbaum <[email protected]> | ||
| * | ||
| * This is free software; you can redistribute it and/or | ||
| * modify it under the terms of the GNU Lesser General Public | ||
| * License version 2.1, as published by the Free Software | ||
| * Foundation. See file COPYING. | ||
| * | ||
| */ | ||
|
|
||
| #include <pthread.h> | ||
|
|
||
| #include "include/types.h" | ||
| #include "include/compat.h" | ||
| #include "common/config.h" | ||
| #include "common/ceph_argparse.h" | ||
| #include "common/errno.h" | ||
| #include "common/pick_address.h" | ||
| #include "global/global_init.h" | ||
|
|
||
| #include "nvmeof/NVMeofGwMonitorClient.h" | ||
|
|
||
| static void usage() | ||
| { | ||
| std::cout << "usage: ceph-nvmeof-monitor-client\n" | ||
| " --gateway-name <GW_NAME>\n" | ||
| " --gateway-address <GW_ADDRESS>\n" | ||
| " --gateway-pool <CEPH_POOL>\n" | ||
| " --gateway-group <GW_GROUP>\n" | ||
| " --monitor-group-address <MONITOR_GROUP_ADDRESS>\n" | ||
| " [flags]\n" | ||
| << std::endl; | ||
| generic_server_usage(); | ||
| } | ||
|
|
||
| /** | ||
| * A short main() which just instantiates a Nvme and | ||
| * hands over control to that. | ||
| */ | ||
| int main(int argc, const char **argv) | ||
| { | ||
| ceph_pthread_setname(pthread_self(), "ceph-nvmeof-monitor-client"); | ||
|
|
||
| auto args = argv_to_vec(argc, argv); | ||
| if (args.empty()) { | ||
| std::cerr << argv[0] << ": -h or --help for usage" << std::endl; | ||
| exit(1); | ||
| } | ||
| if (ceph_argparse_need_usage(args)) { | ||
| usage(); | ||
| exit(0); | ||
| } | ||
|
|
||
| auto cct = global_init(nullptr, args, CEPH_ENTITY_TYPE_CLIENT, | ||
| CODE_ENVIRONMENT_UTILITY, // maybe later use CODE_ENVIRONMENT_DAEMON, | ||
| CINIT_FLAG_NO_DEFAULT_CONFIG_FILE); | ||
|
|
||
| pick_addresses(g_ceph_context, CEPH_PICK_ADDRESS_PUBLIC); | ||
|
|
||
| global_init_daemonize(g_ceph_context); | ||
| global_init_chdir(g_ceph_context); | ||
| common_init_finish(g_ceph_context); | ||
|
|
||
| NVMeofGwMonitorClient gw_monitor_client(argc, argv); | ||
| int rc = gw_monitor_client.init(); | ||
| if (rc != 0) { | ||
| std::cerr << "Error in initialization: " << cpp_strerror(rc) << std::endl; | ||
| return rc; | ||
| } | ||
|
|
||
| return gw_monitor_client.main(args); | ||
| } | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.