deploy fails when public_network contains more than one network #131
Comments
|
Good eye! I didn't even realize that ceph.conf could support multiple networks. I'll look into it and figure out the best way to handle it in the code. |
|
Even after applying #132 I'm having problems getting quorum. What seems to be happening is that ceph.conf is being generated with an incomplete set of mon hosts listed. One of the nodes (the first listed in "mon initial members", the lowest in ASCII sort order, and also by coincidence the oddball node on the different subnet) gets only itself listed in "mon host". The other two nodes get that first IP listed and then their own, but none of them have the full set of 3 listed. What I'm getting at is that maybe my proposed fix wasn't quite right and |
|
Hmmm can you provide some more details?
|
|
The systems are set up just as described in the ceph.com guide "Deploying Ceph (Those are not the real IP addresses.) Here's my Ceph environment: When run, this is relevant part of the generated ceph.conf on bull: This is from superman: This is from zod: So, I'm fairly new to doing anything substantial with Ceph and not familiar |
|
Here's mon_status output from bull: This is from zod: This is from superman: |
|
The way this cookbook works, the mon_initial_nodes is optional. When a new mon is provisioned, the cookbook does a search for any other mons and runs "add_bootstrap_peer_hint" on the new mon, providing the link to the existing cluster. During the log output, you should see this happen. You should see several Are all of the nodes in the same Chef environment? Do you see all 3 nodes when you do a 'knife search node "ceph_is_mon:true AND chef_environment:Chef"' ? What about 'knife search node "(ceph_is_mon:true AND chef_environment:Chef) AND (ceph_bootstrap_osd_key:*)"' ? Do you see 3 sets of add_boostrap_peer_hint commands on all of the chef runs? The mon_addresses function directly produces the mons config option. Also, a small trick in the mon_addresses function: If you are running Chef on a node that already has a running Cephmon node, the cookbook will query it for the monmap's list of IPs. Try stopping the ceph-mon on bull, and try running Chef with just a runlist of ceph::conf. This will merely create the ceph.conf file by using Chef searches, instead of using the local mon's monmap. |
|
Hm... something's amok here: When I run the same command by hand, it succeeds?
Confirmed all Ceph nodes are using the Ceph environment. Your first search only returns bull. The second search (with Chef->Ceph typo fixed) doesn't return anything. Thanks for taking the time to walk me through this. |
|
This cookbook is able to spin up a fresh cluster, and some improvements a few months ago made it much more reliable. The first mon node will do a search, not find any nodes, and create a cluster. The second node will do a search, find the first node, and should connect to it. As for your strange Since your knife search didn't find the two nodes, this gives me some clues about what is going on. The two-node cluster is successfully talking to each other, and so when Chef calls mon_addresses it uses the monmap to generate the ceph.conf. Is the ceph::mon recipe running on those nodes? This is what sets the ceph['is_mon'] attribute, which is what is searched for by any non-mon Ceph member. After you run ceph::mon on the two nodes, your knife search should find them, and then your third node, by running just ceph::conf without a running monitor, should generate a more-correct config file. Then, remove the /var/lib/ceph/mon/ceph-bull directory, so that the single-node cluster is effectively destroyed and Chef can join it to the existing 2 node cluster. Actually, scratch some of that: Looking at your mon_status from before, I think your mon_initial_members is actually confusing things too. This is what is generating the spurious mons from that output, mons that don't have an IP address. This is a completely fresh setup, right? No data? If so, I'd recommend stopping all the ceph-mon daemons, removing all of the /var/lib/ceph/mon directories on the systems, removing mon_initial_members from the environment, and trying again. Run ceph::mon on a first system, and it should create a single-mon cluster, with all of the PGs stuck inactive because there's no storage, that's cool. Doing that above knife search should show that single node. Then, run ceph::mon on a second system. It should successfully run a chef search to find the other node and join to it. The |
|
I finally got a chance to try this. I removed mon_initial_members , removed /var/lib/ceph/mon/ceph-* on the nodes, and hand-ran chef-client on the 3 monitors in succession and this seems to have got everything going! Thanks!
My next issue is that ceph-disk isn't taking /dev/mapper/36* disk device names: But that's something else so this issue can be closed now. |
|
Actually, one more question. So again, I'm a Chef newbie. But, when I added the roles to the nodes I used knife node edit, and added roleceph-mon to each node's run list. However, now knife node show is weird - for bull it shows ceph-mon in Roles. But it doesn't show that role for any of the other mons even though they're in the run list, chef-client has been run, and all 3 show up in the cluster. `knife node search "ceph_is_mon:true" only returns bull. So I'm confused. It seems like it created/added the mons properly, so why isn't chef showing them? |
|
I'm glad to hear that! I think the mon_initial_nodes caveat is due to this cookbook's usage of add_bootstrap_peer_hint, and for hand configurations. About your /dev/mapper/36* named disks.. those look like SAN IDs, are you sure it shouldn't be something like /dev/disk/by-id/36* ? However, I have not tried symlinked devices, so this would be new for me too. I do know that the Roles and Recipes lists are updated by the client, not on the server. So, if you have run just the role[standalone], it would set the Roles and Recipes as zod and superman show. And then, when you set their runlist to include role[ceph-mon] and role[ceph-osd], it will run most of they way, and as you said above, crash in the role[ceph-osd] section. This will cause the node to not be saved back to the server, including the new Roles and Recipes. This would also prevent the nodes from showing up in the ceph_is_mon search, because that attribute isn't being saved up to the server. An alternative explanation: Somewhere inside your role[standalone], you are doing a node.save(). This saves the node before the ceph::mon recipe has run, so it doesn't have the ceph_is_mon attribute set. and then when ceph-osd crashes it can't save the completed node with the ceph_is_mon attribute. |
|
Ah, ok, that makes sense. Both zod and superman are also osd nodes and until I resolve the disk naming, the osd setup is failing, so that would explain it. I didn't realize that something would only show the Role when it had successfully completed. Those /dev/mapper names aren't symlinks, they're block devices (created I think by multipathd, or maybe because of the way I have the LSI HBA configured). I was trying to use those names instead of /dev/disk/* or /dev/sd* because it was more obvious looking at them which devices were my SAS OSD devices and which were the internal SATA OS disks. I can use other names easily enough, but it surprised me since they're actual device nodes, not symlinks. Will try again with different device names and let you know what happens. |
|
Okay. Changed to Now lex and spiderman are the problem... chef got the disks prepared okay, and they show up as prepared in I can't find any other errors in the logs indicating why it's not starting. ceph-osd.0.log shows a lot of lines like this (for all the disks): ...which doesn't really tell me anything. It's not mounting the filesystems or anything. It feels like it's unable to find the cluster or something, but I'm not sure. The generated ceph.conf does have the mon hosts listed correctly and the cluster network specified correctly, and they can talk to each other just fine. I do note that |
|
(Copying the keyring file over solved the |
|
Ah... running |
|
About the disks: You can run the ceph-osd recipe and give it temporary names for the initial setup, and then after that the ceph-osd recipe won't try to initialize any other drives, and in fact won't even check if the original names are still there. So, you can give it the temporary names, and then Chef will think the server is happy and done. This lets you stop ceph-osd and enable multipathd to get the drives happy and secure, and then run ceph-disk-activate to start them up. You might need to tweak the /lib/udev/rules.d/95-ceph-osd.rules file to make sure it ignores the un-multipathed drive during boot and hotplug, but maybe the udev rule order will ensure that the multipath device gets created first, so that the underlying device is locked and ceph-osd will ignore it for being busy. Anyways!
So! You should check on your new OSD server to see if there is a Let me know how it goes! |
|
Ok. All nodes see the bootstrap-osd key from within So, it looks like somehow chef doesn't know the bootstrap key but it did manage to install it on the mons somehow? Not on the osd-only servers though. |
|
@hufman sorry I let you manage this problem. TL;DR for me :p |
|
...and I'm sorry this went way beyond the initial issue! But I do appreciate the help! |
|
Hmmm can I have you try something? What happens if you change recipes/mon.rb:125 to use |
|
To answer your second question first, I don't see that ruby block running (on lex) based on chef-client output. The closest I get is this: zod shows the same thing except the final start succeeds. Those are the only references to anything bootstrap on either machine. Do I need to redeploy a mon from scratch for a valid test? |
|
That code update doesn't seem to have caused any change, even after I stopped the mon on zod, deleted /var/lib/ceph/mon/ceph-zod, and ran chef-client on it hoping that it would update the key. Tried again after also deleting bootstrap-osd/ceph.key, still no luck. Not sure what else I need to do to get that block to run? |
|
Hmmm that's very odd... that ruby_block should be running on all the mon nodes... Did you leave Bull, being only a ceph_mon node, should definitely be running that ruby_block every time chef-client runs. You saw it doing the Alternatively, to get your cluster up and running... you could just put the osd bootstrap key in place manually. The osd cookbook will use the keyring stored in |
|
For what it's worth, I don't see a bootstrap_osd_key defined in zod's node either, but those osds came up okay. (I couldn't find this value defined in chef anywhere.) I'm looking at the same if statement in the code that you're looking at and I have no idea why it's not running either... I'm going to comment out the if statement and see what happens... |
|
Commenting out the
(repeated for all disks) |
|
Well huh, that's tricky. I manually set the fsid in my environment, so it gets added to every node. We are planning to store the automatically-generated fsid into node attributes, similar to the mon and osd secrets, but haven't gotten to it yet. Sorry I forgot about that part :( The instructions should've been: after setting up the first ceph-mon node, save the fsid into the environment. You don't need to redeploy anything, just add the fsid to the environment and maybe clean out any saved node attributes about the fsid. |
|
I was just following along with the guide on wiki.ceph.com, part of which includes setting a cluster id (fsid) in the environment, which I did. Is that what you're talking about? That value only shows up in {bull,zod,superman}'s ceph.conf file, not in the OSD-only servers. It's the same as the value that shows up running ceph -s on any of the nodes, and it's the value I put in the Ceph environment as the fsid. None of the nodes have any other fsid field defined that I can see anywhere, not even the ones where the OSDs came up correctly. So if I understand what you're saying, it's already correct and there are no discrepancies to resolve or remove. But I don't see it when I |
|
I'm so sorry about that! I think this came from PR #114, which was working towards making it so that you can use the ceph cookbook on nodes that aren't in the ceph cluster, to create clients and so on. The initial commit I created did a check in the ceph.conf template to not add the fsid option if it's not a monitor, because I thought that only the monitor would need it. Later I discovered that OSDs do need it, so I amended the PR with a commit to fix that. However, we mildly refactored the cookbook at that same time, and that commit was lost. To fix it, look at the last file in hufman@8225ca9 and apply that diff to your ceph.conf.erb template. I'll work on fixing this in the main repo! |
|
Ah ha! That did it! Everything came up nicely after that. Woohoo! Well, it was a whole chain of things and we never did figure out one of them, but assuming you're pushing this fix back up then I think we can call it good for now. Thanks for all your help! |
|
I believe this has all been resolved and/or moved to other issues:
|
I'm new to ruby, chef and the cookbook so it's possible my analysis is flawed, and I'm unable to suggest a patch, but please bear with me.
I'm having trouble deploying a configuration which lists multiple public networks. The problem seems to be
find_node_ip_in_networkwhich does not check for this possibility and instead passes the full string directly toIPAddr.new, which fails.To be more specific, in my environment I set a default attribute:
"public network": "192.168.10.0/24,192.168.20.0/24"e.g. listing two subnets. Ceph supports this as described here: http://ceph.com/docs/master/rados/configuration/network-config-ref/#network-config-settings
In any case it doesn't look like either
find_node_ip_in_networkor its callermon_addressesmake any any effort to handle multiple networks. The result is that whenchef-clientruns, it dies with a backtrace:The text was updated successfully, but these errors were encountered: