-
Notifications
You must be signed in to change notification settings - Fork 779
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Vsphere support #504
base: master
Are you sure you want to change the base?
Vsphere support #504
Conversation
…i and vCenter systems
hey @jasonberanek - thanks for all the work - I guess before merging I'd need some beta testers - From the @fog repo I've learned it's difficult to keep up with systems you don't own - would you be willing to follow this up for any changes? |
@jedi4ever I would be happy to. I'm going to talk to a few individuals to beta test as well since I have limited access to specific vSphere version. As I have the time, I'll also try to keep up with master changes to make an eventual merge smoother. For those trying to beta test, please look over the docs/vsphere.md file for instructions on using the vsphere provider. |
would be cool to actually turn esx install in to a veewee template - http://www.jedi.be/blog/2010/12/09/automated-vmware-esx-installation-even-in-vmware-fusion/ - that way we can generate a test vsphere anytime :) |
If I can get my hands on a fusion license, I might just work on that next. |
you can get a free fusion license - and this how you automate the installation 💃 |
@@ -14,7 +14,7 @@ end | |||
|
|||
group :test do | |||
gem "rake" | |||
gem "em-winrm", :git => 'git://github.com/hh/em-winrm.git', :ref => '31745601d3' | |||
gem "em-winrm", :git => 'http://github.com/hh/em-winrm.git', :ref => '31745601d3' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes to em-winrm probably should not go into this pull request
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't disagree, though I do have to maintain this as a local change because git:// does not work within my corporate network. The change above wasn't actually meant to be included in the commits, just haven't had a change to get it changed back.
Reverted Gemspec entry for em-winrm to use git:// instead of http:// Added a vsphere group to the Gemspec to hold vsphere specific dependencies, per reviewer recommendations. Removed vsphere specific gems (i.e., rbvmomi) from veewee.gemspec Fixed two errors in CentOS vSphere template: ks.cfg referenced a local OS location from original testing, and a postinstall2.sh file was committed in error. # Please enter the commit message for your changes. Lines starting # with '#' will be ignored, and an empty message aborts the commit. # On branch vsphere-support # Changes to be committed: # (use "git reset HEAD <file>..." to unstage) # # modified: Gemfile # modified: lib/veewee/provider/vsphere/box/create.rb # modified: templates/CentOS-5.8-x86_64-netboot-vsphere/ks.cfg # deleted: templates/CentOS-5.8-x86_64-netboot-vsphere/postinstall2.sh # modified: veewee.gemspec # # Untracked files: # (use "git add <file>..." to include in what will be committed) # # experiment/
Current code will not work with ESXi 5.1 (and probably 5.0), as this version's firewall disables VNC connections by default. See #508 comments for relevant links about changing the firewall settings. I'll get the documentation updated once I have a repeatable process to complete the firewall configuration. |
I've posted a gist with the instructions on how to configure ESXi 5.1 firewall support the internal ESXi VNC server, and by extension the vSphere provider, here: https://gist.github.com/4670943 I plan to update the ESXi 5.1 template for VMware Fusion to include this as a post install script by default, however I am waiting for a pre-requisite to be merged under #513. |
#513 was merged - so go go for this update 👍 |
end | ||
|
||
def retrieve_datastore options | ||
dc = vim.serviceInstance.find_datacenter |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This assumes a single datacenter object, which is fine when connecting directly to an ESXi host, but breaks if you're going through a vCenter that has more than one datacenter defined.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The above code actually picks the first datacenter returned by the vSphere APIs (per the rbvmomi spec), rather than assume there is only one. That said, this behavior is not a sane default in a multiple datacenter environment as it who knows how random that selection might be.
I assume you were seeing behavior where you didn't know which datacenter was chosen, or did you see an actual error? If it was an actual error, then that gives me concerns that even when the datacenter is selected, things may still fail for some other reason.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this case it was just choosing dc1, when the host in question was in dc3. This may have actually been the crux of my problem when using vCenter, as the default datastore chosen was actually the first entry in dc1 (which incidentally was also a folder and not a datastore). It may also have accounted for why it couldn't locate the network I specified, and may not be a problem with distributed vswitches as I had thought before...some more investigation is needed.
--jc
Sent from my iPhone
On Jan 30, 2013, at 6:24 PM, "Jason A. Beranek" [email protected] wrote:
In lib/veewee/provider/vsphere/box/create.rb:
datastore = dc.find_datastore datastore_name
# These checks need to be done in this order
# otherwise the requests to the datastore over
# http will hang. (Observed in ESXi 4.1 update 3)
unless datastore.exists? "isos/"
vim.serviceContent.fileManager.MakeDirectory :name => "[#{datastore_name}] isos", :datacenter => dc
end
unless datastore.exists? "isos/"+filename
env.ui.info "Loading ISO to vSphere Host"
datastore.upload "isos/"+filename, local_path
end
end
def retrieve_datastore options
The above code actually picks the first datacenter returned by the vSphere APIs (per the rbvmomi spec), rather than assume there is only one. That said, this behavior is not a sane default in a multiple datacenter environment as it who knows how random that selection might be.dc = vim.serviceInstance.find_datacenter
I assume you were seeing behavior where you didn't know which datacenter was chosen, or did you see an actual error? If it was an actual error, then that gives me concerns that even when the datacenter is selected, things may still fail for some other reason.
—
Reply to this email directly or view it on GitHub.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I see where that is going wrong. The logic to retrieve the default datastore does validate it actually has a datastore object, that's why its retrieving a folder instead. It would be better if we could use the datastore collection on that object, but that collection is not guaranteed to be available according to the vSphere API docs for the Datacenter ManagedObject).
I'll need to fix that, or better yet fix the other behaviors around working in a multi-datacenter, multi-host environment in order for you to get anything but the behavior you are seeing. That is going to need some thought.
While I'm still working through some other issues when connecting directly to a specific ESXi host in my infrastructure and explicitly passing a network and datastore, I can say currently that some of these defaults and assumptions won't work for those more invested in vCenter like me. Specifically the ones I've already run into that apply to our environment include:
These aren't insurmountable problems to solve, and I actually have some of this code in various forms that I'd be happy to contribute, it would be good to firm up the scope of architectures this should support. If sticking to ESXi only for now makes the most sense, we should probably make that more explicit in the docs. I should mention I'm on vCenter/ESXi 5.0 U1. |
@justinclayton Thanks this in your environment. The vCenter 4.1 environment I tested against is a one host, single datacenter configuration. Your findings are timely as they confirm there are additional use cases, though the use cases are much harder to test. Do you have a good resource for the last three items on your list, and how they relate to problems you are seeing? I understand the multiple datacenters issue, and can probably guess at the problems with the DRS and datastore clusters, but I'm not exactly sure what you mean by datastore folders and distributed vswitches being a problem. As far as addressing those issues, I'd appreciate any code you have to do the appropriate checks or help configure the system as appropriate. My initial thought on how to handle this in the code is to provide the default case with some assumptions (maybe removing the default network and datastore components as being too risky), and if those assumptions are violated and a user hasn't provided an input (e.g., there is more than one datacenter and the user hasn't specified one), raise an error with useful instructions on how to proceed. That way, we can support simpler lab type cases, and also support more complex vCenter configurations. The above said, I don't know how much of that needs to be achieved between now and the initial push of the provider to master, though I'd like to do best effort to at least have the error detection logic in place so a build doesn't blow up in unique and interesting ways. |
I'm currently waiting for our administrators to give me advanced access to our reasonably large vsphere testbed. Then I can also help testing this pull request in detail. I would prefer not to merge it until some reasonable testing has been done by a few people. As soon as this is merged in, people will notice veewee to have vsphere capabilities and expect it to work. Everyone who is adventurous can do the extra work to checkout the code from your repo. |
@ffeldhaus true - but then again it's a way to get faster feedback :D - but agreed let's give it another week to see how things go |
@jedi4ever @ffeldhaus: Based on the input I'm receiving from @justinclayton, I think the support for vCenter environments is unstable for anything but a trivial single node environment (which incidentally was my test environment). Some of the items that I was making opinionated choices likely need to be moved into vm_options to allow for appropriate support, including:
In addition to the above, there are some more validation steps that would be good to conduct early in the execution process. I will try to work out these changes and push them tomorrow, but I need to get a fresh set of eyes on them after sleep. The vSphere provider's ESXi support should be stable after the datastore and network concerns are handled. If the vCenter support appears to need more thought and coordination, then one option would be to caveat the experimental nature of supporting a vCenter deployment and work out the details with future changes as everyone deems appropriate. |
@jasonberanek - I think this is indeed the right approach: validate the assumptions and raise an error / or potentially handle them if we can. That way we can merge it in for people to use it with the default assumptions. And we can state that this provider is experimental. |
@jasonberanek I'm happy to take on some or all of these changes; since the project I was tasked with involved building something like veewee, I'll just shift my efforts into helping get this working. Not sure which of these you've already started on, so it's your call. --jc On Jan 30, 2013, at 8:36 PM, "Jason A. Beranek" [email protected] wrote:
|
@justinclayton I'd appreciate any help you can provide, particularly around On Thu, Jan 31, 2013 at 11:27 AM, Justin Clayton
|
@jasonberanek Great, I should have something to show in a few. By chance, is the datacenter on your small vCenter instance called "ha-datacenter"? I found some names listed at rlane/rbvmomi that might indicate that there are sane defaults that can be assumed in small instances. |
How important is the standalone ESXi host use case vs the vCenter use case? Since the options for datacenters and clusters don't apply to a standalone connection, we'll need more logic around which one is being executed. |
@justinclayton: The ESXi use case is important, as it was the default use case I built for. It allows VMware VMs to be built within an automated build environment without requiring a licensed product. This is ideal if you want to do that build in an isolated location to avoid making your production vCenter credentials available on the machine running the command. The beauty of the vSphere Web Services API is that it works with both stand alone ESXi and vCenter configurations, and the differences are largely in the data returned by the API. Let me address your questions:
I'll push some changes tonight that should demonstrate what I'm talking about. And enhance the comments to make it clearer what the datacenter and host access code is doing. |
changes to support a wider range of vCenter deployments
I haven't been able to get the VNC stuff to work for me. I'm getting as far as the centos 5 iso boot and then I get a VNC timeout error. I'm not familiar with the VNC capabilities of vSphere; is there a client-side requirement I'm missing? |
@justinclayton For vSphere 5.x hosts, you need to enable VNC within the host's firewall. The following gist has the instructions for 5.1: https://gist.github.com/4670943, and I believe the commands are similar for 5.0. I mentioned this earlier in the pull request, but I haven't had the chance to add to the documentation yet. If enabled, you may still have another firewall blocking the VNC port range. |
…figuration, update docs Renamed "host" configuration to "compute_resource" to allign with vSphere documentation Added support for retrieving the compute_resource that should support cluster configurations Added support for validating datacenter and compute resource selection, providing sane error messages (with recovery instructions if available)
First cut at more robust error handling and sane messages started for datacenter and compute_resource. Will take a similar approach to network and datastores to ensure systems don't enter an unstable state. |
After some more digging, I'm concerned about using the VNC method to communicate with a VM during building. While it seems to still be included in even the most recent versions of ESXi, it hasn't been an officially supported feature since the GSX days. It also may not work at all when using vSphere Auto Deploy. I'm assuming (perhaps incorrectly, given that I'm pretty new to veewee) that the VNC bits are just used during the build step to send keystrokes to the booted install media. If this is true, perhaps we could consider a baked floppy image approach instead, like the Windows veewee templates use? I don't know how Ubuntu preseed stuff works, but using a floppy would definitely work for CentOS (or any distro that uses kickstart/anaconda). I'd be willing to work on this as well if others agree this is a good way to go. Alternatively, if I'm way off on my ESX+VNC rantings I would love to be set straight. :-) |
@justinclayton I can't speak specifically to the supported level of VNC in vSphere going forward, so your findings may be right. The ESXi 5.x firewall settings not even having a built in rule for it would indicate its support may be problematic at best. I haven't played around with the floppy functions at all in veewee at this point, but it should be possible to support either floppy or VNC based boots, varying the boot type depending on if someone enters in a boot sequence to type. I'd probably prefer that route pending knowledge that VNC is completely dead, just because I'd worry about the variances of supporting floppy installs for multiple OSes. |
@jasonberanek I agree with the idea of allowing both, although considering that the Windows veewee templates use floppy images today, it sounds like we're already in the business of supporting this variance across different OSes. On Feb 1, 2013, at 1:50 PM, Jason A. Beranek [email protected] wrote:
|
@jedi4ever @ffeldhaus: When either of you has a chance, I'd appreciate your feedback on the discussion above regarding floppy vs. VNC to initiate the boot process. I will note that based on my work trying to get the ESXi 5.1 template in a shape that it can be used immediately for testing the vSphere provider that adding VNC to the ESXi 5.x line is not takes some work. That level of work is probably ok if you are setting up ESXi as a stand alone box to build base VMs, but may not be reasonable nor wise from a security perspective if you were running against a production setup. |
to clarify - the floppy support in windows doesn't use it to boot the system - it merely contains a few files that are picked up by the iso/dvd windows boot. This allows us to just add textfiles (floppy files) to definition vs. creating binary boot floppies for each distro. Actually I don't think for the vsphere support this will help as you'd still need to have a vnc typing to find the ks.cfg from the floppy. The iso doesn't pick up the floppy disk by default (as least in generic Centos not). I've used the vnc enabling support in older versions of esxi fine, the postinstall might be different. My focus would go to - make 1 thing work -have it documented - and add checks to see if it is support. Therefore I'd rather not wait to merge the provider until all issues are sorted out for all scenarios. Let's make the default work - clearly state what the limitations are. |
Sounds good. In the interest of making one thing that works, my plan will We can look at alternative approache for setting up the boot sequence after
|
…figuration Separated out datastore and network selection logic into separate files from create.rb Added support for validating datastore and network selection, providing sane error messages (with recovery instructions if available). Also includes validation that the compute resource can access the datastore and network desired. Added error handling on host ip configuration to inform users of available IP addresses for the machine running veewee
@jedi4ever thanks for the clarification on the windows floppy use. My concept for linux is a bit different: bake a floppy image containing the ks.cfg as well as a boot loader with the path to the ks.cfg embedded. This would remove the need to live type characters to the VM, but would still allow for the cdrom iso to be used as the install media. I've used this approach for many bare metal installations using either floppies or usb flash drives, and it's also similar to the approach you would take in a PXE-based install environment. While the decision to go ahead with VNC will unfortunately preclude me from being able to test this further in my environment, I do understand that I am not the majority of cases, and that a stake has to be put in the ground somewhere. If I ever find a solution to use VNC with Auto Deploy I will certainly be happy to contribute documentation to the project. In the meantime I'd like to continue developing alternative boot methods to be considered in a future merge. |
@jedi4ever @ffeldhaus: With the last set of commits, I think many of the error cases (including VNC support) are covered to avoid a build proceeding in an unstable state. In case of VNC support, I don't know of a way to checking support until after the VM is defined, so that particular error handling will clean up after a failure. At this point, I've done repeat testing on both ESXi 5.0 update 2 and ESXi 5.1. I need to retest a bit on ESXi 4.0/vCenter 4.1, but I don't foresee any changes causes issues at this time. I'll also relook at the documentation this week, to see if there is anything I missed that needs updating. |
@jasonberanek @jedi4ever is this feature still something that is actively being pursued? If so, I would be happy to help move us forward. I've gotten past my VNC hangups. :-) |
@justinclayton I have been busy working other items, so I haven't had much of a chance to develop this further as it hasn't been mainline with my work. More than happy to accept pull requests if @jedi4ever is still interested. At this point, my branch is painfully behind master, so that will have to be reconciled. Also, #559 theorhetically needs to be address to avoid silent failures if VNC doesn't work as expected. |
I have access to a modest vsphere envrionment, is there anything I can do to help testing? |
I'm looking for a solution to build base images on a vCenter environment via the vSphere API. Packer currently does it via SSH which I don't have access to, so this is no option to me. What is the status of vSphere API support via veewee? |
Sadly at this point, I consider this support largely a dead PR. I haven't had the heart to close it, but its woefully behind on any mainline veewee changes that have occurred since 25 Jan. Note, the vCenter support did place requirements that the firewall on the ESXi hosts had to be configured to support VNC, which I would imagine if you don't have an option of ssh you may not have an option for reconfiguring the ESXi host to allow VNC. My recommendation would be to look at an option of creating a Linux VM within your vCenter environment that hosts an instance of VMware Player or Workstation, and using Packer to build a base imagine within that VM, and use the vsphere post processor to then upload it to your vCenter environment. Its a bit involved, but probably the cleanest way to automate the process at present. |
The company I work for has all the possible VMware server licenses, but no Workstation or Fusion unfortunately. 👎 |
@jasonberanek a lot of work you have put in here, please let me know if you would like to finish it? |
Initial support for vSphere provider in veewee (see #399), includes the following commands:
Tested on ESXi 4.0 and vCenter 4.1