Bare Metal Support

[tpletcher-hpe]

Hi Folks,

I'm wondering if there would be interest in formalizing the bare metal objective with a project tracking the effort. I'm having a little bit of a challenge tracking when this support might land with the closing of the meta data issue, etc.

Thoughts?

[samuelkarp]

Hi @tpletcher-hpe!

Thanks for reaching out! We’d love your input in both formalizing the project for bare metal (outside EC2) support and in doing the work to get there. We’ve already started work on our first non-EC2 target: VMware (with support for EKS Anywhere). While our own current priorities mean that we’re going to stay focused on VMware first, we do expect that much of the work for VMware will be reusable in a larger non-EC2 scope.

Here are some of the things we’ve been working on already:

• Kernel support for VMware (in cooperation with Amazon Linux’s kernel team)
• Decoupling from a dependency on EC2’s instance metadata
  • Allow admin container to use SSH public keys from user data (Add "shibaken" generator to populate user-data with public keys from IMDS #1331, shibaken: return empty key list if IMDS returns 404 #1358, Add a readme #19)
  • Add support for kubelet in standalone mode and TLS auth (add support for kubelet in standalone mode and TLS auth #1338)
  • Add support for pulling host-containers from ECR Public (host-ctr: add support for authenticated image pulls from ECR Public #1296)
  • Add support for reading settings from /media/cdrom
• Add the ability to output vmdk via qemu-img (Add the ability to output vmdk via qemu-img #1289)
• Add a vmware-dev variant (Add a vmware-dev variant #1292, Add conditional compilation to early-boot-config #1288, Add a systemd mount unit for CD-ROM devices #1290)

We still have a few other things to figure out and put together, like:

• An alternative “control” container that can function without Amazon SSM
• Finalizing the kernel config for VMware
• Better mechanisms for testing and continuous integration
• Integration into our publishing pipeline

For support on other non-EC2 targets, we think of the work as falling into a few different categories:

• Basic Linux requirements (kernel support & config, partition layouts, installation/imaging mechanisms)
• Bootstrapping Bottlerocket’s own components (the settings system, host containers, publishing tools, updates repository)
• Orchestrator integration (EKS Anywhere, generic Kubernetes, or something else)

And there’s some existing discussion of non-EC2 target environments in #1097.

With all of that said: Our focus today is going to remain on our existing priorities including VMware support for EKS Anywhere. While we can’t really commit to doing all of the bare-metal work yet, if you’re interested in contributing and helping to drive the work for other non-EC2 targets we’d love to have an open line of discussion and work together on a design and approach that could eventually be a “supported” variant of Bottlerocket.

Thanks!
Sam

[zicklag]

Hey folks!

I'm currently investigating bottlerocket as an OS to run my containers, but I don't need an orchestrator. I just need to be able to run Portainer and I will manage my containers through the Portainer web UI ( or my own self-developed agent ).

I believe that I could figure out how to set that up, but I also want to run bottlerocket on my own servers. Not bare metal necessarily, more likely through a cloud VM provider, but at the same time I probably have very similar requirements to bare metal.

So far I've succeeded in building bottlerocket on my machine. I love the build process by the way! So rusty. 😉 Now I want to figure out if I can create an image that I can either boot with firecracker ( so I think I need the vmlinux image and a root filesystem image ) or virtualbox ( not sure what I need for virtualbox, vmdk? ).

I'm not very experienced with some of the finer details of how linux operating systems are built, but I'm a quick learner that is expert-level with Rust and very intrigued with bottlerocket, so I'd definitely be open to contributing as my time allows, in order to get my use-case working.

I'd welcome any guidance on how I might get started running a bottlerocket VM with either firecracker or Virtualbox. I can see that you've gotten a lot of the blockers for non EC2 environments mentioned above working, which is good. Hopefully we're not far from being able to make it work!

Thanks!

[zicklag]

Just to share my experience so far, I've gotten a rough idea of what the build process is actually doing now, and I found out how to get a vmdk out of any variant build by putting image-format = "vmdk" in the Cargo.toml, but if I build either vmware-dev or aws-dev variants, and I try to boot the VMDK in Virtualbox the vm ends up having eternally on:

Waiting for root device /dev/dm-0...

Could use a hint on what that means if anybody knows. :)

[zmrow]

Hi @zicklag out of curiosity - did attach the *.vmdk as the boot disk and the *-data.vmdk as the second disk? I know if you don't attach both you will mostly likely see that error, with some clearer error lines above it. Also, you should use an NVME controller for the disks.

It's also important to note that even if you're able to start booting the OS in Virtualbox, it will fail to retrieve user data from anywhere and hence, fail to fully boot. In VMware, we use the "guestinfo" interface to provide user data; I'm not sure if something similar exists for Virtualbox or not.

As you mentioned above, a Virtualbox or Firecracker variant would be closer to a bare metal line of variants, which we haven't started work on yet today.

[zicklag]

did attach the *.vmdk as the boot disk and the *-data.vmdk as the second disk?

Ah, I didn't do that. I'll try that. Thanks for the tip!

It's also important to note that even if you're able to start booting the OS in Virtualbox, it will fail to retrieve user data from anywhere and hence, fail to fully boot. In VMware, we use the "guestinfo" interface to provide user data; I'm not sure if something similar exists for Virtualbox or not.

That's a good point. In the case of booting bare metal, there would be no such feature, and I don't know that virtualbox has anything similar to "guestinfo" either.

It seems like for these use-cases we might need another way of seeding the user data. Maybe through a REST API endpoint hosted during bootstrapping? The issue would be that you would want to authenticate to the endpoint so that not just anybody could initialize the VM if they got to it before you did. The only way I can think to authenticate, though, would be if you were required to compile your own version of bottlerocket with a trusted public key baked into the image. And only the holder of the private key could initialize the user data. That seems over-complicated, though.

Maybe another way would be by looking for an attached drive with the user data in a file on that drive. 🤔 Then you would just have to format a drive or partition maybe with the user data on it.

I'm trying to get a better understanding of how all of this is put together so it's all a little fuzzy to me right now, but I'll be experimenting and seeing what I can figure out.