External/SSH internal driver for all OS #2000

afbjorklund · 2023-11-15T22:09:48Z

Similar to the docker-machine "generic" driver, bring your own virtual machine (or physical server)

Not so useful in itself, but not so bad when made into a real driver or wrapped with helper scripts...

NAME          STATUS     SSH                   VMTYPE    ARCH      CPUS    MEMORY    DISK      DIR
beaglebone    Running    192.168.7.2:22        ext       armv7l    1       512MiB    4GiB      ~/.lima/beaglebone
core          Stopped    127.0.0.1:0           qemu      x86_64    1       1GiB      100GiB    ~/.lima/core

vmType: ext

arch: "armv7l"
cpus: 1
memory: 512MiB
disk: 4GiB

# We do not have arm-v7 binaries of containerd
containerd:
  system: false
  user: false

ssh:
  address: 192.168.7.2

Requires:

Restore guest agent unix socket functionality #2006

Installed lima-guestagent, and nerdctl from tarballs/binaries.

$ _output/bin/limactl shell beaglebone nerdctl version
Client:
 Version:	v1.7.0
 OS/Arch:	linux/arm
 Git commit:	e674fe7ba6e49f12e88cd9c6c442e7ea5232502c
 buildctl:
  Version:	v0.12.3
  GitCommit:	438f47256f0decd64cc96084e22d3357da494c27

Server:
 containerd:
  Version:	v1.7.6
  GitCommit:	091922f03c2762540fd057fba91260237ff86acb
 runc:
  Version:	1.1.9
  GitCommit:	v1.1.9-0-gccaecfc

limactl guest-install beaglebone

Add hidden command to install the guest components #2028

Hardware

https://www.beagleboard.org/boards/beaglebone-black

You could also use a Raspberry Pi Zero*, or a cloud droplet.

* need the Zero 2, for arm-v7 (previous model was arm-v6)

https://www.raspberrypi.com/products/raspberry-pi-zero-2-w/

Discussion

The Linux physical machines #2029

afbjorklund · 2023-11-18T12:22:23Z

The probes are somewhat annoying when not using cidata, but that was the same story on FreeBSD and others.

Support for FreeBSD guests ? #1508 (comment)

Maybe there should be some fallback implementation, at least for ssh-ready/guestagent install/boot-done ?

sudo diff -q /run/lima-ssh-ready /mnt/lima-cidata/meta-data

install -m 755 /mnt/lima-cidata/lima-guestagent /usr/local/bin/lima-guestagent
sudo /usr/local/bin/lima-guestagent install-systemd

sudo diff -q /run/lima-boot-done /mnt/lima-cidata/meta-data

Like copying the lima-guestagent over the ssh connection (my workaround)

And keeping the instance meta-data (id) somewhere else, like in /etc ?

afbjorklund · 2023-11-22T19:38:33Z

Added support for hostnames, so that you can use nice features like avahi-daemon

NAME           STATUS     SSH                     VMTYPE    ARCH       CPUS    MEMORY    DISK      DIR
raspberrypi    Running    raspberrypi.local:22    ext       aarch64    4       512MiB    32GiB     ~/.lima/raspberrypi

anders@raspberrypi:~ $ sudo mkdir /mnt/lima-cidata
anders@raspberrypi:~ $ sudo touch /mnt/lima-cidata/meta-data
anders@raspberrypi:~ $ sudo touch /run/lima-ssh-ready
anders@raspberrypi:~ $ sudo touch /run/lima-boot-done

Previously I was assuming IP. Now, to have a nice way to install the lima-guestagent

afbjorklund · 2023-11-29T10:52:47Z

Added host key checking:

The authenticity of host 'raspberrypi.local (192.168.0.113)' can't be established.
ECDSA key fingerprint is SHA256:tKIRfOeWP1HeCFLpM0UT30CUWSDXpC7gxPsKHUnS+h4.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

And shortened the timeout, for when the device is not connected. Resolves in 10ms, when cached (or ~250ms).

errors="[field `ssh.address` must be IP: lookup raspberrypi.local: i/o timeout]"

afbjorklund · 2024-01-07T21:40:20Z

Added provision scripts.

Using sudo for system.

pkg/sshutil/sshutil.go

AkihiroSuda · 2024-01-22T14:17:44Z

pkg/limayaml/limayaml.go

 	QEMU VMType = "qemu"
 	VZ   VMType = "vz"
 	WSL2 VMType = "wsl2"
+	EXT  VMType = "ext"


Maybe this should be called "external" to avoid confusion with "extended"

In minikube we ended up calling the driver "ssh", but I don't think it's great

ping @lima-vm/maintainers @lima-vm/reviewers RFC

I still feel "ext" is confusing.
Sounds like some sort of ext2/ext3/ext4 stuff.

I think external is fine. unmanaged or raw could be alternatives, but I prefer external.

ping @afbjorklund WDYT?

AkihiroSuda · 2024-01-22T14:19:38Z

examples/default.yaml

 ssh:
+  # Address for the host.
+  # 🟢 Builtin default: "127.0.0.1" (localhost)
+  address: null


What does localPort mean now for non-local address?

It used as "port", but the old name is kept for compatibility.

If the port is zero, then for non-localhost it will be set to 22.
For localhost (127.0.0.1), it is still assigned to a random port.

Normally it is not set, and only the IP address is used for SSH.

afbjorklund · 2024-03-18T18:20:34Z

Rebased to lima v0.21.0 [RPi runs Debian GNU/Linux 12 (bookworm)]

$ _output/bin/limactl shell raspberrypi nerdctl version
Client:
 Version:	v1.7.5
 OS/Arch:	linux/arm64
 Git commit:	cffed372371dcbea3dc9a646ce5a913fc1c09513
 buildctl:
  Version:	v0.12.5
  GitCommit:	bac3f2b673f3f9d33e79046008e7a38e856b3dc6

Server:
 containerd:
  Version:	v1.7.14
  GitCommit:	dcf2847247e18caba8dce86522029642f60fe96b
 runc:
  Version:	1.1.12
  GitCommit:	v1.1.12-0-g51d5e946

Quite svelte, without the cidata etc: 44K /home/anders/.lima/raspberrypi

NAME           STATUS     SSH                     VMTYPE    ARCH       CPUS    MEMORY    DISK     DIR
raspberrypi    Running    raspberrypi.local:22    ext       aarch64    4       512MiB    32GiB    ~/.lima/raspberrypi

AkihiroSuda · 2024-04-22T00:40:59Z

Would it be possible to test this on CI?

afbjorklund · 2024-04-22T05:56:48Z

Would it be possible to test this on CI?

As long as it is possible to supply a VM, with access through host keys and authorized keys, that should be possible.

I should detail the required steps (with example log), especially now with the addition of the cloud-config generation.

Generate cloud-config outside of cidata.iso too #2271

afbjorklund · 2025-09-06T12:13:58Z

Rebased to v2.0.0-alpha.0, only thing left to do is to find a driver name that is acceptable.

afbjorklund · 2025-09-06T12:32:20Z

These patches should be converted to driver features:

warn about experimental driver
skip creating the "basedisk" file
skip creating the "cidata.iso" file
run provisioning scripts over ssh
skip running certain requirements
allow instance without "images"
validate ssh.address (and required)
allow PID status check for HA only

AkihiroSuda · 2025-09-22T16:49:10Z

website/content/en/docs/config/vmtype/ext.md

+> **Warning**
+> "ext" mode is experimental
+
+"ext" option makes use of an external machine, either a virtual machine or a physical machine.


The idea of "external machines" can be easily confused with "external drivers", so this driver should not be called "ext" ("external").

Let's call this either "generic" or "ssh"

AkihiroSuda · 2025-09-22T16:49:39Z

templates/experimental/ext.yaml

+  writable: true
+
+ssh:
+  address: raspberrypi.local


If you want to put raspberrypi-specific config here, the file name should be probably "raspberrypi.yaml"

AkihiroSuda · 2025-09-22T16:50:03Z

cmd/limactl/main_ext.go

+package main
+
+// Import ext driver to register it in the registry on all platforms.
+import _ "github.com/lima-vm/lima/v2/pkg/driver/ext"


Should this be compiled as an external driver? (w/ docs)

Currently that is not possible, but might be an option (like the other built-in ones)

Why not possible?

AkihiroSuda · 2025-09-22T16:50:42Z

pkg/driver/ext/ext_driver.go

+
+	if _, ok := mountTypesUnsupported[*cfg.MountType]; ok {
+		return fmt.Errorf("mount type %q is explicitly unsupported", *cfg.MountType)
+	}


Any mount type other than REVSSHFS should result in an error

AkihiroSuda · 2025-09-22T16:51:26Z

cmd/limactl/main_ext.go

Can we have a smoke test ?
Maybe using an LXD/Incus instance on GHA

I tried that, but it didn't work out... Help needed

#2000 (comment)

What was the error?

https://github.com/canonical/setup-lxd may work

I guess there is an equivalent of this for Incus too

afbjorklund · 2025-09-22T20:09:02Z

I think we will throw this one out of 2.0 as well (like with 1.0)

Verify ssh host keys, when connecting to a remote server. The first connection will prompt, if not in known_hosts. Print IP instead of Port (when it is 22) for non-local SSH. The field is still called LocalPort, even if is now remote. Signed-off-by: Anders F Björklund <[email protected]>

Don't show log for already cached archive, don't generate cidata.iso for external vm. It is not using cloud-init anyway, and does not need another copy of lima-guestagent and nerdctl-full.tgz Signed-off-by: Anders F Björklund <[email protected]>

Add a shorter timeout for mDNS IP lookup Allow .local hosts to be offline or slow Signed-off-by: Anders F Björklund <[email protected]>

The provision scripts are run with ssh, instead of being started by cloud-init like other drivers. The ExecuteScript was duplicated from sshocker, since it needs support for running with "sudo". Signed-off-by: Anders F Björklund <[email protected]>

afbjorklund force-pushed the external branch from 64ec4ff to b865895 Compare November 15, 2023 22:11

afbjorklund mentioned this pull request Nov 17, 2023

Support for native driver specific guest agent connection #1998

Merged

4 tasks

afbjorklund force-pushed the external branch 2 times, most recently from ea5255d to 9cea961 Compare November 24, 2023 07:33

afbjorklund mentioned this pull request Nov 24, 2023

Add hidden command to install the guest components #2028

Merged

afbjorklund force-pushed the external branch from 9cea961 to dcdb6f8 Compare November 24, 2023 07:41

afbjorklund mentioned this pull request Nov 25, 2023

Libvirt driver, for hypervisors on remote machines #2031

Closed

afbjorklund force-pushed the external branch from dcdb6f8 to 5b84678 Compare November 29, 2023 10:51

afbjorklund force-pushed the external branch 2 times, most recently from 0ab8193 to b91338b Compare January 2, 2024 07:32

afbjorklund mentioned this pull request Jan 2, 2024

External driver, for cloud and physical #2109

Open

afbjorklund force-pushed the external branch from 3998d45 to 5173ce3 Compare January 7, 2024 21:39

afbjorklund force-pushed the external branch from c2abd42 to 7b2e56d Compare January 19, 2024 16:39

afbjorklund marked this pull request as ready for review January 21, 2024 18:59

AkihiroSuda reviewed Jan 22, 2024

View reviewed changes

pkg/sshutil/sshutil.go Outdated Show resolved Hide resolved

AkihiroSuda reviewed Jan 22, 2024

View reviewed changes

afbjorklund force-pushed the external branch from 7b2e56d to a18e091 Compare February 24, 2024 08:38

afbjorklund force-pushed the external branch from a18e091 to 9b8c9ea Compare March 18, 2024 18:10

AkihiroSuda added this to the v0.21.1 milestone Apr 22, 2024

afbjorklund removed this from the v0.22.0 milestone Apr 26, 2024

afbjorklund marked this pull request as draft April 26, 2024 06:01

afbjorklund force-pushed the external branch from 9b8c9ea to e11848a Compare April 28, 2024 12:41

afbjorklund force-pushed the external branch 7 times, most recently from 4a9f4a3 to 1a79286 Compare August 26, 2025 20:17

afbjorklund marked this pull request as ready for review August 27, 2025 05:33

afbjorklund force-pushed the external branch from 1a79286 to a30bbb9 Compare August 27, 2025 20:04

AkihiroSuda added the area/vmdrivers VM driver infrastructure label Sep 2, 2025

afbjorklund force-pushed the external branch from a30bbb9 to d16ab68 Compare September 6, 2025 12:11

afbjorklund force-pushed the external branch from d16ab68 to 77ff3e6 Compare September 6, 2025 12:16

afbjorklund force-pushed the external branch from 77ff3e6 to 4e73d5a Compare September 19, 2025 05:35

AkihiroSuda added this to the v2.0.0 milestone Sep 22, 2025

AkihiroSuda reviewed Sep 22, 2025

View reviewed changes

AkihiroSuda modified the milestones: v2.0.0, v2.1.0 (?) Sep 23, 2025

afbjorklund marked this pull request as draft September 23, 2025 05:44

afbjorklund added 4 commits September 23, 2025 17:52

Add ext driver for external VM machines

9573bbe

Don't show log for already cached archive, don't generate cidata.iso for external vm. It is not using cloud-init anyway, and does not need another copy of lima-guestagent and nerdctl-full.tgz Signed-off-by: Anders F Björklund <[email protected]>

Allow using address like raspberrypi.local

80a1e4e

Add a shorter timeout for mDNS IP lookup Allow .local hosts to be offline or slow Signed-off-by: Anders F Björklund <[email protected]>

afbjorklund force-pushed the external branch from 4e73d5a to 61a5132 Compare September 23, 2025 15:54

External/SSH internal driver for all OS #2000

Are you sure you want to change the base?

External/SSH internal driver for all OS #2000

Conversation

afbjorklund commented Nov 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

afbjorklund commented Nov 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

afbjorklund commented Nov 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

afbjorklund commented Nov 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

afbjorklund commented Jan 7, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

afbjorklund commented Mar 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AkihiroSuda commented Apr 22, 2024

Uh oh!

afbjorklund commented Apr 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

afbjorklund commented Sep 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

afbjorklund commented Sep 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

afbjorklund Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

afbjorklund commented Sep 22, 2025

Uh oh!

Uh oh!

afbjorklund commented Nov 15, 2023 •

edited

Loading

afbjorklund commented Nov 18, 2023 •

edited

Loading

afbjorklund commented Nov 22, 2023 •

edited

Loading

afbjorklund commented Nov 29, 2023 •

edited

Loading

afbjorklund commented Mar 18, 2024 •

edited

Loading

afbjorklund commented Apr 22, 2024 •

edited

Loading

afbjorklund commented Sep 6, 2025 •

edited

Loading

afbjorklund Sep 22, 2025 •

edited

Loading