env86 devlog

[progrium]

I'm working on a tool to help with making v86 images called env86. Right now it lets you run v86 images without setting up a webpage and even get into a TTY from the shell. It also has my virtual network stack so you don't need to run a relay. It can also be used as a relay alternative that's cross platform.

Here is a full demo of it so far, but I have a lot more planned. I'll post updates here. Let me know what you think!

[SuperMaxusa]

Wow, looking forward to new updates!

[copy]

That looks very cool, I can't wait to play around with it.

[progrium]

Just dropped it into a public repo: https://github.com/progrium/env86

It's still very early (Windows and Linux users might not get a console window, please submit an issue if so), but it can be used to create images, boot VMs from the command line, provide a network relay, save changes to an image, and prepare static files to publish the VM to the web.

Let me know (issues or posting here) if you run into anything!

[progrium]

This is just a quick update rant covering some recent developments:

I originally sketched out env86 about 6 weeks ago and I'm pretty happy to finally have it implemented as I imagined. It can create Linux images from Docker or a root FS directory, now automatically detecting if there is a kernel+initrd to set its default configuration to use that. Though kind of a cool thing is that I embed a Linux kernel and initrd in env86 to use when one is not provided for an image.

I also just started a default guest service that will be done by 0.2. This guest service executable is also embedded in the env86 executable, so it can add it to new images using --with-guest in create. The guest service exposes the filesystem from inside (as opposed to the 9p mounted filesystem managed by v86, which won't include other mounts, etc), and also a PTY service. These were the minimum things needed to get the initial apptron.dev demo working. By 0.2 the guest service will let you run processes with a more formal API than just working with a PTY, it will let you dial and connect to local TCP services, and it will let you mount filesystems.

Anyway, that's future stuff, but right now there's already a bunch of things you may have missed. Besides the basic create/boot/prepare workflow documented in the readme, and working with the network stack, there are a couple of neat things I've added since.

Booting a VM by default opens a window to show a canvas element for the screen. This is the main source of cross platform issues as its based on a library I've made to make webview windows and other Electron-y desktop API things available in Go. But these currently use CGO, which makes compiling slow and means it has to link against libraries that requires their headers to be available when building (making Linux in particular annoying to build since it has a few dev package dependencies). This will eventually go away as I port these to purego, which will avoid CGO. However, the important thing is that even if you don't show the console window, it still creates a hidden webview window to run v86 in since it has a lot of JS that is dependent on a browser environment.

I wanted to be able to use env86 to build images on GitHub Actions, but you don't normally have a desktop environment for this to work. So I added support for Chrome Debug Protocol as an alternative way to run v86 in boot. This means you can just have Chrome installed and it will run a headless Chrome. I made a Docker container (on Docker Hub as progrium/env86) that has env86 with Chrome and this is what I'll use in GitHub Actions. To use CDP, you just pass --cdp for the boot command.

One thing I had to do for Apptron is chunk the initial state. Booting a tiny Alpine image to a prompt produces a ~90MB state file. Even compressed this is too large for files to be served from CloudFlare Pages (and probably other CDNs). At first I served it from the network gateway running on Heroku, but this ended up being pretty slow. So I split the initial state into 10MB parts, which not only lets them live on the CDN, but I can fetch them in parallel before assembling into a single arraybuffer to give v86. Originally I worked this out for Apptron, but I added it to env86 so that when you prepare an image, it will split up any initial state, and the recently added env86.js runtime (that wraps libv86.js), now knows how to detect, fetch, and assemble these parts.

The last thing for this update that I just added is a mechanism that lets me use env86 to script making initial state instead of using a Node.js script. So boot has a --save flag that will save the state to initial state for the image before quitting. So you can boot with --save, get it to whatever state, or even from an existing initial state get it to a new state, then Ctrl-D to EOF and exit and it will save a new initial state. To script this like in a build process, there is another flag I added called --exit-on which takes a string pattern. When combined with --ttyS0, it'll watch the output and if the pattern is seen it will quit. So if you have --save and --exit-on you can use boot to produce an initial state.

A few people have helped me work out the Linux experience, which like I mentioned before will for now require your distro equivalent of these Alpine packages: pkgconf webkit2gtk-dev gtk+3.0-dev libayatana-appindicator-dev. The Dockerfile.build under scripts has everything to build env86 in Linux. Until an official release where you can just download a build, building does require Docker to prepare some of the assets that are embedded in env86. But like I mentioned, you could use env86 via a Docker image on Docker Hub, you just have to mount in directories for input/output.

However, I've yet to see how badly this breaks on Windows. So if there are Windows users willing to work with Go and Docker, I'd love to see if you can build and run env86. Getting it to run, eventually with no dependencies, across platforms will make it ... an interesting alternative to Docker for certain use cases.

[SuperMaxusa]

Booting a tiny Alpine image to a prompt produces a ~90MB state file. Even compressed this is too large for files to be served from CloudFlare Pages (and probably other CDNs).

Not sure if this is suitable for your use case, but how about loading that state file and other large files from a storage bucket (like a Cloudflare R2 or AWS S3)?

[progrium]

That would have been the next step, except that splitting it solved the problem even better since it would pull the parts in parallel and from a CDN.

[progrium]

There is now a guest service that you can add to images that opens up a larger API for working with the VM. It lets you:

• Access the filesystem
• Run processes (with similar API to os/exec) optionally with PTY
• Dial TCP sockets
• Mount directories

This is exposed mostly via the run command, which starts to feel a lot like docker run. Now that we have that, it wouldn't be hard to add env86 build that takes something like a Dockerfile. But the most complicated part here is mounting directories. If you specify a mount spec in env86 run, we run a 9P server serving the path you specified, then tunnel a connection to that server through the guest service and mount it. And tonight it works!

There will be another mechanism for mounting in the guest, which uses FUSE and works by implementing a few RPC methods. This can be used as a simpler way to implement virtual filesystems, but is mainly so we can wrap the browser Filesystem API so when running in the browser we can mount user host directories (no 9P necessary).

[progrium]

Here is the last update for a while but it's in video form. I'm not going away just focusing on other projects now that its in a good enough state for me, but if anybody has questions or anything let me know!

https://youtu.be/q2I37ZtYfm8

[SuperMaxusa]

Amazing video and updates!

I have some ideas about initial state you mentioned in your previous posts:

In v3.20.2, virt kernel (slimmed kernel for virtual machines) has ne2k-pci support (#797 (comment)).

From my tests, the rootfs size is about 46 MB (and 96 MB with installed node.js), initial state size is 40 MB (compressed with zstd -19 - 8 MB)

Here is my updated tools/docker/alpine/Dockerfile: SuperMaxusa@f0a536f (need to set ENV KERNEL=virt), I can test it with env86 later if you want.

Also try check lsmod on your lts kernel for running unnecessary modules: #976 (comment)