[WIP] runllc

[lumjjb]

Modifications specific to Low Level containers (runllc). More details added soon, document is WIP.

Overview

General idea is that OCI runtimes and containers do not map directly to unikernels or other low level type technologies and other similar technologies (i.e. UML, LKL, MirageOS, OSv, etc.). So we want to handle everything the container OCI interface throws at us, and provide pluggable parts of it FSHandle, NetworkHandle, ExecHandle where unikernel providers and runtimes can just implement a couple of functions and start orchestrating their unikernels on k8s. i.e. right now runnc is an ExecHandle for solo5-spt, but another ExecHandle could be solo5-hvt, or UML, LKL, etc.

Motivation

• Semantic Gap: Container ßà Low Level (LL) execution units (VMs, unikernels, etc.)
• In the process of writing runnc, we realized that the main things that mattered were FS, network and the actual execution unit. The rest can be handled the same.
• People like k8s, and k8s is becoming leaner and more pluggable, which may allow more suitability for LL workloads
• Exploring other options: UML, LKL, etc.

Goals

• Create a library that users of the LL community can use to
• Bridge the container and unikernel/LL world, provide interfaces that are more familiar to the LL community.

---

Checklist

• Create LL interface (llif/*.go)
• Create FS interface
  • Find inputs for interface
  • Sample LL Handler
  • Finalize interface
• Create Network interface
  • Find inputs for interface
  • Sample LL Handler
  • Finalize interface
• Create Exec interface
  • Find inputs for interface
  • Sample LL Handler
  • Finalize interface
• Plumbing to move state around different LL Handlers and container state
  • Options (pass in container config)
  • InMemoryObjects (pass in memory)
  • How to handle Exec? Or just revamp model to work around exec? Since exec is run in the child process not in the factory process
• Create library for input of LLC
• Apply changes from Fix missing /etc directory in rootfs #77
• Handle changes specific to nabla Initial memory minimum fix #79

---

Integration Points

In factory_nabla.go create:

Storage:

runnc/libcontainer/factory_nabla.go

Lines 168 to 171 in 46ededd

	fsPath, err = createRootfsISO(config, containerRoot)
	if err != nil {
	return nil, err
	}

Network:

runnc/libcontainer/factory_nabla.go

Lines 174 to 180 in 46ededd

	err = network.CreateTapInterface(nablaTapName(id), nil, nil)
	if err != nil {
	if fsPath != "" {
	os.Remove(fsPath)
	}
	return nil, fmt.Errorf("Unable to create tap interface: %v", err)
	}

In runnc_cont.go:

Disk

runnc/runnc-cont/runnc_cont.go

Lines 138 to 141 in 46ededd

	disk, err := setupDisk(r.Disk)
	if err != nil {
	return fmt.Errorf("could not setup the disk: %v", err)
	}

Network

runnc/libcontainer/factory_nabla.go

Lines 159 to 180 in 46ededd

	// If it is a pause container for kubernetes, set config so that init
	// will just pause instead of executing a nabla
	fsPath := ""
	if isPauseContainer(config) {
	config, err = applyPauseHack(config, containerRoot)
	if err != nil {
	return nil, err
	}
	} else {
	fsPath, err = createRootfsISO(config, containerRoot)
	if err != nil {
	return nil, err
	}
	}
	err = network.CreateTapInterface(nablaTapName(id), nil, nil)
	if err != nil {
	if fsPath != "" {
	os.Remove(fsPath)
	}
	return nil, fmt.Errorf("Unable to create tap interface: %v", err)
	}

runnc/runnc-cont/runnc_cont.go

Lines 143 to 166 in 46ededd

	if r.IsInDocker {
	// The tap device will get the IP assigned to the Docker
	// container veth pair.
	r.IPAddress, r.GateWay, r.IPMask, mac, r.Tap, err = network.CreateMacvtapInterfaceDocker("eth0")
	if err != nil {
	return fmt.Errorf("could not create %s: %v", r.Tap, err)
	}
	} else if r.IsInKubernetes {
	// The tap device will get the IP assigned to the k8s nabla
	// container veth pair.
	// XXX: This is a workaround due to an error with MacvTap, error was :
	// Could not create /dev/tap8863: open /sys/devices/virtual/net/macvtap8863/tap8863/dev: no such file or directory
	r.IPAddress, r.GateWay, r.IPMask, mac, err = network.CreateTapInterfaceDocker(r.Tap, "eth0")
	if err != nil {
	return fmt.Errorf("could not create %s: %v\n", r.Tap, err)
	}
	} else {
	err = network.CreateTapInterface(r.Tap, &r.GateWay, &r.IPMask)
	if err != nil {
	// Ignore networking related errors (i.e., like if the TAP
	// already exists).
	fmt.Fprintf(os.Stderr, "Could not create %s: %v\n", r.Tap, err)
	}
	}

Exec

runnc/runnc-cont/runnc_cont.go

Lines 168 to 222 in 46ededd

	_, err = os.Stat(r.UniKernelBin)
	if err != nil {
	// If the unikernel path doesn't exist, look in $PATH
	unikernel, err := exec.LookPath(r.UniKernelBin)
	if err != nil {
	return fmt.Errorf("could not find the nabla file %s: %v", r.UniKernelBin, err)
	}
	r.UniKernelBin = unikernel
	}
	unikernelArgs, err := CreateRumprunArgs(r.IPAddress, r.IPMask, r.GateWay, "/",
	r.Env, r.WorkingDir, r.UniKernelBin, r.NablaRunArgs)
	if err != nil {
	return fmt.Errorf("could not create the unikernel cmdline: %v\n", err)
	}
	var args []string
	if mac != "" {
	args = []string{r.NablaRunBin,
	"--x-exec-heap",
	"--mem=" + strconv.FormatInt(r.Memory, 10),
	"--net-mac=" + mac,
	"--net=" + r.Tap,
	"--disk=" + disk,
	r.UniKernelBin,
	unikernelArgs}
	} else {
	args = []string{r.NablaRunBin,
	"--x-exec-heap",
	"--mem=" + strconv.FormatInt(r.Memory, 10),
	"--net=" + r.Tap,
	"--disk=" + disk,
	r.UniKernelBin,
	unikernelArgs}
	}
	fmt.Printf("nabla-run arg %s\n", args)
	// Set LD_LIBRARY_PATH to our dynamic libraries
	env := os.Environ()
	newenv := make([]string, 0, len(env))
	for _, v := range env {
	if strings.HasPrefix(v, "LD_LIBRARY_PATH=") {
	continue
	} else {
	newenv = append(newenv, v)
	}
	}
	newenv = append(newenv, "LD_LIBRARY_PATH=/lib64")
	err = syscall.Exec(r.NablaRunBin, args, newenv)
	if err != nil {
	return fmt.Errorf("Err from execve: %v\n", err)
	}

In libcontainer/container_nabla.go cleanup:

Storage

runnc/libcontainer/container_nabla.go

Lines 269 to 271 in 46ededd

	if err := os.RemoveAll(c.root); err != nil {
	return err
	}

Network

runnc/libcontainer/container_nabla.go

Lines 273 to 275 in 46ededd

	if err := network.RemoveTapDevice(nablaTapName(c.id)); err != nil {
	return err
	}

edge cases:

Pause Hack:

runnc/libcontainer/factory_nabla.go

Lines 162 to 166 in 46ededd

	if isPauseContainer(config) {
	config, err = applyPauseHack(config, containerRoot)
	if err != nil {
	return nil, err
	}