Home

LKL/SMP

Welcome to the lkl-linux SMP prototype wiki!

What is this

• Try to add SMP support in LKL.
• x86_64 and ARM64 are supported.
• A simple hack for ARM64 build support.
• A simple netmap networking backend driver.
• A networking framework improvement of adding batch operations ability.

Status

• at early stage = buggy and similar performance with UP ;)
• So far, I only did some basic tests for networking stack with shared library build.
• See also at https://github.com/lkl/linux/issues/370

Supported platform and features

1. x86_64 and ARM64 are supported.
2. I disabled file systems and NLS_* components for a while too.

Build LKL/SMP prototype

    # Checkout branch smp first in current working directory
    # This will build a SMP enabled binary for x86_64, default.
    $ make -C tools/lkl hijack

If you want to build netmap virtio backend, you need to execute below build steps

ln -s ${YOUR_NETMAP_SRCTREE} tools/lkl/netmap
make -C tools/lkl netmap=yes hijack

It is same as standard LKL building process, but you need to execute below commands before using hijack shared library:

• For LKL 4.11

#! /bin/bash -x

LKL_ARCH_OBJS=`find arch/lkl/ -name built-in.o | xargs`

ld -r -o tools/lkl/vmlinux-lkl.o init/built-in.o --start-group \
	usr/built-in.o ${LKL_ARCH_OBJS} \
	kernel/built-in.o certs/built-in.o mm/built-in.o fs/built-in.o ipc/built-in.o\
	security/built-in.o crypto/built-in.o block/built-in.o lib/lib.a lib/built-in.o \
	drivers/built-in.o sound/built-in.o firmware/built-in.o net/built-in.o virt/built-in.o \
	--end-group .tmp_kallsyms2.o

objcopy -R .eh_frame -R .syscall_defs \
	-Glkl_start_kernel -Glkl_sys_halt -Glkl_syscall -Glkl_trigger_irq -Glkl_get_free_irq\
	-Glkl_put_irq -Glkl_is_running -Gjiffies_64 \
	tools/lkl/vmlinux-lkl.o tools/lkl/vmlinux-lkl-X.o

ld -T arch/lkl/kernel/lkl.lds -shared -z combreloc --build-id -o tools/lkl/libvmlinux.so tools/lkl/vmlinux-lkl-X.o

• For LKL 4.13

#! /bin/bash -x

ld -r -o tools/lkl/vmlinux-lkl.o \
	--whole-archive built-in.o --no-whole-archive \
	--start-group lib/lib.a --end-group \
	.tmp_kallsyms2.o

objcopy -R .eh_frame -R .syscall_defs \
	-Glkl_start_kernel -Glkl_sys_halt -Glkl_syscall -Glkl_trigger_irq -Glkl_get_free_irq\
	-Glkl_put_irq -Glkl_is_running -Gjiffies_64 \
	tools/lkl/vmlinux-lkl.o tools/lkl/vmlinux-lkl-X.o

ld -T arch/lkl/kernel/lkl.lds -shared -z combreloc --build-id -o tools/lkl/libvmlinux.so tools/lkl/vmlinux-lkl-X.o

Hijack library

It hasn't special magic here, we can just use 'tools/lkl/bin/lkl-hijack.sh' to play it. however, please note that this script also is hacked by SMP prototype project.

Fast try

In my experiences, the simplest way to run LKL may be using network namespace in Linux, below are my helper scripts.

Setup two network namespaces

#! /bin/sh
ip netns del veth0 2>/dev/null
ip netns del veth1 2>/dev/null

modprobe veth

ip link add veth0 type veth peer name veth1
ip netns add veth0
ip netns add veth1
ip link set dev veth0 netns veth0
ip link set dev veth1 netns veth1
ip netns exec veth0 ifconfig veth0 10.0.0.1 hw ether 4:4:4:4:4:4 mtu 60000
ip netns exec veth1 ifconfig veth1 10.0.0.2 hw ether 2:2:2:2:2:2 mtu 60000
ip netns exec veth0 ifconfig lo down
ip netns exec veth1 ifconfig lo down

ip netns exec veth0 tc qdisc  add dev veth0 handle ffff: ingress
ip netns exec veth0 tc filter add dev veth0 parent ffff: u32 match u32 0 0 action drop

ip netns exec veth1 tc qdisc  add dev veth1 handle ffff: ingress
ip netns exec veth1 tc filter add dev veth1 parent ffff: u32 match u32 0 0 action drop

Start up iperf server

LKL_HIJACK_DEBUG=1023 \
LKL_HIJACK_DEBUG=1 \
LKL_HIJACK_SINGLE_CPU=1 \
LKL_HIJACK_NET_MTU=1500 \
LKL_HIJACK_NET_MAC=4:4:4:4:4:4 \
LKL_HIJACK_NET_NETMASK_LEN=8 \
LKL_HIJACK_NET_IP=10.0.0.1 \
LKL_HIJACK_NET_IFTYPE=raw \
LKL_HIJACK_NET_IFPARAMS=veth0  \
LKL_HIJACK_BOOT_CMDLINE="debug verbose initcall_debug=yes mem=4G" \
LKL_HIJACK_BOOT_CMDLINE="debug verbose initcall_debug=no mem=1G" \
LKL_HIJACK_SYSCTL="net.ipv4.tcp_mem=115494       153993  230988;net.ipv4.tcp_rmem=4096        87380   6291456;net.ipv4.tcp_wmem=4096        16384   4194304" \
ip netns exec veth0 \
./tools/lkl/bin/lkl-hijack.sh ~/network/inst/bin/iperf3 -s

Note:

1. Please replace ~/network/inst/bin/iperf3 with path name of your iperf binary.
2. Use sudo command to execute above script.

Start up iperf client

LKL_HIJACK_DEBUG=1023 \
LKL_HIJACK_DEBUG=1 \
LKL_HIJACK_NET_MTU=1500 \
LKL_HIJACK_NET_MAC=2:2:2:2:2:2 \
LKL_HIJACK_NET_NETMASK_LEN=8 \
LKL_HIJACK_NET_IP=10.0.0.2 \
LKL_HIJACK_NET_IFTYPE=raw \
LKL_HIJACK_NET_IFPARAMS=veth1 \
LKL_HIJACK_BOOT_CMDLINE="verbose initcall_debug=yes mem=4G panic_on_warn" \
LKL_HIJACK_SYSCTL="net.ipv4.tcp_mem=115494       153993  230988;net.ipv4.tcp_rmem=4096        87380   6291456;net.ipv4.tcp_wmem=4096        16384   4194304" \
ip netns exec veth1 \
./tools/lkl/bin/lkl-hijack.sh ~/network/inst/bin/iperf3 -c 10.0.0.1 $*

Note:

1. Please replace ~/network/inst/bin/iperf3 with path name of your iperf binary.
2. Use sudo command to execute above script.

Known bugs

Changes

• 4.13 porting: remove changeset of adding AArch64 support, since LKL can support them itself.