-
Notifications
You must be signed in to change notification settings - Fork 50
Getting Started
Mizar is the data-plane implementation of the next-generation cloud network.
Note: This code relies on patched XDP driver that is not in upstream kernel yet, please follow this guide to use the patched kernel
We currently use make to build the entire data-plane code. We recommend using an ubuntu-based host for development and functionality testing. To compile, test, and experiment Mizar on an existing host, please follow the following steps: Clone the Mizar repository
$ git clone https://github.com/futurewei-cloud/mizar.git ~/mizar
$ cd ~/mizar
$ git submodule update --init --recursive
Run the bootstrap script. This step will do the following: examine your kernel version, update the kernel with the patched version, install all needed packages for development, and create a docker image buildbox:v2 to enable you to start running the functional tests.
$ ./bootstrap.sh
Compile and run tests. The make test step will run both unit and functional test. If this step passes, then you have everything needed to develop, test, and run Mizar on a single box!
$ make
$ make test
(Optional) You may want to run unit tests and get coverage reports by:
$ make run_unittests
$ make lcov
(Optional) To execute functional tests only, run:
$ make functest
(Optional) To execute a specific functional test
make run_func_test TEST=<test_class_name>
The previous steps allow you to run the functional tests as well as compiling Mizar. If you are interested only in rapidly compiling the code and running unit tests, you can follow the next steps on any Docker supported operating system:
Set up a docker image with the dependancies
sudo docker build -f ~/Mizar/test/Dockerfile -t buildbox:v2 ~/Mizar/test
Run the buildbox container:
sudo docker run -it -v ~/Mizar:/Mizar buildbox:v2
Inside the container, /Mizar will point to the same directory ~/Mizar on your host!
You can then run make and make lcov normally from the container. You will not be able to run make test as functional tests will not run.
We can test out Mizar interactively via Python and Docker.
For example, start python by running the below.
sudo python3
Then while inside python, run the following to import the test controller.
from test.trn_controller.controller import controller
from test.trn_controller.droplet import droplet
from test.trn_controller.common import cidr
from test.trn_func_tests.helper import *
import unittest
from time import sleep
We can now create three docker containers with Mizar by running the following.
droplets = {
"left": droplet("left"),
"right": droplet("right"),
"switch": droplet("switch"),
}
We can program these "droplets" by instantiating a controller object.
c = controller(droplets)
Then, running the commands below will create a VPC with 2 endpoints and 1 switch.
c.create_vpc(3, cidr("16", "10.0.0.0"), [])
c.create_network(3, 1, cidr("16", "10.0.0.0"), ["switch"])
ep_left = c.create_simple_endpoint(3, 1, "10.0.0.2", "left")
ep_right = c.create_simple_endpoint(3, 1, "10.0.0.3", "right")
Opening another command line window, we can try to ping ep_right from ep_left.
./tools/ps_docker_ips.sh
sudo docker exec -it your_ep_left_container_name /bin/bash
ip netns exec gnv_ns1 ping 10.0.0.3
In order to capture packets on an interface running either the transit Agent or the transit XDP programs, we use the xdpcap tool by cloudflare: https://github.com/cloudflare/xdpcap
The tools directory includes a compiled binary of xdpcap. The bpf map xdphooks are pinned in '/sys/fs/bpf/' directory
For example to capture traffic on the main interface running the transit XDP program from a test droplet run:
~/Mizar/tools/xdpcap /sys/fs/bpf/eth0_transit_pcap - | tcpdump -nr -
Similarly packets shall be captures on a peer interface running the transit agent by running:
~/Mizar/tools/xdpcap /sys/fs/bpf/peer1_transit_agent_pcap - | tcpdump -nr -
The bootstrap step applies a kernel update the has the following patch:
The architecture relies on an XDP program (transit agent) running on the veth peer in the root namespace (See the design document for details on the role of that program). The below diagram illustrates the packet path at a high level.
+-----------+
| veth0 |
+-----------+
^ |
| |
|
| Packet may be
| cloned
| Namespace of veth
| |
-------------+-----------+-------------------------------
| |
| v Root Namespace
| +-------+
| |Transit|
| | Agent |
+-+-------+-------+-+
| | veth0 - peer |
+-+-----------------+
| |
+---------+-----------------+----+
| eth0 | |
+--+---------------+--------+----+
| Transit | |
| XDP | |
|(Switch/router)| v
+---------------+
^
|
|
|
In the current generic XDP driver (https://lwn.net/Articles/720072/) tcp packets are not processed by XDP, if they are cloned. The reason is that the kernel clones the TCP packets (as they may be retransmitted) and the generic XDP is not executed for cloned packets as they may be changed.
The transit agent does not change the packet, but rather tunnel it (by adjusting its buffer head in an already created buffer). Only the extended part in the buffer is being written as an outer packet header. So we applied the following patch for the moment to verify the functionality of the data-plane for TCP. By doing this, we may encounter a performance penalty (or even an edge case that we did not test yet). But that is sufficient for this proof of concept. Ideally, we should have a separate XDP driver for this case, unless the mainstream kernel logic is modified.
diff --git a/net/core/dev.c b/net/core/dev.c
index fc676b2610e3..6df713df6b4f 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -4328,11 +4328,16 @@ static u32 netif_receive_generic_xdp(struct sk_buff *skb,
int hlen, off;
u32 mac_len;
+#if 0
/* Reinjected packets coming from act_mirred or similar should
* not get XDP generic processing.
*/
if (skb_cloned(skb) || skb_is_tc_redirected(skb))
return XDP_PASS;
+#endif
+
+ if (skb_unclone(skb, GFP_ATOMIC))
+ return XDP_DROP;
/* XDP packets must be linear and must have sufficient headroom
* of XDP_PACKET_HEADROOM bytes. This is the guarantee that also