-
Notifications
You must be signed in to change notification settings - Fork 5
ODL stress test performance report v1.2
Download below, the latest NSTAT Performance Stress Tests Report in pdf form.
[5/19/2016]: Performance Stress Tests Report v1.2: "Berylium Vs Lithium SR3"
In this report we investigate several performance and scalability aspects of the OpenDaylight Beryllium controller and compare them against the Lithium release. Beryllium outperforms Lithium in idle switch scalability tests both with Multinet (realistic OF1.3) and MT–Cbench (artificial OF1.0) emulators. A single Beryllium instance can boot ”Linear” and ”Disconnected” idle Multinet topologies of up to 6400 switches, as opposed to 4800 with Lithium. In general, Beryllium manages to successfully boot topologies that Lithium fails to, or it can boot them faster. In MT–Cbench tests the superiority of Beryllium is even more evident, being able to successfully discover topologies that expose themselves to the controller at a faster rate as compared to Lithium. With regard to controller stability, both releases exhibit similarly stable behavior. In addition, Multinet–based tests reveal traffic optimizations in Beryllium, witnessed by the reduced volume related to MULTIPART messages. Finally, in NorthBound performance tests Beryllium seems to underperform in accepting flow installation requests via RESTCONF, but as far as the end–to–end flow installation is concerned it is the clear winner. In many extreme cases of large topologies (e.g. 5000 switches) or large flow counts (1M), Beryllium manages to successfully install flows while Lithium fails.
In this report we investigate several performance and scalability aspects of the OpenDaylight Beryllium controller and compare them against the Lithium release. The investigation focuses on both the SouthBound (SB) and NorthBound (NB) controller’s interface and targets the following objectives:
- controller throughput,
- switch scalability,
- controller stability (sustained throughput),
- flow scalability and flow provisioning time
For our evaluation we have used NSTAT @NSTAT, an open source environment written in Python for easily writing SDN controller stress tests and executing them in a fully automated and end–to–end manner. NSTAT’s architecture is exhibited in [fig:nstat_architecture] and its key components are the
- SDN controller,
- SouthBound OpenFlow generator,
- NorthBound flow generator (RESTCONF ).
The SDN controller used in this series of experiments is OpenDaylight. We perform and demonstrate results from the Lithium SR3 and Beryllium RC2 releases.
For SouthBound (SB) generation we have used both MT–Cbench @MTCBENCH and Multinet @MULTINET. For NorthBound (NB) generation NSTAT uses custom scripts @NORTHBOUNDFLOWGENERATOR, originally developed by the OpenDaylight community @OPENDAYLIGHT, for creating and writing flows to the controller configuration data store.
MT–Cbench is a direct extension of the Cbench @CBENCH emulator which uses threading to generate OpenFlow from multiple streams in parallel. The motivation behind this extension was to be able to boot–up and operate network topologies much larger than those with the original Cbench, by gradually adding switches in groups. We note here that, as in the original Cbench, the MT–Cbench switches implement only a minimal subset of the OF1.0 protocol, and therefore the results presented for that generator are expected to vary in real–world deployments.
This gap is largely filled using Multinet @MININET as an additional SB generator, as it uses OVS virtual switches that accurately emulate the OF1.3 protocol. The goal of Multinet is to provide a fast, controlled and resource efficient way to boot large–scale SDN topologies. This is achieved by booting in parallel multiple isolated Mininet topologies, each launched on a separate virtual machine, and all connected to the same controller[^1]. Finally, by providing control on the way that switches are being connected to the SDN controller, one can easily test various switch group size/delay combinations and discover configurations that yield optimal boot–up times for a large–scale topology.
In our stress tests we have experimented with emulated switches operating in two modes, idle and active mode: swit-ches in idle mode do not initiate any to the controller, but rather respond to messages sent by it. Switches in active mode consistently initiate to the controller, in the form of PACKET_IN messages. In most stress tests, MT–Cbench and Multinet switches operate both in active and idle modes.
[^1]: As an example, Multinet can create more than 3000 Openvswitch OF1.3 switches over moderate virtual resources (10 vCPUs, $<$40GB memory) in less than 10 minutes.
The general architecture of NSTAT is depicted in \figurename\ref{fig:nstat_architecture}. The NSTAT node lies at the heart of the environment and controls all others. The test nodes \texttt{--}controller node, SB node(s), NB node\texttt{--} are mapped on one or more physical or virtual interconnected machines. Unless otherwise stated, in all our experiments every node is mapped on a separate virtual machine.
NSTAT is responsible for automating and sequencing every step of the testing procedure, including building and deploying components on their corresponding nodes, initiating and scaling SB and NB \traffic, monitoring controller operation in terms of correctness and performance, and producing performance reports along with system and application health state and profiling statistics. Each of these steps is highly configurable using a rich and simple JSON--based configuration system.
Each stress test scenario features a rich set of parameters that determine the interaction of the SB and NB components with the controller, and subsequently trigger varying controller behavior. Such parameters are for example the number of OpenFlow switches, the way they connect to the controller, the number of NB application instances, the rate of NB/SB \traffic~etc., and in a sense these define the "dimensions" of a stress test scenario.
One of NSTAT's key features is that it allows the user to specify multiple values for one or more such dimensions at the same time, and the tool itself takes care to repeat the test over all their combinations in a single session. This kind of exhaustively exploring the multi--dimensional experimental space offers a comprehensive overview of the controller's behavior on a wide range of conditions, making it possible to easily discover scaling trends, performance bounds and pathological cases.
Stability tests explore how controller throughput behaves in a large
time window with a fixed topology connected to it,
[fig:sbidlestabmincropfinal],
[fig:sbactivestabmtcbcropfinal]. The goal is to detect
performance fluctuations over time.
The controller accepts a standard rate of incoming and its response throughput is being sampled periodically. NSTAT reports these samples in a time series.
The purpose of this test is to investigate the stability of the controller to serve standard requirements of a large scale Multinet topology of idle switches.
During main test execution Multinet switches respond to ECHO and MULTIPART messages sent by the controller at regular intervals. These types of messages dominate the total traffic volume during execution.
NSTAT uses the oftraf @OFTRAF to measure the outgoing traffic of the
controller. The metrics presented for this case are the OpenFlow
packets and bytes collected by NSTAT every second,
[fig:05010112hoursstresstest0102].
In order to push the controller performance to its limits, the controller is executed on the bare metal and Multinet on a set of interconnected virtual machines
[!t]
[fig:addcontrollertimerateN10000]
[!t]
[fig:addcontrollertimerateN100000]
- topology size per worker node: 200
- number of worker nodes: 16
- group size: 1
- group delay: 2000ms
- topology type: "Linear"
- hosts per switch: 1
- period between samples: 10s
- number of samples: 4320
- persistence: enabled
- total running time: 12h In this case switch topology size is equal to 3200.
The results of this test are presented in \figurename\ref{fig:05_01_01_12_hours_stress_test_01_02}
In this series of experiments NSTAT uses a fixed topology of active MT–Cbench switches to generate for a large time window. The switches send artificial OF1.0 PACKET_IN messages at a fixed rate to the controller, which replies with also artificial OF1.0 FLOW_MOD messages; these message types dominate the exchanged between the switches and the controller. We evaluate the controller both in “RPC” and “DataStore” modes.
In order to push the controller performance to its limits, all test nodes (controller, MT–Cbench) were executed on bare metal. To isolate the nodes from each other, the CPU shares feature of NSTAT was used @NSTATCPUSHARES.
[!t]
[fig:addcontrollertimerateN1000000]
- controller: "DataStore" mode
- generator: MT--Cbench, latency mode
- number of MT-–Cbench threads: 10
- topology size per MT--Cbench thread: 50 switches
- group delay: 8s
- number of samples: 4320
- period between samples: 10s
- persistence: enabled
- total running time: 12h In this case, the total topology size is equal to 500 switches.
The results of this test are presented in \figurename\ref{fig:05_02_01_ds_mode_12_hours}.
- controller: "RPC" mode
- generator: MT--Cbench, latency mode
- number of MT--Cbench threads: 10
- topology size per MT--Cbench thread: 50 switches
- group delay: 8s
- number of samples: 4320,
- period between samples: 10s
- persistence: enabled
- total running time: 12h In this case, the total topology size is equal to 500 switches.
The results of this test are presented in \figurename\ref{fig:05_02_02_rpc_mode_12_hours}.
From [fig:05010112hoursstresstest0102] it is apparent
that both releases exhibit stable behavior in a time window of 12 hours.
A first observation is that Beryllium has lower idle overhead compared
to Lithium. To investigate the exact reasons for this discrepancy we
analyzed the traffic during a sample 1–min stability test with a
Linear–switch Multinet topology. Both controllers were configured to
request statistics every 5000 ms. The traffic breakdown for both cases
is depicted in Table [table:idlestabilitytestsconclusions].
[table:idlestabilitytestsconclusions]
It is made clear that the reduced in Beryllium is attributed to the reduction of MULTIPART messages, both incoming and outgoing, possibly as a result of optimizations implemented in Beryllium in this area.
As we can see in [fig:050201dsmode12hours],
[fig:050202rpcmode12hours], both controllers exhibit in
general stable performance, at almost the same throughput levels.
Lithium performs slightly better than Beryllium in “RPC” mode. In all
four cases, a transient performance degradation is observed for a
relatively large period. After analyzing system metrics we concluded
that this behavior could be related to an increase in the total used
memory of our system (see
[fig:050201dsmode12hours], [fig:050202rpcmode12hours])
as a result of possible memory leak.
[!t]
[fig:flowsinstallationrateN100010000]
[!t]
[fig:flowsinstallationrateN1000001000000]
With flow scalability stress tests we try to investigate both capacity
and timing aspects of flows installation via the controller NB
(RESTCONF) interface, [fig:nbflowscalmincropfinal].
This test uses the NorthBound flow generator @NORTHBOUNDFLOWGENERATOR to create flows in a scalable and configurable manner (number of flows, delay between flows, number of flow writer threads). The flows are being written to the controller configuration data store via its NB interface, and then forwarded to an underlying Multinet topology as flow modifications, where they are distributed into switches in a balanced fashion.
The test verifies the success or failure of flow operations via the controller’s operational data store. An experiment is considered successful when all flows have been installed on switches and have been reflected in the operational data store of the controller. If not all of them have become visible in the data store within a certain deadline after the last update, the experiment is considered failed.
Intuitively, this test emulates a scenario where multiple NB applications, each controlling a subset of the topology[^1], send simultaneously flows to their switches via the controller’s NB interface at a configurable rate.
The metrics measured in this test are:
-
Add controller time (t
add): the time needed for all requests to be sent and their response to be received (as in @OPENDAYLIGHT). -
Add controller rate (r
add): radd= N / tadd, where N the aggregate number of flows to be installed by worker threads. -
End–to–end installation time (t
e2e): the time from the first flow installation request until all flows have been installed and become visible in the operational data store. -
End-to-end installation rate (r
e2e): re2e= N / te2e
In this test, Multinet switches operate in idle mode, without initiating any apart from the MULTIPART and ECHO messages with which they reply to controller’s requests at regular intervals.
[^1]: subsets are non-overlapping and equally sized
For both Lithium and Beryllium we used the following setup
- topology size per worker node: 1, 2, 4, 35, 70, 330.
- number of worker nodes: 15
- group size: 1
- group delay: 3000ms
- topology type: “Linear”
- hosts per switch: 1
- total flows to be added: 1K, 10K, 100K, 1M
- flow creation delay: 0ms
- flow worker threads: 5
- persistence: disabled
In this case switch topology size is equal to: 15, 30, 60, 525, 1050, 4950.
The results of this test are presented in
[fig:addcontrollertimerateN10000],
[fig:addcontrollertimerateN100000],
[fig:addcontrollertimerateN1000000].
The results of this experiment are presented in
[fig:flowsinstallationrateN100010000], [fig:flowsinstallationrateN1000001000000].
Regarding NB performance, as it is expressed in terms of add controller time/ rate, Lithium performs generally better than Beryllium, but this trend tends to diminish and finally gets reversed for large topologies. Notably, Be outperforms Li by a large factor in the 1M flows/5K switches case.
However, regarding end–to–end performance, Beryllium is the clear winner, outperforming Lithium in all cases of topology sizes and flow counts. In many extreme cases where Lithium fails, Beryllium manages to successfully install flows. This happens for example in the case for 1M flows, or in some 5K switch topologies. This witnesses improvements in the Beryllium release relating to flow installation and discovery.
- Nikos Anastopoulos
- Panagiotis Georgiopoulos
- [Konstantinos Papadopoulos](mailto:[email protected]
Intro
Stress Tests
- Switch scalability test with active MT-Cbench switches
- Switch scalability test with active Multinet switches
- Switch scalability test with idle MT-Cbench switches
- Switch scalability test with idle Multinet switches
- Controller stability test with active MT-Cbench switches
- Controller stability test with idle Multinet switches
- Flow scalability test with idle Multinet switches
Emulators
Monitoring tools
- OpenFlow monitoring tools
Design
Releases
ODL stress tests performance reports
Sample Performance Results