Find Peak Flow Mods per Second in various scenarios.

[Ktmi]

This is less an issue, and more of a request for performance measurements. In #187, @viniarck requested I check how pacing affected the flow mod rate in various scenarios. However, we never tested for what the peak flow mod rate would be for flow manager, nor did we test how the max flow mod rate would be affected by latency between the switch and the controller.

[Ktmi]

So for my experiment, I decided to try deploying 10,000 flows in a single request via the HTTP api. Here's the IO graph of the packets flow mod packets captured, with a sampling interval of 1 second.

This looked to be about 500 is flow mods per second, but didn't really explain the peaks and troughs, so I decided to look at it at a finer sampling interval of 50ms:

What we see here is that Kytos is able to sustain sending between 30 and 40 flow mods every 50 ms (or about 600 to 800 flow mods a second), but then has to stop sending flow mods for about a half second.

To see if the pauses was a kytos issue, I checked on all traffic going through the LO, to see if there was a pause:

Turns out there was. Maybe this is because I'm running on a VM, and when the hypervisor does a context switch on the cores the VM is running on no packets can be sent.

[viniarck]

@Ktmi, that seems similar to my local results, good to have those values documented. I was seding 10k flows for 1 dpid in a single request, the flows mods on wire on a / sec scale, peaked at 1000 (a bit above than what you've found, I'm not running on a VM), and steady with 600 flow mods / sec. So, yes, around 500 flow mods / sec is a good guideline.

Instrumenting the msg out handler, it also reflected that indeed the number of ops was around 60000 / 60 = 1000 (which matches the peaks at 1000)

In a scale of 50 msec, I didn't notice suspensions as you had, but it wasn't steady either

Let's stick with this 500 flow mods / sec per dpid recommendation. @Ktmi, can you double check if sending for more than one dpid if it still sustains a similar ratio?

[Ktmi]

@viniarck We already have evidence against being able to maintain the same ratio when sending to multiple dpids. In this post, one of the test cases visualized was sending 1000 flow mods to 9 different switches, and the combined flow mod rate was about the same as what I got here, around 500 per second, with the per switch somewhere around 50 per second. It seems that we have a global cap across all DPIDS for how fast we can send flow mods.

[viniarck]

@viniarck We already have evidence against being able to maintain the same ratio when sending to multiple dpids. In this post, one of the test cases visualized was sending 1000 flow mods to 9 different switches, and the combined flow mod rate was about the same as what I got here, around 500 per second, with the per switch somewhere around 50 per second. It seems that we have a global cap across all DPIDS for how fast we can send flow mods.

Right. That's a good data point. OK. Let's keep this in mind, and move on. Let's recommend the 500 flow mods / sec, including globally, and when that's not enough we'll see what else can be done. But, that's already very helpful to to determine certain pacers ceiling. Good.