Reassess default kytos.conf api thread pool size and the related api_concurrency_limit

[viniarck]

This is for reassessing default kytos.conf api thread pool size and the related api_concurrency_limit.

Current potential problems:

• The current default 160 has been tweaked once, but I'm not sure for how many EVCs and network convergence were tested at the time. We know that mef_eline, flow_manager and telemetry_int are the most request intensive ones, so this should be reassessed with a scenario that is not too massively large but still realistic to a production network, for instance 360 EVCs in place (so, maybe leave some buffer and go for 400 EVCs).

• We can also consider keeping the default, and only documented default recommended values. We'd like to have a somewhat reasonable default value that won't need too much tweak out of the box, while still with a value that doesn't cause other issues, we need to find a balance.

When it API concurrency gets limited it returns 503 contributing to stability, however currently there are other open bugs that lead to major issues such as:

kytos-ng/mef_eline#483
kytos-ng/mef_eline#485

These related bugs will need to be fixed too, but we need to review if api_concurrency_limit needs to be reajusted and/or document recommended values for a given EVC scalability.

cc'ing @Ktmi and @italovalcy

[Ktmi]

Did some testing, and here are the results:

Test Case 1

Test Environment:

• Default pacing (100 flow mods per NApp per DPID)
• 400 Total EVCS
• 200 Affected by Link Down

API Threadpool Size	Status
1024	PASS
512	PASS
256	PASS
128	PASS
64	FAIL

Test Case 2

Notes: Previous tests clearly weren't hitting it hard enough, stepping it up a bit here.

Test Environment:

• Pacing Disabled using ignore_pace
• 400 Total EVCS
• 400 Affected by Link Down

API Threadpool Size	Status
512	PASS
256	PASS
128	FAIL
192	FAIL
224	PASS

Test Case 3

Test Environment:

• Pacing Disabled using ignore_pace
• 800 Total EVCS
• 800 Affected by Link Down

API Threadpool Size	Status
256	FAIL
512	PASS
384	PASS
320	FAIL
352	FAIL
384 (REDO)	FAIL
448	PASS
416	PASS

Conclusion

To me it seems that having around 1 API thread per 2 EVCs is near the tipping point of stability with this. I would err towards 2 API threads for every 3 EVCs. So assuming we have a system with 400 EVCs, then it should be around 260 API threads.

[viniarck]

Great results and data points @Ktmi. It'll serve well as current and future reference.

Did you also have a chance to see if increasing the api thread pool size if it's consuming too much extra RAM or not much? Can we settle on a 512 api thread pool size for the default size on kytos.conf and then explain on changelog some guidelines that you pointed out here? If it's not much let's slightly overshoot, since we know that at the moment we still don't have some full error handling for certain requests after their retries exceed.

As know, and for other to be aware too, we currently have a bit of extra RAM consumption #404 that will be investigated in the future, but if we're not adding too much it's OK

[Ktmi]

@viniarck Checking the RAM usage, it doesn't appear to use much ram. For comparison running at 512 API threads,, with the 800 EVCs, results in using around ~370MB of ram, with a total of around 860 kernel threads. Running with 256 API threads with 800 EVCs results in using around ~360 MB of ram, with a total of 720 kernel threads.

[viniarck]

@viniarck Checking the RAM usage, it doesn't appear to use much ram. For comparison running at 512 API threads,, with the 800 EVCs, results in using around ~370MB of ram, with a total of around 860 kernel threads. Running with 256 API threads with 800 EVCs results in using around ~360 MB of ram, with a total of 720 kernel threads.

Cool. Good to know. Let's go for 512 then? That seems like a good compromise while leaving plenty of room while supporting many production-like networks considering AmLight as our baseline reference.