[YUNIKORN-2037] Testing the throughput of Yunikorn

[wusamzong]

What is this PR for?

Capture the outcomes of KWOK-based throughput tests, while details about the tools and examples employed will be found in separate PRs. Upon acceptance of the tools PR, a link to the tool will be integrated into this document, accompanied by additional tutorials on KWOK environment building.
PR links- lightweight monitor tool, example&Kwok-setup-tool

Todos

• [*] - Task

What is the Jira issue?

• Open an issue on Jira https://issues.apache.org/jira/browse/YUNIKORN-2037/

How should this be tested?

Screenshots (if appropriate)

Questions:

• - The licenses files need update.
• - There is breaking changes for older versions.
• - It needs documentation.

[pbacsko]

@wusamzong could you rename the JIRA and the PR to sth like "Testing the performance of Yunikorn" or "Testing the throughput of Yunikorn"? The phrase "performance of throughput" sounds weird and does not make sense.

[wusamzong]

@wusamzong could you rename the JIRA and the PR to sth like "Testing the performance of Yunikorn" or "Testing the throughput of Yunikorn"? The phrase "performance of throughput" sounds weird and does not make sense.

Sure! Thank you for your correction

[imliuda]

I wonder what is the performance about 100 applications, 500 applications, and so.

[pbacsko]

I wonder what is the performance about 100 applications, 500 applications, and so.

Througput is definitely based on apps/pods ratio, queue hiearchy, etc. Things like failed headroom checks and pending pods slow things down.

[zhuqi-lucas]

@wusamzong
One question about plugin mode, if we compare plugin mode with preEnqueue and without preEnqueue, and preEnqueue is applied to newer k8s version.

[wusamzong]

@zhuqi-lucas Sorry for my delayed response. In my experiment, I observed no significant difference between using the plugin mode with preEnqueue and without preEnqueue. The experiment was conducted on Kubernetes version v1.27.8 & YuniKorn 1.4.0, deploying 5 applications, each with 1000 pods. The throughput of the YuniKorn plugin with PreEnqueue was 50 pods/sec, while the throughput without PreEnqueue was marginally higher at 50.01 pods/sec. Attached is a flame graph that compares the performance of YuniKorn in standard mode, plugin mode with PreEnqueue, and plugin mode without PreEnqueue.

[zhuqi-lucas]

Thanks @wusamzong for info, it makes sense to me, because we use preEnqueue to improve autoscaling case when we make pod unschedulable we support not going to scheduling cycle with preEnqueue plugin, but our performance testing case will not reach the autoscaling part, it's enough for this case.

[pbacsko]

@wusamzong could you upload the binary files so I can examine them? Thanks.

[wusamzong]

@pbacsko Sure! Here is the result of the pprof.
standard-mode.samples.cpu.001.pb.gz
standard-mode.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz
plugin-mode.samples.cpu.001.pb.gz
plugin-mode.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz

[wusamzong]

Thanks @wusamzong for info, it makes sense to me, because we use preEnqueue to improve autoscaling case when we make pod unschedulable we support not going to scheduling cycle with preEnqueue plugin, but our performance testing case will not reach the autoscaling part, it's enough for this case.

Okay, thank you for clarifying the outcomes of this experiment. I now have a much better understanding of the plugin aspect.

[pbacsko]

@pbacsko Sure! Here is the result of the pprof. standard-mode.samples.cpu.001.pb.gz standard-mode.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz plugin-mode.samples.cpu.001.pb.gz plugin-mode.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz

Thanks. Standard mode looks pretty solid. I haven't seen anything that stands out and needs immediate addressing.

[wilfred-s]

Is there anything that is left to do on this or should we add this as is to show the current state?

[wusamzong]

I'll make an effort to upload the remaining test cases as soon as possible!

[pbacsko]

@wusamzong is there a docs update which explains how to run these tests? It's OK to see the results, but what if I want to run this locally on my machine?

[wusamzong]

Certainly! I plan to submit a PR to k8shim, adding the scripts used for the performance test to the yunikorn-k8shim/deployments/kwok-perf-test. Along with this, I'll include a readme that provides instructions on how to utilize these scripts.

[pbacsko]

Ok, thanks, this PR looks good.

[pbacsko]

+1