Build Load Testing as part of Fluent Bit release testing

[zhonghui12]

We (AWS for Fluent Bit team) want to discuss with Fluent Bit and the community about building load testing as part of Fluent Bit release testing.

AWS for Fluent Bit provides Fluent Bit support for AWS customers and we mainly work on contributions of AWS-related plugins. Recently we've launched ECS load testing framework and built it as part our release testing. The framework will run load testing with different throughput levels (10Mb/s, 20Mb/s, 30Mb/s) on ECS and send logs to AWS output plugins (CloudWatch, Kinesis, Firehose, S3) and validate these logs based on log loss and log duplication.

From AWS for Fluent Bit 2.23.0, we provide benchmarks of our release testing in the release note. We believe the load test and its result can raise the bar for releases and ensure reliability and stability.

This framework is part of the previous discussion: #4390 and as it is out now, we want to collect more feedback here how customers want to use this as part of Fluent Bit release testing.

FYI - our next step is to build EKS load testing framework which will provide more references for EKS and K8s customers. Will post update here if there is any progress on that.

[zhonghui12]

We will continue contribute to the testing and please comment if any input/output plugins you want us to include. Thanks.

[agup006]

Adding @patrick-stephens to the discussion

[patrick-stephens]

I'm quite interested in this, it is something I want to hang off the staging environment we now have prior to release so we can run load and resilience tests.

I'm not really sure where the testing from AWS is though, could you point me at it and how to run it so I can see how we could integrate it?

[agup006]

@zhonghui12 would you be able to point us to tests for AWS?

[zhonghui12]

@patrick-stephens @agup006
Sure. Our load test is located here: https://github.com/aws/aws-for-fluent-bit/tree/mainline/load_tests. And I have a PR to explain how we can use it: aws/aws-for-fluent-bit#312. However, current version fits our teams(AWS) pipeline more so feel free to contact me offline and we could collaborate together how it can be merged into your environment.

[patrick-stephens]

Sounds good, ping me 'pat' 'at' 'calyptia' 'dot' 'com' and we'll arrange something.

[patrick-stephens]

To keep folks in the loop on the community side, I'm going to set some time aside in the next few weeks to see if I can get the AWS tests running on the OSS infrastructure. There is still some aspects to update particularly around documentation and deployment but this is what we want to exercise and improve.
@zhonghui12

[aman556]

Hello @patrick-stephens, Thanks for following up here. By mentioning E2E what I mean is that after the package installation we run the test which is checking all the functionality of the package what it meant for. Like for fluent -bit checking the logs and the metrics processing.
Thanks!

[patrick-stephens]

Yeah to be clear that is hard to test every combination (not just of input, filter, output but also configuration of each, e.g. storage vs in-memory buffering): I think you have a specific set of plugins you want to test work together which is fine so an integration test showing that various input plugins, combined with various filters feeding various output plugins works. Note that the AWS contribution above is a loading test that does exercise some of this (maybe all if you want AWS outputs?).

Can you enumerate what you need to test, on which platforms (the AWS contribution is container only currently)?

We do currently have simple smoke tests of packages and containers as part of the release process, i.e. they run up with some default configuration and respond correctly. This is very basic and intended to verify the packaging process only, testing that plugins work is part of the integration test. We also want to add resilience and load testing, hence the AWS contribution above.

[aman556]

So Tanzu Community edition supports bare metal i.e docker provider and three clouds AWS, Azure, vSphere. So If a test can be written in shell which we can run inside ubuntu environment inside above providers.

hard to test every combination
Let me rephrase by all the functionality I just want that the test can check the core function/feature of the package in case of fluent-bit checking the logs and the metrics processing that it have processed/created them or not.

[aman556]

Also can you please show me the path for these tests maybe i found the thing there which i am looking for.