Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge ElasticSearch tests to speed up full runs #124

Closed
wants to merge 1 commit into from

Conversation

ryanohnemus
Copy link
Contributor

There are currently 2 es tests: basic and compress.

For compress it had an entirely separate elasticsearch-compress and fluentbit-compress yaml files.

Differences are as follows:

Compression:
The elasticsearch file is only different than the basic one because it specified http.compression=true. According to the documentation, that is already the default when you do not use https, which we do not.

From the doc

http.compression
(Static, boolean) Support for compression when possible (with Accept-Encoding). If HTTPS is enabled, defaults to false. Otherwise, defaults to true.
Disabling compression for HTTPS mitigates potential security risks, such as a BREACH attack. To compress HTTPS traffic, you must explicitly set http.compression to true.

Therefore while the config looks different it's not actually different, so there's no need for a second elasticsearch config artifact.

elasticsearch-compress.yaml was removed.

Fluentbit:
The only differences between these tests were the es output in fluentbit-compress.yaml config had the compress gzip enabled. To cut out setup/teardown times. I removed the fluentbit-compress.yaml and added a second es OUTPUT in the fluentbit-basic.yaml that has compress gzip enabled. This now writes to a 2nd index in es called fluentbit-compress.

The basic intent here is to have a single fluentbit deployment that can attempt test sending with and without compression via 2 different outputs.

fluentbit-compress.yaml was removed.

The tests:
The only differences from the tests themselves is that the compressed version added a --compressed flag to the curl to request a compressed response.

compress.bats was removed. I merged the compressed test down into the basic.bats. The fluentbit and es deployment's happen in the setup_file function. Then 2 tests run quickly in sequence cutting a minute out of full runs.

Signed-off-by: ryanohnemus <[email protected]>
@ryanohnemus
Copy link
Contributor Author

========================
Starting tests.
========================

Fluentbit repository: ghcr.io/fluent/fluent-bit - tag: latest


1..8
ok 1 chunk rollover test in 230000ms
ok 2 test fluent-bit forwards logs to elasticsearch fluentbit (non-compressed) index in 5000ms
ok 3 test fluent-bit forwards logs to elasticsearch fluentbit-compressed (compressed) index in 0ms
ok 4 test fluent-bit adds k8s labels to records in 64000ms
ok 5 verify config in 6000ms
ok 6 test fluent-bit forwards logs to opensearch default index in 50000ms
ok 7 test fluent-bit forwards logs to AWS OpenSearch hosted service default index in 1000ms # skip Skipping Hosted OpenSearch When 'HOSTED_OPENSEARCH_HOST=localhost'
ok 8 Verify K8S cluster accessible in 0ms


========================
All tests passed!
========================

@ryanohnemus ryanohnemus deleted the es_speed_up branch January 28, 2024 20:14
@ryanohnemus ryanohnemus restored the es_speed_up branch January 28, 2024 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant