[pull] master from cloud-bulldozer:master #7

pull · 2021-03-30T19:17:04Z

See Commits and Changes for more details.

Can you help keep this open source service alive? 💖 Please sponsor : )

Adding log generator script and container file. Updated setup.cfg to set openshift and kubernetes version

This is needed as it takes a couple of seconds for the desired version in the clusterversion spec to get updated after an upgrade has been triggered. This commit also enables upgrading with force and even when there are warnings around image verification.

Signed-off-by: Sai Sindhur Malleni <[email protected]>

Add kafka python dependency

Signed-off-by: Sai Sindhur Malleni <[email protected]>

This is a corner case needed during our log testing. To accurately count the number of synthetic log messages generated by the log generator pod, we want to disable the regular SNAFU logging. This will make it harder to debug, but this is only something a user optionally sets in the benchmark-operator CR. The reason we want to do this is, we plan to deploy only the log-generator pods to a separate namespace and forward those logs to a separate kafka topic and then we can reliably read off the offset in the topic without reading each message to verify that all log messages generated by the log generator pod made it to kafka. Signed-off-by: Sai Sindhur Malleni <[email protected]>

Signed-off-by: Sai Sindhur Malleni <[email protected]>

- Update backend details to the doc sent to ES - Close kafka consumer - Increase the `end_time` used to get the kafka offset This is because, unlike ES, the timestamp is the one created by Kafka when the log message is recorded in Kafka. For example, in the case of ES, the timestamp added by FluenD becomes the timestamp field in ES, so we can query accunrately for log messages between the `start_time` and `end_time` of the log generation test. However, in the case of Kafka, we cannot accurately check for the log messages generated during the test because the records in Kafka and their corresponding timstamps for the offsets could be well after the `end_time` of the test as the messages slowly show up in the kafka cluster and there is no way to use the timestamp from FluentD as the timstamp for the kafka offset. The FluentD timestamp is actually in the content of the Kafka record. We currently check for all messages from the `start_time` and before the `timeout` configured in the CR. Signed-off-by: Sai Sindhur Malleni <[email protected]>

Signed-off-by: Sai Sindhur Malleni <[email protected]>

Added functionality for running sysbench

This commit adds a wrapper to run dns-perf which sends queries to the specified DNS server given a profile with the records in order to test the DNS performance. It parses the results and generated a json payload which can be indexed into Elasticsearch to help with the analysis.

Add DNS performance workload wrapper

Bumps [notebook](http://jupyter.org) from 6.4.7 to 6.4.10. --- updated-dependencies: - dependency-name: notebook dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]>

updates: - [github.com/Lucas-C/pre-commit-hooks: v1.1.10 → v1.1.13](Lucas-C/pre-commit-hooks@v1.1.10...v1.1.13) - [github.com/pre-commit/pre-commit-hooks: v4.1.0 → v4.2.0](pre-commit/pre-commit-hooks@v4.1.0...v4.2.0) - [github.com/psf/black: 21.12b0 → 22.3.0](psf/black@21.12b0...22.3.0) - [github.com/PyCQA/pylint: v2.12.2 → v2.13.8](pylint-dev/pylint@v2.12.2...v2.13.8) - [github.com/pre-commit/mirrors-mypy: v0.931 → v0.950](pre-commit/mirrors-mypy@v0.931...v0.950) - [github.com/asottile/pyupgrade: v2.31.0 → v2.32.0](asottile/pyupgrade@v2.31.0...v2.32.0)

This is needed for fixing an issue with running multiple pairs using NodePort. Related PR: cloud-bulldozer/benchmark-operator#755 Signed-off-by: Sai Sindhur Malleni <[email protected]>

This reverts commit f2a86c9.

This reverts commit bde152e.

updates: - [github.com/Lucas-C/pre-commit-hooks: v1.1.13 → v1.3.0](Lucas-C/pre-commit-hooks@v1.1.13...v1.3.0) - [github.com/pre-commit/pre-commit-hooks: v4.2.0 → v4.3.0](pre-commit/pre-commit-hooks@v4.2.0...v4.3.0) - [github.com/psf/black: 22.3.0 → 22.6.0](psf/black@22.3.0...22.6.0) - [github.com/PyCQA/pylint: v2.13.8 → v2.14.4](pylint-dev/pylint@v2.13.8...v2.14.4) - [github.com/pre-commit/mirrors-mypy: v0.950 → v0.961](pre-commit/mirrors-mypy@v0.950...v0.961) - [github.com/asottile/pyupgrade: v2.32.0 → v2.34.0](asottile/pyupgrade@v2.32.0...v2.34.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Bumps [notebook](http://jupyter.org) from 6.4.10 to 6.4.12. --- updated-dependencies: - dependency-name: notebook dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate updates: - [github.com/PyCQA/flake8: 4.0.1 → 5.0.2](PyCQA/flake8@4.0.1...5.0.2) - [github.com/PyCQA/pylint: v2.14.4 → v2.14.5](pylint-dev/pylint@v2.14.4...v2.14.5) - [github.com/pre-commit/mirrors-mypy: v0.961 → v0.971](pre-commit/mirrors-mypy@v0.961...v0.971) - [github.com/asottile/pyupgrade: v2.34.0 → v2.37.3](asottile/pyupgrade@v2.34.0...v2.37.3) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Bumps [nbconvert](https://github.com/jupyter/nbconvert) from 6.4.0 to 6.5.1. - [Release notes](https://github.com/jupyter/nbconvert/releases) - [Commits](jupyter/nbconvert@6.4.0...6.5.1) --- updated-dependencies: - dependency-name: nbconvert dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bug where the single character flag was `-worker` instead of `-w`

* rename sysbench folder to adhere to naming standard The sysbench folder lacked the wrapper suffix causing it to get snagged in pylint * remove dead lines in pylintrc The two lines do not have any impact on pylint * exclude hammerdb directory from being pylinted Since the hammerdb directory does not have the `_wrapper` suffix it is not being excluded from existing pylint rules * fix arugments-diff pylint for FuncAction Complaint from pylint when overriding `func` from `FuncAction` class. In both use cases it was used as a static method. * ignore pylint argument-differs error in uperf The uperf wrapper does not have a clean up and pylint complains it does not have the right number of arguments. * pylint fix for test_config * fix pylint issue by making lambda into a function Co-authored-by: Edwin Chong <[email protected]>

If ES connection fails and the tool=archive, then exit with a failure instead of only warning the user. Co-authored-by: Edwin Chong <[email protected]>

updates: - [github.com/Lucas-C/pre-commit-hooks: v1.3.0 → v1.3.1](Lucas-C/pre-commit-hooks@v1.3.0...v1.3.1) - [github.com/psf/black: 22.6.0 → 22.8.0](psf/black@22.6.0...22.8.0) - [github.com/PyCQA/flake8: 5.0.2 → 5.0.4](PyCQA/flake8@5.0.2...5.0.4) - [github.com/PyCQA/pylint: v2.14.5 → v2.15.3](pylint-dev/pylint@v2.14.5...v2.15.3) - [github.com/pre-commit/mirrors-mypy: v0.971 → v0.981](pre-commit/mirrors-mypy@v0.971...v0.981) - [github.com/asottile/pyupgrade: v2.37.3 → v2.38.2](asottile/pyupgrade@v2.37.3...v2.38.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate updates: - [github.com/psf/black: 22.8.0 → 22.10.0](psf/black@22.8.0...22.10.0) - [github.com/PyCQA/pylint: v2.15.3 → v2.15.5](pylint-dev/pylint@v2.15.3...v2.15.5) - [github.com/pre-commit/mirrors-mypy: v0.981 → v0.982](pre-commit/mirrors-mypy@v0.981...v0.982) - [github.com/asottile/pyupgrade: v2.38.2 → v3.2.0](asottile/pyupgrade@v2.38.2...v3.2.0) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* allow sysbench tool to take arguments The wrapper for sysbench only took a file for arguments. Added the ability to specify arguments through CLI using --args="" and arugements msut be surrounded by quotes * fix sysbench pylint Co-authored-by: Edwin Chong <[email protected]>

* adding dockerfile to generate nighthawk wrapper * Wrapper code for nighthawk workload * Resolved comments in the first iteration of PR-446 * Resolved comments in the second iteration of PR-446 * added README.md indicating steps to generate nighthawk-base image * resolved PR comments on nit picks --------- Co-authored-by: root <[email protected]> Co-authored-by: Vishnu Challa <[email protected]>

it misses 1 or 2 messages inconsistently irrespective of the message size, rate, duration, so adding a tolerance for ES

Adding log generator script and container file (#254)

2f5fee3

Adding log generator script and container file. Updated setup.cfg to set openshift and kubernetes version

pull bot added the ⤵️ pull label Mar 30, 2021

chaitanyaenr and others added 28 commits April 1, 2021 06:35

Add __init__ to log_generator_wrapper

f2fe29a

replaced flake with pre-commit hooks

45e91a0

fixed code style issues [PEP8 standard]

ad355ba

fixed unassigned var issue with fs_drift

16aad5d

Adding image_pull test

efe0599

added github actions for lint ci

9550a04

replaced tox with github actions

6524dbf

Add kafka backend for log verification

a9b670f

Signed-off-by: Sai Sindhur Malleni <[email protected]>

Add kata python dependency

c50dcdc

Merge pull request #271 from dry923/kata_python

6fa85d4

Add kafka python dependency

Use different group_ids per log generator pod

0e828b9

Signed-off-by: Sai Sindhur Malleni <[email protected]>

Send messages received by kafka to ES

d36f5b7

Signed-off-by: Sai Sindhur Malleni <[email protected]>

Added functionality for running sysbench

d1b7a40

Minor refactoring

3ed602b

Signed-off-by: Sai Sindhur Malleni <[email protected]>

Merge pull request #278 from acalhounRH/add_sysbench_prototype

3981522

Added functionality for running sysbench

fix ceph cachedrop URL and env. var. name

2bcf6c8

pycodestyle changes

f6346da

after pre-commit changes required by CI

f149bcf

no ofed check for mlnx nic

c412d97

Added functionality for creating and processing archive file

2c3a05a

Merge pull request #267 from chaitanyaenr/dns_perf_wrapper

2113db0

Add DNS performance workload wrapper

Add build, doc and other items to gitignore

9e88576

Add initial docs from sphinx-quickstart

a8769f3

Change docs theme to readthedocs

2965c19

dependabot bot and others added 30 commits May 6, 2022 13:31

Bump notebook from 6.4.7 to 6.4.10 in /requirements/py39-reqs

9df610b

Bumps [notebook](http://jupyter.org) from 6.4.7 to 6.4.10. --- updated-dependencies: - dependency-name: notebook dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]>

Bump notebook from 6.4.7 to 6.4.10 in /requirements/py38-reqs

4b410d8

Bumps [notebook](http://jupyter.org) from 6.4.7 to 6.4.10. --- updated-dependencies: - dependency-name: notebook dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]>

Assign uperf server port as an option

6631b3a

This is needed for fixing an issue with running multiple pairs using NodePort. Related PR: cloud-bulldozer/benchmark-operator#755 Signed-off-by: Sai Sindhur Malleni <[email protected]>

add ocp platform and networktype in scale index

f2a86c9

Revert "add ocp platform and networktype in scale index"

bde152e

This reverts commit f2a86c9.

Revert "Revert "add ocp platform and networktype in scale index""

146b5f4

This reverts commit bde152e.

added exit for uperf failures

2b10493

fix: coremark-pro flag

e20439e

Bug where the single character flag was `-worker` instead of `-w`

error out when using archive tool (#459)

0a42264

If ES connection fails and the tool=archive, then exit with a failure instead of only warning the user. Co-authored-by: Edwin Chong <[email protected]>

Allow messages received to be more than sent

e03677a

Add messages received output to test

59ba77c

more logs

55271e9

allowing a negligible message loss

0c9a7fe

it misses 1 or 2 messages inconsistently irrespective of the message size, rate, duration, so adding a tolerance for ES

making the tolerance to default

22488c5

removed extra chars

4b98b38

limit numpy version due to AttributeError (#484)

1876c51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[pull] master from cloud-bulldozer:master #7

[pull] master from cloud-bulldozer:master #7

pull bot commented Mar 30, 2021 •

edited

Loading

[pull] master from cloud-bulldozer:master #7

Are you sure you want to change the base?

[pull] master from cloud-bulldozer:master #7

Conversation

pull bot commented Mar 30, 2021 • edited Loading

pull bot commented Mar 30, 2021 •

edited

Loading