Add testbench #39

tibersam · 2022-02-10T16:50:25Z

This PR implements some parts of #29.
It adds three tests:

Unittest of analysisfunctions used in other tests
Test of different inputs into python worker
Test of the controller running

This PR does not implement all of the features of #29 and is more thought as a start of adding automated testing to ARCHIE

tibersam · 2022-02-15T11:27:30Z

@lukasauer and @aewag can you review this?

aewag · 2022-02-15T13:34:14Z

tests/test_controller.py

+from hdf5logger import hdf5collector
+
+
+def compare_hdf5_files():


Locally, I used h5diff two compare an output vs the golden hdf5 file.

Can we also use this here (or a similar tool)?

I have no experience with h5diff. What i want to avoid is a failed test if the experiment index numbers change. Can h5diff cope with it?

It will most likely print an error as the files don't match. Does this happen as well with only a single runner? I would have expected that the output of ARCHIE should be deterministic in this case.

I will not use h5diff, as it is not python native and i want to use python assert. However i will use the same trick i am using to compress the collected analysis data from the experiment datas and check if it is empty.

Now the testbenches also assert if indexing stays consistant by asserting the experiment group title and testing extracted index numbers.

lukasauer

Thanks for the pull request! A couple of additional things I noticed:

The test take quite a long time to run. Is there a way to reduce this? Perhaps by reducing the number of faults? Ideally we would want something that can also be run locally to check if something broke.

Your tests use the stm32f0discovery QEMU machine, which is only available in our downstream version of QEMU. I want to keep our patches on top of QEMU to minimum. It would therefore be great, if we can use a machine that is in upstream QEMU for the tests. For example stm32vldiscovery.

lukasauer · 2022-02-16T12:15:49Z

.github/workflows/buildtester.yml

@@ -16,7 +16,7 @@ jobs:
      uses: actions/checkout@v2

    - name: install packages
-      run: sudo apt update; sudo apt upgrade -y; sudo apt install -y build-essential ninja-build libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev python3-tables python3-pandas python3-prctl
+      run: sudo apt update; sudo apt upgrade -y; sudo apt install -y build-essential ninja-build libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev python3-tables python3-pandas python3-prctl python3-pytest python3-pytest-cov python3-deepdiff


We might also want to add these to a new requirements file (tests/requirements.txt perhaps) so it's easier to get the test running locally. What do you think?

You are right, i will add a new requirements like you proposed.

Is this resolved? If yes, I think we can remove these here and link to the requirements file? BTW I didn't see deepdiff in the requirements.

deepdiff is no longer needed. I have pivoted to another solution. It is resolved from my perspective.

lukasauer · 2022-02-16T12:25:45Z

tests/test_python_worker.py

+from faultclass import python_worker, Fault
+
+
+class dummy_args:


I think test_args would be a better name here

lukasauer · 2022-02-16T12:26:21Z

tests/test_python_worker.py

+            [Fault(134217728, 0, 2, 1000, 3, 134217950, 1, 0)],
+            "tests/testdata/worker_data.xz",
+        ),
+        ([Fault(3, 2, 1, 0, 4, 134217954, 1, 0)], "tests/testdata/worker_register.xz"),


nit: Please use the same format as for the other entries

This formatting was introduced by black, i can add an ignore around it if you want.

lukasauer · 2022-02-16T12:27:30Z

tests/test_python_worker.py

+)
+def test_python_worker(fault, comparedata):
+    parguments = pythonworker_enviroment()
+    # print(parguments)


nit: please remove commented out code, here and in the other files as well

tibersam · 2022-02-28T15:52:22Z

The test take quite a long time to run. Is there a way to reduce this? Perhaps by reducing the number of faults? Ideally we would want something that can also be run locally to check if something broke.

The tests take most of their time in the comparing of the results. I will look into how to speed up the process, probably using h5diff. Also it can be speed up by using the -n option of pytest, aka running the test locally in parallel.

Your tests use the stm32f0discovery QEMU machine, which is only available in our downstream version of QEMU. I want to keep our patches on top of QEMU to minimum. It would therefore be great, if we can use a machine that is in upstream QEMU for the tests. For example stm32vldiscovery.

I can understand this, however the miniblink currently is the default binary we ship with archie and use to verify that the installation worked. Therefore i decided to use it for the tests.

* get_analysis_data_from_group retrieves tbinfo, tbexec and meminfo and returns dict * get_experiment_group execute adds also fault configuration * get_goldenrun_data returns the data of golden run

tibersam · 2022-03-03T13:13:26Z

The test take quite a long time to run. Is there a way to reduce this? Perhaps by reducing the number of faults? Ideally we would want something that can also be run locally to check if something broke.

Currently all 6 tests introduced with this PR take ~90s on my laptop. Coverage tracing increases the duration of the tests to ~170s. The controller test takes the longest with ~65s, analysisfunction takes ~5s, and python_worker takes ~17s. All times taken without running tests in parallel. If you look into the github runner in my repository all test take on the github vm ~ 4m30s. I think that these test can be run locally without long delays. Also if you specify which testbench to run you can for example exclude the controller test until you have finished the change.

- test.sh calls make in the faultplugin folder and then calls pytest to test archie - conftest.py is a file needed by pytest to include archie into its searchpath. Otherwise will throw import error

aewag

conftest.py: For what is the empty file needed?

aewag · 2022-03-10T11:37:53Z

.github/workflows/buildtester.yml

@@ -16,7 +16,7 @@ jobs:
      uses: actions/checkout@v2

    - name: install packages
-      run: sudo apt update; sudo apt upgrade -y; sudo apt install -y build-essential ninja-build libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev python3-tables python3-pandas python3-prctl
+      run: sudo apt update; sudo apt upgrade -y; sudo apt install -y build-essential ninja-build libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev python3-tables python3-pandas python3-prctl python3-pytest python3-pytest-cov python3-deepdiff


Is this resolved? If yes, I think we can remove these here and link to the requirements file? BTW I didn't see deepdiff in the requirements.

tibersam · 2022-03-10T12:13:49Z

conftest.py: For what is the empty file needed?

I have written the reason in the readme.
The Problem: If you call pytest (not python3 -m pytest) the Archie folder is not added to the search path of the python interpreter. However if it finds the conftest.py pytest adds the folder it is contained in to the search path. Why this is not the default of pytest is beyond me.

conftest.py is used to setup common fixtures needed by the different test benches, currently it is not used as this but might be in the future. We currently need it to "hack" the searchpath problem...

aewag · 2022-03-10T13:40:26Z

@lukasauer and me just discussed that we would like to keep this PR for now open. We have some more local changes and todos we would like to solve prior to merging this. One major change is to rebase ARCHIE on top of the newest QEMU version, which introduces some breaking changes and which is already prepared by Lukas.
So it probably makes sense to wait until this has landed and after that progress further with this PR.

tibersam · 2022-03-28T16:08:55Z

@aewag Cool. Then I will wait. Do you have a time frame when you will update? Like in a week, in a month, in two months ...?

lukasauer · 2022-03-28T16:18:05Z

We are waiting for QEMU 7.0 to be released, which should be soon (currently it's at rc1).

tibersam · 2022-07-22T14:08:54Z

@lukasauer Do you have a timeline when you will update the project?

lukasauer · 2022-07-22T16:42:51Z

Sorry for the delay, this is currently planned for next month.

tibersam · 2023-02-12T11:44:07Z

@lukasauer Sorry for the long absence. When PR #55 passes, should the test bench use both examples instead of miniblink for testing? I would then try to rework the testbench the coming weeks to reflect the changes introduced since this PR was opened.

aewag · 2023-02-14T17:06:31Z

@lukasauer Sorry for the long absence. When PR #55 passes, should the test bench use both examples instead of miniblink for testing? I would then try to rework the testbench the coming weeks to reflect the changes introduced since this PR was opened.

We have currently more changes internally. I think it may be best to wait for the rework until they are landed.

tibersam force-pushed the add_testbench branch 2 times, most recently from 9c46583 to a1ed441 Compare February 14, 2022 12:15

aewag reviewed Feb 15, 2022

View reviewed changes

lukasauer requested changes Feb 17, 2022

View reviewed changes

tibersam added 7 commits March 2, 2022 00:10

Add analysis data retrieval functions using dictionaries

dc0bbb3

* get_analysis_data_from_group retrieves tbinfo, tbexec and meminfo and returns dict * get_experiment_group execute adds also fault configuration * get_goldenrun_data returns the data of golden run

Add test for analysisfunctions used in other test functions

2fec00e

Add tests for python worker with needed result data

8bdae66

Add test to run complete setup and check output hdf5 file

b5a3d60

Add pytest.ini in root folder

c9096b8

Update buildtester to run test bench

ae84868

Added requirements.txt for pytest

752bd5a

tibersam force-pushed the add_testbench branch from 62edb83 to 752bd5a Compare March 1, 2022 23:10

tibersam requested a review from lukasauer March 9, 2022 12:44

tibersam added 12 commits March 9, 2022 17:49

fixup! Add test for analysisfunctions used in other test functions

a0b7cdd

fixup! Add test to run complete setup and check output hdf5 file

a7d62d0

fixup! Add tests for python worker with needed result data

1553dd3

fixup! Added requirements.txt for pytest

116932a

fixup! Update buildtester to run test bench

c1f8edb

fixup! Add test for analysisfunctions used in other test functions

5d4d26d

fixup! Add test to run complete setup and check output hdf5 file

cf15318

Add script to test archie and add empty conftest.py

af8b22e

- test.sh calls make in the faultplugin folder and then calls pytest to test archie - conftest.py is a file needed by pytest to include archie into its searchpath. Otherwise will throw import error

fixup! Add tests for python worker with needed result data

862f7d1

fixup! Add script to test archie and add empty conftest.py

7b439dd

fixup! Added requirements.txt for pytest

119e13b

Added test Readme

8470ac7

tibersam requested a review from aewag March 9, 2022 20:55

aewag reviewed Mar 10, 2022

View reviewed changes

fixup! Update buildtester to run test bench

f731890

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add testbench #39

Add testbench #39

tibersam commented Feb 10, 2022

tibersam commented Feb 15, 2022

aewag Feb 15, 2022

tibersam Feb 15, 2022

aewag Feb 15, 2022 •

edited

Loading

tibersam Feb 28, 2022

tibersam Mar 1, 2022

lukasauer left a comment

lukasauer Feb 16, 2022

tibersam Feb 28, 2022

tibersam Mar 1, 2022

aewag Mar 10, 2022

tibersam Mar 10, 2022

lukasauer Feb 16, 2022

lukasauer Feb 16, 2022

tibersam Feb 28, 2022

lukasauer Feb 16, 2022

tibersam commented Feb 28, 2022

tibersam commented Mar 3, 2022

aewag left a comment

aewag Mar 10, 2022

tibersam commented Mar 10, 2022

aewag commented Mar 10, 2022

tibersam commented Mar 28, 2022

lukasauer commented Mar 28, 2022

tibersam commented Jul 22, 2022

lukasauer commented Jul 22, 2022

tibersam commented Feb 12, 2023

aewag commented Feb 14, 2023

		from hdf5logger import hdf5collector


		def compare_hdf5_files():

		from faultclass import python_worker, Fault


		class dummy_args:

Add testbench #39

Are you sure you want to change the base?

Add testbench #39

Conversation

tibersam commented Feb 10, 2022

tibersam commented Feb 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aewag Feb 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukasauer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tibersam commented Feb 28, 2022

tibersam commented Mar 3, 2022

aewag left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tibersam commented Mar 10, 2022

aewag commented Mar 10, 2022

tibersam commented Mar 28, 2022

lukasauer commented Mar 28, 2022

tibersam commented Jul 22, 2022

lukasauer commented Jul 22, 2022

tibersam commented Feb 12, 2023

aewag commented Feb 14, 2023

aewag Feb 15, 2022 •

edited

Loading