CHIP Mbed functional testing

[ATmobica]

Guys, I think that the development of functional tests is an open matter and will require discussion and making some decisions.
For this reason, I am starting this discussion and I encourage you to be active.

The starting point can be a Wiki page Mbed CHIP functional testing.
It described the result of my work on how to approach this issue, but as I mentioned, the case is open.

I am waiting for your opinions, comments and suggestions.

[pan-]

Thanks @ATmobica or putting up the document; the steps make sense.

I believe the build step should be independent from the test itself. As you mentioned, binaries can be created locally or builded in the cloud and then pulled from an artifact repository.

However, it is unclear for me where the test fixture setup (such as flashing the board) goes. I'd prefer if it was done by the test itself so we can run all the tests from a single command. For other type of fixture setup such as network setup, I don't know where the most convenient place is to setup them.

[ATmobica]

1. Yes, I also agree. Now, we can use available scripts for building apps locally or investigate the options (maybe create some scripts) for downloading artifacts from a repository. But sure, It will be a separate step for functional testing, triggered by CI for example.
2. Sure, I am going to use the pytest arguments for that. The same approach as BLE example. Single command with the necessary parameters