twister help/document revisit

[hakehuang]

twister is getting larger and larger, can we start to do tidy the twister document and help command line:

1. all the supported features shall be listed in help
2. all paramenter shall have an example
3. all parameter combination and dependency shall be well documented
4. for given command lint all parameters shall be print out and easy to be searched(grep)
5. need find a way that parameter is organized in one place. e.g the way with zephyr\boards\arc\nsim\support

[abrodkin]

It's also good to have a simple way to:

1. Print default state of all options
2. Dump entire set of options with its values being effectively used including those overridden from default with user's settings.

As a good example might be GCC printing its built-in defines taking into account user-set options:

64-bit system

$ gcc -dM -E - < /dev/null | grep LP64
#define __LP64__ 1
#define _LP64 1

32-bit system (note -m32 added)

$ gcc -dM -E -m32 - < /dev/null | grep LP64

[hakehuang]

it is to provide a clear list on all what are set for give commands.
take zephyr\boards\arc\nsim\support as examples

[wangnuannuan]

https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/pylib/twister/twisterlib.py#L2145 here the twister reads the CMakeCache.txt, we can extract variables from it

[hakehuang]

and at the end of parse_arguments, call print(vars(parser)), then we can get all option settings for twister

[nashif]

this is too meta, if you find issues with documentation please file a specific bug or submit a PR to fix it, if you have new ideas for how things can be done better, same thing. The tool itself documents all available options with --help, there is also lots of docs in doc/guides/test/twister.rst.

[hakehuang]

"-B", "--subset",
help="Only run a subset of the tests, 1/4 for running the first 25%%, "
"3/5 means run the 3rd fifth of the total. "
"This option is useful when running a large number of tests on "
"different hosts to speed up execution time.")

this option can only allow first 1/4 case to run, then how to run the rest 3/4? this option is usful when we have multiply boards to run.

Then we can use -B 1/4 ; -B 2/4; -B 3/4; -B 4/4 run them on different boards or buildkite agents task

[hakehuang]

--unrecognized-section-test

what is this options for? detect unrecognized binary sections?

[hakehuang]

parser.add_argument(
"--filter", choices=['buildable', 'runnable'],
default='buildable',
help="""Filter tests to be built and executed. By default everything is
built and if a test is runnable (emulation or a connected device), it
is run. This option allows for example to only build tests that can
actually be run. Runnable is a subset of buildable.""")

is this option in use? if set to runnable, seems all cases will run

[PerMac]

Just to clarify "--subset": It is heavily used in CIs, where execution can be parallelized between multiple independent agents. The idea is to divide a list of tests that will be executed into X parts and each of them is executed by calling twister --subset i/X X times, for i in range(X). If you first run twister --subset 1/4 to get the rest it would require calling the remaining 3 parts with 3 commands. I don't see a point to script an option for splitting the execution into uneven parts (like 25% first, and 75% later). I think this option doesn't help with "usful when we have multiply boards to run."

[PerMac]

Runnable will filter out all tests which cannot be executed, e.g build-only tests. The other one (buildable) looks like being useless as it equals to not using any filter, all tests are buildable. It is just a false dichotomy, also in the code. In fact, runnable is a subset of buildable. These are not disjunctive groups. I believe the "buildable" can be removed from the code with very little effort.