Introduce "exec-runnables-recipe" resolver #6032
Conversation
Signed-off-by: Cleber Rosa <[email protected]>
The exec-test and tap resolvers share the same code that verifies the condition of the reference given being an executable file. This would be enough to consolidate them into a single method, but there's going to be a third users of the same check, so there's more reason for that yet. Signed-off-by: Cleber Rosa <[email protected]>
clebergnu
added
the
customer:Passt
clebergnu
force-pushed
the
resolver_exec
branch
from
19f0082
to
458e53d
Compare
This resolver is somewhat of a hybrid between the "exec-test" and the "runnables-recipe" resolvers. It runs an executable, and attempts to read from its STDOUT content that will be treated as runnables-recipe JSON content. If that succeeds, the content will be returned as test resolutions. This is useful for executable tests or test generators that will output the tests dinamically. Signed-off-by: Cleber Rosa <[email protected]>
This adds support for used defined arguments to be passed while running the executables that will generate the runnables recipe JSON content. It gives the opportunity for either calling the executables with a particular option that will output the runnables (instead of other action), or tweaking the type of runnables that will be generated. Signed-off-by: Cleber Rosa <[email protected]>
clebergnu
force-pushed
the
resolver_exec
branch
from
458e53d
to
a882899
Compare
There was a problem hiding this comment.
Hi @clebergnu, I don't have many comments to the code itself, but I think we should discuss the overall behaviour of exec-runnables-recipe resolver.
IIUIC, with your solution we can combine exec-runnables-recipe and exec-test resolvers together, which IMO is not good behaviour, because we could end up in situations where tests are run during resolving phase. I would propose two possible solutions here.
- If
--resolver-run-executablesoption is used all the other resolvers are disabled. This is easy solution, but we will lose the possibility of running multiple test types at once. --resolver-run-executableswon't be a boolean, but it will be list where user can specify executables which should be directly run byexec-runnables-reciperesolver. All the other references will go through normal resolution process. I would prefer this kind of solution.
What do you think, about this problem.
|
|
||
| content, _ = process.communicate() | ||
| try: | ||
| runnables = json.loads(content) |
There was a problem hiding this comment.
This whole part from 258 to 280 is duplication of RunnablesRecipeResolver code. IMO, it would be better to call the RunnablesRecipeResolver or create some Base class for holding the json validation a resolving code.
| "to the executable when executing it as a producer of " | ||
| "runnables-recipe JSON content." | ||
| ) | ||
| settings.register_option( |
There was a problem hiding this comment.
I am not sure if this is the best solution, what would happen when there would be more than one run-executables-resolver, all of them should get the same arguments?
What about have the arguments as part of the reference behind :. Like this:
$ avocado list --resolver-run-executables examples/nrunner/resolvers/exec_runnables_recipe_kind.sh:tap
True. Although I believe the solution we have for that is really the use of different suites. Users should be aware of the behavior, and control, through the use of different suites, if executables are meant to be run during the resolver phase or not.
Right.
I think this is flexible, but adds "special" options and behavior to the overall resolver approach.
Like I said before, I think users should pick the references themselves that should be executed for resolutions, and the ones that should not (that will be executed as tests instead). I'll go ahead and better document the behavior and put a note about how users can not (or should not) combine those two different types of executables in a single suite. |
This resolver is somewhat of a hybrid between the "exec-test" and the "runnables-recipe" resolvers.
It runs an executable, and attempts to read from its STDOUT content that will be treated as runnables-recipe JSON content. If that succeeds, the content will be returned as test resolutions. This is useful for executable tests or test generators that will output the tests dynamically.
Fixes: #6009