Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Easier introspection of available tests (reading of test manifest) #6009

Closed
clebergnu opened this issue Aug 26, 2024 · 1 comment · Fixed by #6032
Closed

Easier introspection of available tests (reading of test manifest) #6009

clebergnu opened this issue Aug 26, 2024 · 1 comment · Fixed by #6032
Assignees
Labels
customer:Passt Requirements/issues raised by the Passt project enhancement

Comments

@clebergnu
Copy link
Contributor

Is your feature request related to a problem? Please describe.
The exeter framework integrates with Avocado by producing output that the "runnables-recipe" resolver can understand (JSON representations of Runnables).

But, that requires an intermediate step, that is, the executable test files have to be executed in a special way (currently with --avocado) so that JSON is produced, and can then be consumed by Avocado's "runnables-recipe" resolver.

Describe the solution you'd like
It'd be convenient to not require the intermediate step (although it should still be an option). The most straightforward solution is to have a new resolver configuration for the format of the command line that will produce the manifest of tests.

A configuration such as:

[resolver.exec_test]
manifest_command = {uri} --avocado

Describe alternatives you've considered
The solution described above has the drawback of requiring the execution of binaries/scripts that may not be completely trusted.

The resolver plugins that ship with Avocado do not execute any untrusted code (for instance an avocado list $(path-to-test-repo) won't execute any of the tests found. Avocado's safeloader does a static analysis for Python code, which could be extended. For shell based tests, it could also read some hints from the code. For binaries, maybe reading data added to an ELF section?

@clebergnu clebergnu added enhancement customer:Passt Requirements/issues raised by the Passt project labels Aug 26, 2024
@clebergnu clebergnu added this to the 108 - Codename TBD milestone Aug 26, 2024
@mr-avocado mr-avocado bot moved this to Triage in Default project Aug 26, 2024
@avocado-framework avocado-framework deleted a comment Aug 26, 2024
@clebergnu clebergnu self-assigned this Aug 26, 2024
@clebergnu clebergnu moved this from Triage to Long Term (Next Q) Backlog in Default project Aug 26, 2024
@clebergnu clebergnu moved this from Long Term (Next Q) Backlog to Short Term (Current Q) Backlog in Default project Aug 26, 2024
@github-project-automation github-project-automation bot moved this from In progress to Done 108 in Default project Sep 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
customer:Passt Requirements/issues raised by the Passt project enhancement
Projects
Status: Done 108
Development

Successfully merging a pull request may close this issue.

2 participants
@clebergnu and others