plugin metadata loader with pytest and bash example #136

jscotka · 2021-04-27T13:59:25Z

It is WIP, so please no stylish comments, I expect more fundamental comments of this architecture design

example of plufing for FMF to discover pytest tests and bash.

cool features:

able to add additional metadata directly via python decorators
mixing of FMF and plugin metadata together
transformation of pytest marks to FMF metadata
- mark.skip transformed to enabled: False
- custom marks are transformed to tags
Added .fmf/config sa way how to configure plugins

output after cd tests/tests_plugin/:

$ PLUGINS=../../fmf/plugins/pytest.py,../../fmf/plugins/bash.py  fmf show

/pure_fmf
author: Jan Scotka <[email protected]>
summary: Pure FMF test case
test: ./runtest.sh

/runtest
author: Jan Scotka <[email protected]>
summary:  Basic test case: 1
tag:  tier1
test: ./runtest.sh

/test_basic/TestCls/test
author: Jan Scotka <[email protected]>
summary: test_basic.py TestCls test
test: python3 -m pytest -m '' -v test_basic.py::TestCls::test

/test_basic/test_fail
author: Jan Scotka <[email protected]>
summary: test_basic.py test_fail
test: python3 -m pytest -m '' -v test_basic.py::test_fail

/test_basic/test_parametrize[a]
author: Jan Scotka <[email protected]>
summary: test_basic.py test_parametrize[a]
test: python3 -m pytest -m '' -v 'test_basic.py::test_parametrize[a]'

/test_basic/test_parametrize[b]
author: Jan Scotka <[email protected]>
summary: test_basic.py test_parametrize[a]
test: python3 -m pytest -m '' -v 'test_basic.py::test_parametrize[b]'

/test_basic/test_parametrize[c]
author: Jan Scotka <[email protected]>
summary: test_basic.py test_parametrize[a]
test: python3 -m pytest -m '' -v 'test_basic.py::test_parametrize[c]'

/test_basic/test_pass
author: Jan Scotka <[email protected]>
summary: This is basic testcase
tag: Tier1
test: python3 -m pytest -m '' -v test_basic.py::test_pass

/test_basic/test_skip
author: Jan Scotka <[email protected]>
enabled: False
summary: test_basic.py test_skip
test: python3 -m pytest -m '' -v test_basic.py::test_skip

TODOs

probably extension is not enoung, could be regexp instead of ".py" could be e.g. ".*tests/.*\py"
how to behave if no data found, maybe return None and it will behave as tree node creating is skipped. (None != {} in case {} create the Node)
what does with modification? not easy to write data back to non fmf files, e.g. pytest tests. e.g. adding extra-nitrate key
- solution could be add another required method for plugin to write data back for each plugin. Could be harder to parse python code and e.g. add decorator to method. if not exists (but some magic with inspect and looking into function definition source code lines and put there another line with decorator may work well and not so hard to implement inspect.getfile(item.function), inspect.getsourcelines(item.function)[1] and insert new line into the file if not found inside other decorators already in: [line for line in inspect.getsourcelines(item.function)[0] if re.match(r"\s*@FMF",line)])
- Or store owerwritten data to separate FMF file using merging+scattering feature, could be easy. (But sorting and file priorities could cause issues here)

EDIT

Implemented how to modify items for plugin (pytest plugin example for new items)

Focus items for review:

generic config file inside .fmf dir for fmf (now used just for plugin loading)
changed sorting of files order `(main.fmf, ..., somename.*, somename.fmf (fmf as last in case of same names), ... )
modification overriding to FMF files when plugin does not support write method
loading plugins by file path or via python way via modules like a.b.c
using env variable to load (override plugins) PLUGINS env var (could be renamed to whatever)
where to store plugins (some could be part of FMF project) or strictly as separated in project? (then TMT needs to have some possibility to install some pip module before runnig discover on target system or part of target repo)
what about extension of plugin by another config (originally I have way in my pytest plugin to use own config, to be able to tweak output format, e.g. say that every TMT.tag will be translaated to tag+) e.g. I'm configuring test: item via this in various ways (another for downstream and upstream)

lgtm-com · 2021-04-27T14:07:04Z

This pull request introduces 2 alerts when merging 6f0291c into c08e904 - view on LGTM.com

new alerts:

1 for NotImplemented is not an Exception
1 for Nested loops with same variable

lgtm-com · 2021-04-28T07:00:03Z

This pull request introduces 2 alerts when merging 71f3e44 into c08e904 - view on LGTM.com

new alerts:

1 for NotImplemented is not an Exception
1 for Nested loops with same variable

lgtm-com · 2021-04-28T07:23:32Z

This pull request introduces 2 alerts when merging 521c0ba into c08e904 - view on LGTM.com

new alerts:

1 for NotImplemented is not an Exception
1 for Nested loops with same variable

jscotka · 2021-04-28T09:16:57Z

added to hacking session as topic.

lgtm-com · 2021-04-28T09:52:26Z

This pull request introduces 2 alerts when merging 9aa90b8 into c08e904 - view on LGTM.com

new alerts:

1 for NotImplemented is not an Exception
1 for Nested loops with same variable

lgtm-com · 2021-04-29T12:38:16Z

This pull request introduces 3 alerts when merging 94b77e4 into c08e904 - view on LGTM.com

new alerts:

2 for NotImplemented is not an Exception
1 for Nested loops with same variable

lgtm-com · 2021-05-24T12:27:23Z

This pull request introduces 4 alerts when merging 5ae40a7 into 6ffce89 - view on LGTM.com

new alerts:

3 for Unused import
1 for Incomplete ordering

lgtm-com · 2021-05-25T13:11:25Z

This pull request introduces 2 alerts and fixes 1 when merging 4734429 into 6ffce89 - view on LGTM.com

new alerts:

1 for Unused import
1 for Incomplete ordering

fixed alerts:

1 for Unused import

lgtm-com · 2021-05-25T14:05:58Z

This pull request fixes 1 alert when merging e5912aa into 6ffce89 - view on LGTM.com

fixed alerts:

1 for Unused import

lgtm-com · 2021-05-25T14:17:24Z

This pull request fixes 1 alert when merging 244c828 into 6ffce89 - view on LGTM.com

fixed alerts:

1 for Unused import

Trasnfer code from fmf_metadata directly to FMF add there important improvement of file nadling Use cnfig inside .fmf dir to load plugins adjust tests and skip data files

lgtm-com · 2021-05-26T07:45:33Z

This pull request fixes 1 alert when merging 3d1330b into 6ffce89 - view on LGTM.com

fixed alerts:

1 for Unused import

lgtm-com · 2021-05-26T08:17:01Z

This pull request fixes 1 alert when merging e913936 into 6ffce89 - view on LGTM.com

fixed alerts:

1 for Unused import

lukaszachy · 2021-05-26T09:37:26Z

fmf/plugin_loader.py

+import inspect
+import os
+import re
+from functools import lru_cache


fmf should support also python2.7 where this is not available.

lukaszachy

Few comments for now, I need to understand the code for the rest :)

Also an exception when PLUGINS contains broken path could be better.

lukaszachy · 2021-05-26T09:39:23Z

fmf/plugin_loader.py

+@lru_cache(maxsize=None)
+def enabled_plugins(*plugins):
+    plugins = os.getenv(PLUGIN_ENV).split(
+        ",") if os.getenv(PLUGIN_ENV) else plugins


I believe TMT splits on space, let's be consistent.

Yes, I can change it to whatever more consistent solution, I also thought to use : as std path separator, or but is worse thanks to it is more common to have spaces in paths, than : or , chars

lukaszachy · 2021-05-26T09:40:08Z

fmf/constants.py

+MAIN = "main" + SUFFIX
+IGNORED_DIRECTORIES = ['/dev', '/proc', '/sys']
+# comma separated list for plugin env var
+PLUGIN_ENV = "PLUGINS"


FMF_PLUGINS to mitigate name conflict

Yes, sounds reasonable.

lukaszachy · 2021-05-26T09:45:25Z

fmf/plugins/pytest/constants.py

+        "test": """
+cls_str = ("::" + str(cls.name)) if cls.name else ""
+escaped = shlex.quote(filename + cls_str + "::" + test.name)
+f"python3 -m pytest -m '' -v {escaped}" """


I'm not sure about pytest as a require :/ As it is now it is a 'hidden' require - you need to remember. But on the other hand it leaves up to user whether rpm or pip install is better.

yes, it is now hidden, you are rigt, and theoretically it is also why I've let it configurable, not hardcoded. in my project fmf_metadata I have own configs for upstream and dowstream data generation. and in case you have just unittests then you an execute it directly via python unittest executor, or via some nose so that pytest is just default way.

lukaszachy · 2021-05-26T09:54:13Z

fmf/plugins/pytest/plugin.py

+            if key == "skip":
+                TMT.enabled(False)(func)
+            elif key == "skipif":
+                # add skipif as tag as well (possible to use adjust, but


Could skipif be evaluated at this point? Imho mark.skipif(False) should end up as enabled: False and not tag: SKIP False

It is not possible, because evaluation will be done in sense of TMT on guest, although TMT discovery is done on host system.
This skipIf is little bit problem to incorporate somehow easily into FMF metadata, but this tag was the simplest solution and probably the easy to understand.
Probably the beast one could be to trasfer it to something like:

adjust: when: interpreter: python code: False enabled: False

and execute adjust condition on guest system. But nothing similar is supported right now.

jscotka · 2021-05-26T13:38:05Z

/packit test

happz · 2023-06-27T09:47:34Z

fmf/plugins/pytest/tmt_semantic.py

+        return cls._set_fn(name)
+
+
+class TMT(metaclass=__TMTMeta):


One thing I can say I disagree with right away: this approach taints fmf with a custom code to make a single library user happy and makes fmf do things that are IMO well beyond its scope.

First, it makes adding and changing tmt specifications harder, adding a new place that needs to be updated when specification changes. But, more importantly: I don't think it's up to fmf to inspect the input or validate it in any way.

Is fmf validating or converting tmt keys when reading them from a genuine fmf file? I don't think that's the case, and it should not happen here either. If I put tag: false into my main.fmf of a test, fmf library will not complain, it will not touch the value, it will not check against a tmt specification schema, it will read it, put it into a fmf tree, and it will deliver the key and its value to tmt. End of story. It's tmt who's responsible for validating the input and reporting its type is invalid and unsupported.

Would it be possible to make the pytest plugin as "dumb" as the actual fmf processing, i.e. treating the input values as opaque blobs of key & value bits, and stacking them into a final fmf tree? I bet it would be possible with some clever Python magic, either have an open decorator accepting all kinds of kwargs or some dynamic decorators - but I strongly believe the openness here is a good thing, not a bug - fmf should not care about key names or value types, it does not care about them when reading them from *.fmf files.

# Probably the simples, but less readable: @fmf( tag=['foo', 'bar'], adjust=[ {...} ...) # Or maybe allow YAML-in-a-string and parse it as if read from an fmf file? @fmf( """ author: [email protected] tag: - foo - bar adjust: ... """) # Or with some __getattr__ magic, this could be possible by creating dynamic decorators aware of their name, i.e. the key: @fmf.author('[email protected]') @fmf.tag(['foo', 'bar']) ...

tmt is and should remain the sole owner of the semantics. Just like any other tool using fmf to read data. It's not up to fmf to judge, nobody asked fmf's opinion on keys from *.fmf files, why would it matter here?

Should a competing test case management tool appear, build on top of fmf too, but with a different set of keys, would fmf learn its semantics as well? And when another two appear, these brand new and cool mmt and ttm tools, will fmf again learn about their semantics? I don't think that's a good road to take :)

you are right, semantics, could be part of another tool. I've Implemented it as one module, because in this case it is all in one solution. I've wanted to show complete solution. My idea was that in FMF there will be just support for plugin, and implementation of plugins will be separate project or part of TMT. But plugin solution alone is hard to test itself. so that complet solution also proves how it works and it also show usecase in TMT or whatever else. I'm familiar to split it. But we have to agreed on the solution for plugin. and improve this part to have it testable as much as possible and when we found it is good enough then I'll split it.

Syntax this is exactly what is supported there it is using internal method getattibute to handle caller name and then base on semantics it is decided whats wrong and whats implemented and well structured:

@fmf.author('[email protected]')

see code https://github.com/teemtee/fmf/pull/136/files#diff-90dd8e8769d61574c4e3b75280a1a7d8425336e9ab8ec30b4ef71543d44f9120R72

jscotka mentioned this pull request Apr 27, 2021

Support exploring metadata from Python source code #103

Open

jscotka force-pushed the py_plugin branch 3 times, most recently from 4b87247 to 71f3e44 Compare April 28, 2021 06:52

jscotka mentioned this pull request Apr 28, 2021

Adding new framework (plugin?) teemtee/tmt#740

Open

jscotka force-pushed the py_plugin branch from 71f3e44 to 521c0ba Compare April 28, 2021 07:15

jscotka requested review from psss and lukaszachy April 28, 2021 07:49

jscotka added discuss enhancement New feature or request labels Apr 28, 2021

psss marked this pull request as draft April 28, 2021 10:19

jscotka changed the title ~~pytest plugin metadata loader~~ plugin metadata loader with pytest and bash example Apr 28, 2021

jscotka force-pushed the py_plugin branch 3 times, most recently from c0f184a to 46f442a Compare May 4, 2021 10:27

jscotka force-pushed the py_plugin branch from 5a21bc2 to 5ae40a7 Compare May 24, 2021 12:18

jscotka force-pushed the py_plugin branch 4 times, most recently from 6383667 to 42b4965 Compare May 25, 2021 13:45

jscotka force-pushed the py_plugin branch from 649082b to 3497318 Compare May 26, 2021 07:36

pytest plugin metadata loader

3d1330b

Trasnfer code from fmf_metadata directly to FMF add there important improvement of file nadling Use cnfig inside .fmf dir to load plugins adjust tests and skip data files

jscotka force-pushed the py_plugin branch from 3497318 to 3d1330b Compare May 26, 2021 07:38

replace f-strings bny format due to python2 compatibility

e913936

jscotka marked this pull request as ready for review May 26, 2021 09:32

lukaszachy reviewed May 26, 2021

View reviewed changes

jscotka mentioned this pull request May 6, 2022

implement programatic expansion of metadata - macros #152

Open

happz reviewed Jun 27, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

plugin metadata loader with pytest and bash example #136

plugin metadata loader with pytest and bash example #136

jscotka commented Apr 27, 2021 •

edited

Loading

lgtm-com bot commented Apr 27, 2021

lgtm-com bot commented Apr 28, 2021

lgtm-com bot commented Apr 28, 2021

jscotka commented Apr 28, 2021

lgtm-com bot commented Apr 28, 2021

lgtm-com bot commented Apr 29, 2021

lgtm-com bot commented May 24, 2021

lgtm-com bot commented May 25, 2021

lgtm-com bot commented May 25, 2021

lgtm-com bot commented May 25, 2021

lgtm-com bot commented May 26, 2021

lgtm-com bot commented May 26, 2021

lukaszachy May 26, 2021

lukaszachy left a comment

lukaszachy May 26, 2021

jscotka May 26, 2021

lukaszachy May 26, 2021

jscotka May 26, 2021

lukaszachy May 26, 2021

jscotka May 26, 2021

lukaszachy May 26, 2021

jscotka May 26, 2021

jscotka commented May 26, 2021

happz Jun 27, 2023

jscotka Jul 3, 2023 •

edited

Loading

plugin metadata loader with pytest and bash example #136

Are you sure you want to change the base?

plugin metadata loader with pytest and bash example #136

Conversation

jscotka commented Apr 27, 2021 • edited Loading

cool features:

TODOs

EDIT

Focus items for review:

lgtm-com bot commented Apr 27, 2021

lgtm-com bot commented Apr 28, 2021

lgtm-com bot commented Apr 28, 2021

jscotka commented Apr 28, 2021

lgtm-com bot commented Apr 28, 2021

lgtm-com bot commented Apr 29, 2021

lgtm-com bot commented May 24, 2021

lgtm-com bot commented May 25, 2021

lgtm-com bot commented May 25, 2021

lgtm-com bot commented May 25, 2021

lgtm-com bot commented May 26, 2021

lgtm-com bot commented May 26, 2021

Choose a reason for hiding this comment

lukaszachy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jscotka commented May 26, 2021

Choose a reason for hiding this comment

jscotka Jul 3, 2023 • edited Loading

Choose a reason for hiding this comment

jscotka commented Apr 27, 2021 •

edited

Loading

jscotka Jul 3, 2023 •

edited

Loading