tmt vs fmf

[abitrolly]

I've seen fmf mention in docs more than often, but with no explanation what does it mean. Looks like fmf is the test format, and tmt is test runner for it. So, for example, the convention to put test in plans/ comes from fmf, and not from tmt. Am I right?

[FrNecas]

Your understanding is quite close though not completely precise. fmf is just a data format built on top of YAML (with extra features, e.g. inheritance, added). fmf does not know anything about tests/plans/stories, it's all just data. tmt is what adds the semantics to that data based on specification.

Moreover, plans do not necessarily even have to be placed under plans/. The metadata "type" (test/plan/story) is identified based on the required keys. If a .fmf file contains the key execute: then it is considered a plan. If it contains test:, it is considered to be a test.

Does this explanation make it a bit more clear? Do you have any suggestion how to improve the docs so that it is not confusing to you?

[happz]

There's also this piece: https://tmt.readthedocs.io/en/stable/questions.html#what-is-the-difference-between-fmf-and-tmt

Maybe it needs to be promoted more, how about a link right at the end of https://tmt.readthedocs.io/en/stable/overview.html#description? Both TMT and FMF are described there, "Do you want to know more? Follow the link". It helped our little team to grasp the difference between tmt and fmf.

[psss]

There's also this piece: https://tmt.readthedocs.io/en/stable/questions.html#what-is-the-difference-between-fmf-and-tmt

Yes, this should be the best place for clarification of the difference. @abitrolly, if the description is not clear there, please help us to make it more understandable.

Maybe it needs to be promoted more, how about a link right at the end of https://tmt.readthedocs.io/en/stable/overview.html#description?

I'd say the description already tries to summarize what tmt does and at the end also mentions fmf which a brief description, so this sounds like a duplication. But if it would help, let's a link there.

[abitrolly]

This description of fmf doesn't speak tests to me.

The Flexible Metadata Format fmf is used to store data in both human and machine readable way close to the source code. Thanks to inheritance and elasticity metadata are organized in the structure efficiently, preventing unnecessary duplication.

Is it a convention for filesystem layout? Or just an extension to YAML? How does it compare to YAML by example?

Looks like "plain text database" or some hierarchical data serialization format. I could probably better understand it in comparison with popular formats used in CI, like Docker (for inheritance), and files from GitLab Actions, Travis, Ansible, Kubernetes. And while I can imagine YAML inheritance to some degree (better with examples), I can not extract the meaning from the phrase "elasticity metadata".

Also, if it just a format, why tmt creates .fmf dir and not .tmt dir in fmf format?

Or putting it another way, I can not really understand what a format is about without figuring out why it was created in the first place? So starting with a problem why plain YAML is not sufficient for tests would be a good start.

[lukaszachy]

And to complete the answer - TMT recognizes its objects (Plan, Test and Story) based on attribute they contain - 'test' makes Test, 'execute' makes Plan and 'story' makes Story.

There are no plans to abandon fmf from tmt, so currently directory (or repo) can hold tmt metadata only if it is also a fmf repository.
But it doesn't need to be the other way - you are free to use fmf for any project you want - no need to bind yourself with tmt and its test/story/plan.

[idorax]

Or putting it another way, I can not really understand what a format is about without figuring out why it was created in the first place? So starting with a problem why plain YAML is not sufficient for tests would be a good start.

Right here I'd like to ask why plain YAML is not sufficient for (managing and executing) tests too.

To end users, it looks more reasonable to write plain YAML although they should comply with the metadata specification (i.e. FMF) used by tmt. In other words，for example, if FOO.fmf is named as FOO.yaml or FOO.yml, end users can realize that they should abide by the syntax of YAML at first sight, which may be more friendly to end users.

[lukaszachy]

Right here I'd like to ask why plain YAML is not sufficient for (managing and executing) tests too.

How does YAML help with

1. Define attribute and its value on single place and inherit it to all children nodes
2. Extend attribute value with another value

To illustrate:

• I want to be "contact" for any test in the repo, however I don't want to remember adding that value to each metadata file and also to have single place to change it once someone else takes over (or adds themselves as secondary owner)
• All tests need to require component 'foo' but some might have additional requires. Also foo might be called differently on rhel-6 than in fedora-rawhide. I need to have single place to change it.
• There are several instances of same test, all I need to change is single environment value.

For few tests is is ok to write everything, but having 'inheritance' (using parent's value in current node) and 'merging' to slightly tweak attributes is necessary for hundreds of tests.

[happz]

Right here I'd like to ask why plain YAML is not sufficient for (managing and executing) tests too.

To end users, it looks more reasonable to write plain YAML although they should comply with the metadata specification (i.e. FMF) used by tmt. In other words，for example, if FOO.fmf is named as FOO.yaml or FOO.yml, end users can realize that they should abide by the syntax of YAML at first sight, which may be more friendly to end users.

It is sufficient. Just like an assembly language has all the functionality and power needed to develop a full-fledged Firefox clone. Can it be done? Yes. Would it be a pleasant experience? I don't think so. Higher-level languages add extra tools and concepts to make development easier, and so does FMF when it comes to organizing test metadata. FMF adds features plain YAML does not offer, aiming at more or less complex organizations of trees of keys and values. That's very handy for developing and maintaining a hierarchy of scenarios, plans, and tests, making it more suitable than plain YAML.

With plain YAML, I suspect TMT would have to implement many of FMF's features anyway, for example, inheritance and virtual tests are extremely helpful in my projects - so we would end up with files with the .yaml suffix, "Inheritance", "Virtual" and "Adjust" features in TMT's toolbox, and no FMF library. That would rob people outside of TMT of the benefits of inheritance between subtrees and adjust for runtime modifications, for example - features that might be very handy when organizing a public library or genealogy tree. With FMF as a stand-alone library/tool, all those features are available to others, not tied to TMT.

In a way, it is similar to a C preprocessor, or templating engines like Jinja: an extra layer of functionality between raw files and the actual tool, providing useful features without bundling those features into the tool itself.

[abitrolly]

https://fmf.readthedocs.io/en/stable/features.html should state what fmf format can do.

@lukaszachy that's very nice. Although I still need clarification to understand what's going on here - teemtee/fmf#173 (comment)