Use another plan within a plan

[lukaszachy]

There should be a way to reference plan from another plan to avoid duplication and have single source of truth (or limited sources of truth).

Possible use case in Integration test of several components:
Component A wants to run selected tests of component B to verify they work.
Component B has non trivial setup which needs to run (e.g. prepare/ansible)
Set of tests for Compoment B is selected by its owner (not owner of the component A)

For all of these best would be to have way how to reference plan for integration testing of component B and use it.

Details to think about:

• context inheritance ?
• environment inheritance ?
• replace / extend steps?

One possible way to solve this is teemtee/fmf/issues/99

[pcahyna]

One possible way to solve this is teemtee/fmf/issues/99

I think this is a bad idea. fmf trees represent inheritance, i.e. a "is-a" relationship. You don't want to inherit the leaf plans from the root plan that references them, because the reference is a "has-a" relation - only a link. If anything, you might think of making the top-level plan to inherit from linked plans, but that would mean multiple inheritance, therefore solving conflicts, so you actually don't want this either - that would mean not being able to link conflicting plans from one top-level.

In short, this should be a purely tmt-level feature, not fmf.

[sopos]

agree

The child plans should be treated as namespaces of the parent plan. It is necessary to be able to handle an unlimited number of plans linking.

Example:
A master plan SSG_Tier1 refers to SST1_Tier1 which refers to ComponentX_Tier1 and SST2_Tier1 which refers to ComponentY_Tier1. The ComponentX_Tier1 and ComponentY_Tier1 plans define all the HW requirements, preparations steps, etc.
The result should be one single plan SSG_Tier1 which would contain environments tight to the namespaces SST1_Tier1/ComponentX_Tier1 and SST2_Tier1/ComponentY_Tier1.

This would allow to visualize results per child plans in a foldable / hierarchical way.

[happz]

One real-life use case where we would really appreciate this or a similar feature, it would allow us to share more complex prepare and adjust stanzas between Fedora and other products. For LLVM, we start treating Fedora as a single source of truth, it does make things easier to manage, and we can re-use tests easily. It would be much easier if we could re-use whole plans, i.e. including nicely crafted and tested prepare and adjust, rather than copy & pasting these bits and pieces into our downstream plans: we can discover remote tests from Fedora repo, but as soon as a test needs special preparation, this information is lost to tmt - which sees only its downstream plan - and must be repeated in the said downstream plan.

In our use case, the discovered plans would be executed in the context given to the original plan, i.e. the parent/child (or namespaces) relationship does seem to match #975 (comment). We'd expect the remote plans to run within the context given to the original plan - in our use case, the original, first plan is a mere bootstrap plan, a dummy, its only purpose is to redirect tmt to the actual plan in Fedora repository.

[sopos]

Of course the ability to leave the prepare specifics on the component owner and not having to copy everything is crucial.

Without that the integration testing is almost impossible as the component owner may even not know that someone else is running their tests. If the owner changes the structure the other party user must react on that as well now.

Also we need to be able to keep everything under the explicitly created plan - no automatic plans creation should happen as we need to have predictable plan names e.g. for gating purposes.

[psss]

The remote plan import specification has been merged today in #1145.

Details to think about:

• context inheritance ?
• environment inheritance ?
• replace / extend steps?

As the very first step, it does not allow inheritance or any step extension.

[psss]

The implementation should not be too complicated, proposing for 1.15.

[sopos]

The main point here is if it should be an extension of the remote plan (just an ability to specify more plans under one umbrella) or if it is going to be something more sophisticated with an ability of inheritance / merging of fmf structures.

Some of the use cases could be also achieved using a separate feature plan template which I thinking to file an issue for. I'll add the link later.

[pcahyna]

@sopos even if it is not something more sophisticated, it is not necessarily just an extension of remote plan. The possible difference is: as I understand it, remote plan is currently intended to replace the plan that references it. I.e. if a plan A references a remote B, B essentially replaces A. While for plans within a plan, if B1, B2, B3 are within plan A, they do not replace it, but are nested under A. As you say, it is "more plans under one umbrella", and A is the umbrella, the B n are the more plans under it. But in the case of remote plans, I think there is no umbrella - there is only only one level of hierarchy, i.e. no actual hierarchy.

Or do you want the same behavior for plans within a plan? That is, make A disappear and replace it by the included B1, B2, B3 at the same level where A was, without a hierarchy? I think not, because it contradicts what you wrote earlier: "This would allow to visualize results per child plans in a foldable / hierarchical way."

[mkoncek]

I would like to add in our use case for this feature.
We want to test roughly 100 packages. Therefore we want to add some minimal plan.fmf file to each one of the package which would reference one remote plan. Because we need some shared prepare step.
Our current workaround is to actually prepare the environment from within the tests, which can be referenced.

See an example: https://src.fedoraproject.org/rpms/ant/blob/rawhide/f/plans/javapackages.fmf

[psss]

Thanks for sharing this real-life use case with us. This is exactly what remote plans are intended for. For common preparation we also have a draft ot support for remote ansible playbooks in #1054 which could possibly be useful for your use case as well.

[psss]

@adiosnb, if you're looking into this, the fmf.Tree.node() method might come handy:

import fmf
plan = fmf.Tree.node(dict(url="https://github.com/teemtee/tmt/", name="/plans/features/core"))

It stores the fetched fmf metadata in a cache directory. @lukaszachy, do you see any hitch in using fmf cache for this purpose?

[lukaszachy]

Problem is with non-fmf files plan uses. fmf.Tree.node locks cache dir during reading the tree however after that clears the lock. So at the time fmf.Tree.node returns another process might checked out another commit.

I suspect that we need to copy-in prepare/finish scripts etc.

[psss]

I see, good point. Then it will be better to create a separate git clone for this purpuse. The remote plan can contain discover --how fmf without an url as well, which means we might need that repo for test discovery as well.

[adiosnb]

I see, good point. Then it will be better to create a separate git clone for this purpuse. The remote plan can contain discover --how fmf without an url as well, which means we might need that repo for test discovery as well.

I have a question regarding the tmt execution. I've been trying to reuse internal classes to run tests inside cloned repo. However, the implementation is probably not as good as it should be. Do you think it might be worth trying another tmt process (e.g. os.popen('tmt ...')?