Implement import and export for manual tests

[psss]

Based on the outlined specification of manual test data let's implement import/export of manual test cases from/to nitrate. In addition to the attributes defined for the new manual attribute, the test key should contain special value manual. For now, manual test cases should be skipped by default when executing tests. In the future we can add a support for semi-automated execution.

[psss]

@hegerj, interested to look into this?

[kkaarreell]

I would propose to consider storing test phases (setup, ...) in a separate files in git and reference these file instead of describing the step in fmf file directly. I can imagine that some test could be large, also including code snippets etc. and describing it in the fmf file in yaml could be difficult for the user. Moreover, such approach would easy test case export from TCMS/NITRATE.

[hegerj]

While I agree about fmf files being cluttered with these new data I am also not sure extra files are the best solution. It would mean user couldn't just open on file with all metadata which would also worsened user friendliness.

I don't understand what you mean with that last point @kkaarreell .

[kkaarreell]

Test steps are not metadata, such as commands inside runtest.sh are not metadata.

Regarding the last point, Nitrate/TCMS TCs consist of "Setup", "Actions", "Breakdown", "Expected result". I am proposing storing fields content in separate text files and only referring those filenames from setup/step/expect/cleanup fmf sections - well, there is not 1-1 mapping so the export would have to adjust them a bit.

[psss]

Well, one of the main points of moving to fmf was to have all the test metadata at one place. But @kkaarreell is right that (in one point of view) manual test instructions are not test metadata. On the other hand in a test case management system steps would be just one of the test case fields.

I can imagine we could extend the spec in the future to allow referencing external files as well. It could look like this:

manual: test.yaml

For now I'd suggest we keep it simple, everything in a single fmf file to move forward with the implementation and extend the functionality in the future.

Thinking about it a bit more, perhaps this could be an fmf feature, basically something like explicit include. @lzachar, @hegerj, what do you think?

[kkaarreell]

On the other hand in a test case management system steps would be just one of the test case fields.

Well, that would be highly tool dependent, IIRC in Polarion creating a test case could be more complex (multiple steps), definitely not one field. Even in Nitrate it is not one field but four. Similarly, github/gitlab lets you to edit the code in WebUI and you won't call it test metadata. So no, manual test steps are not test metadata, this is the test itself.

Anyway, my point was not having manual instructions in yaml format as you don't really know what is the format of original data in Nitrate (well, it is some rich text but people tend to mix instructions and code snippets).

[thrix]

@kkaarreell

So no, manual test steps are not test metadata, this is the test itself.

Even with script: field we have not just test metadata in the file, but the test also :)

[kkaarreell]

Even with script: field we have not just test metadata in the file, but the test also :)

I guess that attribute would be used only in very small portion of fmf tests. The above still stands (in general).

[hegerj]

@lukaszachy @Athwale What do you think about this issue?

[Athwale]

I looked at a couple of random manual test cases in nitrate.
Some of them do not have any written steps or it is not much text, a few lines or single words. They are manual because the script needs user input or special conditions to run.
Some of them have all the text in just one phase.
Some have 5-10 lines in each phase.
If you separate the phases into files you will end up with files containing a few lines or even just one word.
Obviously I do not know how all subsystems use the manual cases, how they write them and how common are the styles.
I suggest filtering out manual test cases from TCMS/Polarion/any other system and going through a sample of a some for each sst that is going to use fmf to get an idea what is most common and what would be the best way to convert it.
One more option then could be to have one file that would contain all the phases with some headers.

[lukaszachy]

First - How user friendly should it be to use manual tests outside of the tmt command?
With 'write once read many times' in mind and that the primary target of manual cases is human - I think we should support mark up language for describing steps.

What would you say if

• test: manual.md is used as required attribute value to mark manual test
• manual.md is in Markdown format
• skeleton of the manual.md is defined together with mapping for tmt export

We will loose 'elasticity' of manual steps, but using that is a 'vendor lock-in' to fmf.
But on the other hand:

• Description of manual cases is outside of the metadata (easier to read, review PRs..)
• Markdown can be transformed into whatever format target case management tool uses
• Skeleton will help user to write good steps

[kkaarreell]

I second that. Honestly, short manual tests are in many cases poorly described manual tests. I like the markdown idea, though it raises a question what to do if the document syntax won't be correct. Eventually, with defined skeleton you may still be able to parse individual steps and serve them to the user when running the test from the console (e.g. by converting md snippet to text using pandoc or printing md directly). I think I would actually support both in the end, plain text and markdown.

[psss]

Thanks for the idea. Sounds interesting at the first sight. But when I was thinking about it a bit more here are some thoughts which vote against it:

• A clear markdown structure will have to be defined to be able to parse it back into individual steps, with fmf and yaml we get all of this for free, here we would have to implement a special parser and writer
• We loose the ability to inherit common/shared data, for example setup instructions, see the share.fmf example of how this could prevent unnecessary duplication
• Skeleton will help user to write good steps: We get the same with fmf skeleton or not?
• Description of manual cases is outside of the metadata: I believe the fmf "import" functionality would provide the same benefit
• So far all the config is yaml only, this introduces yet another format use has to learn

Not that I would be strongly against the proposal, but so far I do not see advantage here.

[lukaszachy]

Thanks for the idea. Sounds interesting at the first sight. But when I was thinking about it a bit more here are some thoughts which vote against it:

* A clear markdown structure will have to be defined to be able to parse it back into individual steps, with `fmf` and `yaml` we get all of this for free, here we would have to implement a special parser and writer

Please do not place ease of implementation before user experience. If you have longer description of manual case, would you really prefer to read it as an unformatted string in yaml?

Computers are for people, not the other way.

* We loose the ability to inherit common/shared data, for example setup instructions, see the [share.fmf](https://github.com/psss/tmt/blob/master/examples/manual/share.fmf) example of how this could prevent unnecessary duplication

I don't think this example, which from my PoV not easily readable without being processed by fmf is worth locking user to fmf and yaml.

* Skeleton will help user to write good steps: We get the same with `fmf` skeleton or not?

We will. But my point is readability.

* Description of manual cases is outside of the metadata: I believe the fmf "import" functionality would provide the same benefit

Import is one time event. From there on we will be creating new cases under tmt.

[psss]

A clear markdown structure will have to be defined to be able to parse it back into individual steps, with fmf and yaml we get all of this for free, here we would have to implement a special parser and writer

Please do not place ease of implementation before user experience. If you have longer description of manual case, would you really prefer to read it as an unformatted string in yaml?

Computers are for people, not the other way.

I'm not advocating against user experience. Perhaps just prefer plain text with structure which usually covers 90% of my needs. But I understand there can be more complex manual test scenarios.

On the other hand, I think tmt is not supposed to be the ultimate test case management solution, so lightweight manual test case support is just fine with me.

We loose the ability to inherit common/shared data, for example setup instructions, see the share.fmf example of how this could prevent unnecessary duplication

I don't think this example, which from my PoV not easily readable without being processed by fmf is worth locking user to fmf and yaml.

To get the whole test data tmt test show is all you need. And of course, you don't have to use inheritance, you can still freely duplicate the same information if you will.

Skeleton will help user to write good steps: We get the same with fmf skeleton or not?

We will. But my point is readability.

This is probably a matter of taste. But I see your point. Creating new manual test cases using tmt test create --template manual would help in a similar way.

But the main point: For infra services this seems to be a blocker for mass-migration of their tests to git. I would like to find a solution soon. So let's move forward.

Could you please prepare a pull request with updated specification and examples (see #198) showing how this would work with the external markdown document?

[kkaarreell]

I would repeat myself. Manual test cases often contain snippets of code, configuration etc. I can't imagine storing such data in yaml (esp. if the snippet would contain also yaml) and preserving ease of use and readability.

[psss]

We've agreed with @lukaszachy that he will prepare updated specification and examples using referenced external markdown document. Let's continue when we have something tangible.

[kkaarreell]

I know. I just wanted to support his arguments. :-)

[lukaszachy]

Let me comment why 'manual' attribute is kept but changed to boolean.

We have required attribute test and it makes sense - if fmf structure has this attribute is is a test.
In case of automated test it points to the test script ("automation, execute this script")
Originally we would have manual(.md) as script name for manual cases ("human, execute this document").
But having dedicated value doesn't feel right for me - I need to remember which value it is, filtering is done by test:manual.md which isn't clear on the first look.

So I propose to keep manual attribute and let test point to markdown document (yes, any name it needs to be markdown (for now))

Filtering can be done by (filter: manual:false) which imho speaks for itself.

[lukaszachy]

PR #247 updated, comments more than welcome.
Should it work for you, @kkaarreell?

[kkaarreell]

I have added once comment. Otherwise looks good to me. Btw, how do you propose to implement export of manual TCMS TCs to the markdown format?

[lukaszachy]

I'm afraid that due to the free form nature of TCMS it has to be '1. import, 2. edit, 3. export'.
We will be idempotent once it is in MarkDown document for export. Might be not possible to keep it that way for re-import but will try.

So for the initial import:

• Empty sections are skipped.
• If in HTML try to convert to MD, otherwise copy-in text as is
• Setup maps to Setup
• Breakdown maps to Cleanup
• Actions - Import into top of Test section
• Results - Import into bottom of Test section
  Since there won't be any Step and Expect it will be invalid document and user will be forced to do the review and reorganize it

Export of MD (nitrate)
Convert each section to HTML and fill Setup, Actions, Breadown.
Ignore Results

Feel free to raise objections, ideas. I'm not really happy with ignoring Results, but all I could think of was to map Test to both where Expect sections are copied to Results and all what is left to Actions. Might need a separator inside the TCMS if there are more steps/outcomes. However for readability I think whole Test should be in Actions together

BTW, I've found out that kiwitcms moved to single text field and advises users to keep text there in given-when-then structure

[psss]

The specification update #247 has been merged. @hegerj, we are looking forward to the implementation! :)

Regarding the export I basically agree with @lukaszachy, only for the step/expect pairs I would also consider adding them as lists into the respective field. So for example

# Test

## Step
Do one.
## Step
Check one.

## Step
Do two.
## Step
Check two.

## Step
Do three.
## Step
Check three.

Would result into something like this in Nitrate:

Actions:        Results:

1.              1.
Do one.         Check one.
2.              2.
Do two.         Check two.
3.              3.
Do three.       Check three.

What do you think?

[lukaszachy]

I understand Step/Expect as logical, not necessary atomic pairs. The specification doesn't restrict user to use list on their own and it will end with a mess if they do.
Also what to do with additional text?

# Test
Lorem ipsum

## Step
do this
### On Fedora
do this as well
1. with 
2. multiple
3. commands

## Result
The outcome

[psss]

For the export I would go only with the Step/Result section content and ignore the rest. I mean, take the whole step content and paste it in the Action field. In the above-mentioned example Lorem ipsum would not be synced. This would limit usage of extra notes for those who will still depend on content in nitrate, but I don't think this would be a significant limitation.

[lukaszachy]

@Athwale, @kkaarreell As requesters of manual cases - What is your preference for the export to TCMS? (tl;dr: Can we leave empty Results box in TCMS?:)
Also feedback for the import is welcomed.

[Athwale]

Empty Expected Results: field in TCMS is fine as long as the expected command outputs are visible at least somewhere. I have seen people only use one of the available fields sometimes in TCMS anyway.
The import description seems ok to me.

As for a convert function, currently the tmt test convert works with the cloned test case on your disk. You go in the folder with the test and run the command there. Most manual test cases do not have any files in any git (some do have though), they are only in TCMS. So it will have to work differently here.

[kkaarreell]

I do not have a preference. My only concern is to have the imported/exported content copied over and readable for the user.

[psss]

@hegerj, any update here? Do you plan to work on this in the near future? This is blocking infra services migration.

[Athwale]

Are there any news about this. The deadlines are inevitably approaching at a steady pace and we would like to be prepared. :D

[psss]

I've checked with @hegerj today, he's planning to start looking into it this week.

[psss]

For reference, python-nitrate currently does not support updating manual test case fields. These are read-only because at the time of implementation the Nitrate API did not allow to change these. The latest API reference suggests there has not been any change. I'm checking with @tkdchen whether this feature could be possibly implemented.

[hegerj]

This issue is being addressed in #319

[psss]

Hint from @tkdchen: For updating manual test case fields TestCase.store_text() could be used. So it seems there is a way how to implement export as well. I've created a python-nitrate issue psss/python-nitrate#23 for that.

[psss]

I believe the core part of the functionality has been covered by #319, #347 and #362. Let's track further improvements in dedicated issues or pull requests.