automated tutorials and testing

[ilgooz]

In progress

We have tutorials in our docs.

Tutorials might quickly get out of sync if the templates used for scaffolding changes or between SDK versions and some other reasons. They should be kept up to date and always properly work with the latest version of the CLI. Whenever we make a change in the develop branch, we should make sure that tutorials are not broken.

To fix this this problem we basically need to run tests for tutorials which requires us to write tutorials in a special format, but not entirely different from Markdown format. If Markdown supports it, we basically need to add some some metadata to each step in a tutorial and later use that metadata to run tests from a tutorial.

Running tests

• Create a new package named tutorials under the integration tests dir.
• In the tutorials package, create a test suite that will automatically pick up tutorials from docs/docs/guide (we may need to refactor the docs dir if needed).
• For each tutorial, create a new tutorial testing suite and Add(path string) all the .md files for that tutorial to the suite.
• Run() the tests.
• Convert all tutorials under docs/docs/guide to the new format, to add metadata so the test suite can use it interpret and run tests.

An example from the scavenge tutorial:

01-hunt.md

Doesn't have any steps

02-game.md

Doesn't have any steps

03-scaffolding.md

1st step is about scaffolding a chain with no module. It is originally written in the .md file like below:

```bash
ignite scaffold chain github.com/username/scavenge --no-module
```

We can add a hidden comment, as a metadata, just above the code block to give instructions to the test runner about how tun interpret this step:

[//]: # (exec)

```bash
ignite scaffold chain github.com/username/scavenge --no-module
```

The test runner in the beginning should create a temporary directory and then in this step, recognize the exec instruction and then execute ignite scaffold ...module command.

If the instruction is unknown, then tests should fail.

The interoperation part of the Markdown file and executing steps could be a generic package can be created under pkg/ and used in integration/tutorials. This way, if we plan to add a new command set ignite tutorial, logic can be reused.

The ignite tutorial could be about scaffolding an from a tutorial but not necessarily running test over it.

2nd step could be converted as:

[//]: # (exec)

```bash
cd scavenge
```

3rd step is same as the 2nd step

4th step is same as the 2nd step

...

07-keeper.md

...

As different from the exec instruction, here in this step below tutorial asks user to edit a certain file (x/scavenge/keeper/msg_server_submit_scavenge.go ), that part of tutorial can be rewritten like this:

Review the file below:

[//]: # (replace-func: msgServer.SubmitScavenge)

```
x/scavenge/keeper/scavenge.go
```

See the `SetScavenge` updates that were made in the `keeper` package, like the `store.Set` method that writes a Scavenge into the store:

```go
import (
    "context"

    "github.com/username/scavenge/x/scavenge/types"
    sdk "github.com/cosmos/cosmos-sdk/types"
    sdkerrors "github.com/cosmos/cosmos-sdk/types/errors"
    "github.com/tendermint/tendermint/crypto"
)

func (k msgServer) SubmitScavenge(goCtx context.Context, msg *types.MsgSubmitScavenge) (*types.MsgSubmitScavengeResponse, error) {
	// ...
}

```

In this step, replace-func instruction:

• Keep the function name on mind to later use.
• Read the first following code block to get the file name.
• Find the function name from AST
• Rewrite AST -with a help of 3rd party high level tool- to replace the existent function with the one defined in the second code block.
• Adjust import statements if needed so app can compile properly.

...

TODO: define execute query, tx, relayer steps like in the tutorial

[salmad3]

As we launched tutorials.ignite.com and due to the current tutorial workflow, this would not apply anymore. However, let's convert to discussion as the concepts here are good reference.