-
Notifications
You must be signed in to change notification settings - Fork 5
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[QT-696] Add
enos scenario outline (#128)
* [QT-696] Add `enos scenario outline`
Add metadata support to Enos scenarios to allow us to quickly read an
outline of a scenario and determine what quality characteristics are
verified.
We introduce new DSL schema:
- `quality`: A new top-level object that describes a quality
characteristic.
- `quality.description`: An required string attribute in the `quality`
block that describes the quality characteristic in detail.
- `scenario.description`: An optional string attribute that describes
the scenario.
- `scenario.step.description`: An optional string attribute that
describes the scenario step.
- `scenario.step.verifies`: An optional `quality` attribute. Can be
either a singular `quality` or a list of `quality`'s.
Using these new blocks and attributes a scenario author can now more
fully describe the purpose of the scenario, each step, and which quality
characteristics are verified by it.
To see the generated outline you can use `enos scenario outline
<optional-filter>` to generate a scenario outline.
The `scenario outline` sub-command supports a new output formatter:
`html`. Use the HTML formatter and your browser for a cleaner overview
if so desired, e.g.:
```
enos scenario outline --format html > outline.html
open outline.html
```
* Fix deprecations in `.golangci.yml`
* Fix warnings output by `golangci-lint`
* Silence `buf` lint error that is irrelevant to us
* Support decoding `quality` blocks
* Support decoding `scenario.description`
* Support decoding `scenario.step.description`
* Support decoding `scenario.verifies`
* Add `OutlineScenario()` RPC to the enos service.
* Add acceptance test for `enos scenario outline`
* Restructure matrix decoder into its own type. This allows us to keep a
copy of the original matrix around to use in the outline
* Add new decoder target for scenario outlines
* Add `ShowOutline()` to `basic` UI
* Add `ShowOutline()` to `machine` UI
* Create a shim HTML UI
* Add `ShowOutline()` to `html` UI
* Bump version to 0.0.28
* Add missing documenation for `globals` DSL to `README.md`
* Add missing documenation for `sample` DSL to `README.md`
* Add documenation for `quality` DSL to `README.md`
* Add missing documenation for `enos scenario sample` command to
`README.md`
* Add documenation for `enos scenario outline` command to `README.md`
Signed-off-by: Ryan Cragun <[email protected]>- Loading branch information
1 parent
e10f3ab
commit d654099
Showing
45 changed files
with
4,605 additions
and
2,106 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,166 @@ | ||
| // Copyright (c) HashiCorp, Inc. | ||
| // SPDX-License-Identifier: MPL-2.0 | ||
|
|
||
| package acceptance | ||
|
|
||
| import ( | ||
| "context" | ||
| "fmt" | ||
| "path/filepath" | ||
| "testing" | ||
|
|
||
| "github.com/stretchr/testify/require" | ||
| "google.golang.org/protobuf/encoding/protojson" | ||
|
|
||
| "github.com/hashicorp/enos/proto/hashicorp/enos/v1/pb" | ||
| ) | ||
|
|
||
| func TestAcc_Cmd_Scenario_Outline(t *testing.T) { | ||
| t.Parallel() | ||
|
|
||
| for _, test := range []struct { | ||
| dir string | ||
| out *pb.OutlineScenariosResponse | ||
| fail bool | ||
| }{ | ||
| { | ||
| dir: "scenarios/scenario_outline", | ||
| out: &pb.OutlineScenariosResponse{ | ||
| Outlines: []*pb.Scenario_Outline{ | ||
| { | ||
| Scenario: &pb.Ref_Scenario{ | ||
| Id: &pb.Scenario_ID{ | ||
| Name: "multiple_verifies", | ||
| Description: "This is a multiline description\nof the upgrade scenario.\n", | ||
| }, | ||
| }, | ||
| Matrix: &pb.Matrix{ | ||
| Vectors: []*pb.Matrix_Vector{ | ||
| {Elements: []*pb.Matrix_Element{ | ||
| {Key: "arch", Value: "amd64"}, | ||
| {Key: "arch", Value: "arm64"}, | ||
| }}, | ||
| {Elements: []*pb.Matrix_Element{ | ||
| {Key: "distro", Value: "ubuntu"}, | ||
| {Key: "distro", Value: "rhel"}, | ||
| }}, | ||
| }, | ||
| }, | ||
| Steps: []*pb.Scenario_Outline_Step{ | ||
| { | ||
| Name: "test", | ||
| Description: "This is an indented\nmultiline step description.\n", | ||
| Verifies: []*pb.Quality{ | ||
| { | ||
| Name: "inline", | ||
| Description: "an inline quality that isn't reused", | ||
| }, | ||
| { | ||
| Name: "the_data_is_durable", | ||
| Description: "The data is durable\nafter an upgrade.\n", | ||
| }, | ||
| { | ||
| Name: "the_tests_pass", | ||
| Description: "The tests all pass!", | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| Verifies: []*pb.Quality{ | ||
| { | ||
| Name: "inline", | ||
| Description: "an inline quality that isn't reused", | ||
| }, | ||
| { | ||
| Name: "the_data_is_durable", | ||
| Description: "The data is durable\nafter an upgrade.\n", | ||
| }, | ||
| { | ||
| Name: "the_tests_pass", | ||
| Description: "The tests all pass!", | ||
| }, | ||
| }, | ||
| }, | ||
| { | ||
| Scenario: &pb.Ref_Scenario{ | ||
| Id: &pb.Scenario_ID{ | ||
| Name: "singular_verifies", | ||
| Description: "This is a multiline description\nof the upgrade scenario.\n", | ||
| }, | ||
| }, | ||
| Matrix: &pb.Matrix{ | ||
| Vectors: []*pb.Matrix_Vector{ | ||
| {Elements: []*pb.Matrix_Element{ | ||
| {Key: "arch", Value: "amd64"}, | ||
| {Key: "arch", Value: "arm64"}, | ||
| }}, | ||
| {Elements: []*pb.Matrix_Element{ | ||
| {Key: "distro", Value: "ubuntu"}, | ||
| {Key: "distro", Value: "rhel"}, | ||
| }}, | ||
| }, | ||
| }, | ||
| Steps: []*pb.Scenario_Outline_Step{ | ||
| { | ||
| Name: "test", | ||
| Description: "This is an indented\nmultiline step description.\n", | ||
| Verifies: []*pb.Quality{ | ||
| { | ||
| Name: "the_tests_pass", | ||
| Description: "The tests all pass!", | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| Verifies: []*pb.Quality{ | ||
| { | ||
| Name: "the_tests_pass", | ||
| Description: "The tests all pass!", | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| } { | ||
| t.Run(test.dir, func(t *testing.T) { | ||
| t.Parallel() | ||
| enos := newAcceptanceRunner(t) | ||
|
|
||
| path, err := filepath.Abs(filepath.Join("./", test.dir)) | ||
| require.NoError(t, err) | ||
| cmd := fmt.Sprintf("scenario outline --chdir %s --format json", path) | ||
| out, err := enos.run(context.Background(), cmd) | ||
| if test.fail { | ||
| require.Error(t, err) | ||
|
|
||
| return | ||
| } | ||
|
|
||
| require.NoError(t, err) | ||
| got := &pb.OutlineScenariosResponse{} | ||
| require.NoError(t, protojson.Unmarshal(out, got)) | ||
| require.Len(t, got.GetOutlines(), len(test.out.GetOutlines())) | ||
| for i := range test.out.GetOutlines() { | ||
| require.EqualValues(t, test.out.GetOutlines()[i].GetScenario().GetId().GetName(), got.GetOutlines()[i].GetScenario().GetId().GetName()) | ||
| require.EqualValues(t, test.out.GetOutlines()[i].GetScenario().GetId().GetDescription(), got.GetOutlines()[i].GetScenario().GetId().GetDescription()) | ||
| require.EqualValues(t, test.out.GetOutlines()[i].GetScenario().GetId().GetFilter(), got.GetOutlines()[i].GetScenario().GetId().GetFilter()) | ||
| require.EqualValues(t, test.out.GetOutlines()[i].GetScenario().GetId().GetUid(), got.GetOutlines()[i].GetScenario().GetId().GetUid()) | ||
| require.EqualValues(t, test.out.GetOutlines()[i].GetMatrix().String(), got.GetOutlines()[i].GetMatrix().String()) | ||
| require.Len(t, got.GetOutlines()[i].GetVerifies(), len(test.out.GetOutlines()[i].GetVerifies())) | ||
| for q := range test.out.GetOutlines()[i].GetVerifies() { | ||
| require.EqualValues(t, test.out.GetOutlines()[i].GetVerifies()[q].GetName(), got.GetOutlines()[i].GetVerifies()[q].GetName()) | ||
| require.EqualValues(t, test.out.GetOutlines()[i].GetVerifies()[q].GetDescription(), got.GetOutlines()[i].GetVerifies()[q].GetDescription()) | ||
| } | ||
| for s := range test.out.GetOutlines()[i].GetSteps() { | ||
| require.EqualValues(t, test.out.GetOutlines()[i].GetSteps()[s].GetName(), got.GetOutlines()[i].GetSteps()[s].GetName()) | ||
| require.EqualValues(t, test.out.GetOutlines()[i].GetSteps()[s].GetDescription(), got.GetOutlines()[i].GetSteps()[s].GetDescription()) | ||
| for q := range test.out.GetOutlines()[i].GetSteps()[s].GetVerifies() { | ||
| require.EqualValues(t, test.out.GetOutlines()[i].GetSteps()[s].GetVerifies()[q].GetName(), got.GetOutlines()[i].GetSteps()[s].GetVerifies()[q].GetName()) | ||
| require.EqualValues(t, test.out.GetOutlines()[i].GetSteps()[s].GetVerifies()[q].GetDescription(), got.GetOutlines()[i].GetSteps()[s].GetVerifies()[q].GetDescription()) | ||
| } | ||
| } | ||
| } | ||
| }) | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| # Copyright (c) HashiCorp, Inc. | ||
| # SPDX-License-Identifier: MPL-2.0 | ||
|
|
||
| module "test" { | ||
| source = "../scenario_generate_pass_0/modules/foo" | ||
| } | ||
|
|
||
| quality "the_tests_pass" { | ||
| description = "The tests all pass!" | ||
| } | ||
|
|
||
| quality "the_data_is_durable" { | ||
| description = <<-EOF | ||
| The data is durable | ||
| after an upgrade. | ||
| EOF | ||
| } | ||
|
|
||
| scenario "singular_verifies" { | ||
| description = <<EOF | ||
| This is a multiline description | ||
| of the upgrade scenario. | ||
| EOF | ||
|
|
||
| matrix { | ||
| arch = ["amd64", "arm64"] | ||
| distro = ["ubuntu", "rhel"] | ||
| } | ||
|
|
||
| step "test" { | ||
| description = <<-EOF | ||
| This is an indented | ||
| multiline step description. | ||
| EOF | ||
|
|
||
| verifies = quality.the_tests_pass | ||
|
|
||
| module = module.test | ||
|
|
||
| variables { | ||
| input = matrix.arch | ||
| anotherinput = matrix.distro | ||
| } | ||
| } | ||
| } | ||
|
|
||
| scenario "multiple_verifies" { | ||
| description = <<EOF | ||
| This is a multiline description | ||
| of the upgrade scenario. | ||
| EOF | ||
|
|
||
| matrix { | ||
| arch = ["amd64", "arm64"] | ||
| distro = ["ubuntu", "rhel"] | ||
| } | ||
|
|
||
| step "test" { | ||
| description = <<-EOF | ||
| This is an indented | ||
| multiline step description. | ||
| EOF | ||
|
|
||
| verifies = [ | ||
| quality.the_tests_pass, | ||
| { | ||
| name : "inline", | ||
| description : "an inline quality that isn't reused", | ||
| }, | ||
| quality.the_data_is_durable, | ||
| ] | ||
|
|
||
| module = module.test | ||
|
|
||
| variables { | ||
| input = matrix.arch | ||
| anotherinput = matrix.distro | ||
| } | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,3 +8,6 @@ breaking: | |
| lint: | ||
| use: | ||
| - DEFAULT | ||
| ignore_only: | ||
| PACKAGE_DIRECTORY_MATCH: | ||
| - proto/hashicorp/enos/v1/enos.proto | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.