Document new testing approach (#1048)

* document new testing approach

* remove unused workflow branches

* remove stale folder

* fix link

* document new testing approach

* remove unused workflow branches

* fix link

* Extend tests details in contributing guide

* Typos

Co-authored-by: Julio Castillo <[email protected]>

.github/workflows/linting.yml

@@ -16,8 +16,6 @@ name: "Linting"
 on:
   pull_request:
     branches:
-      - fast-dev
-      - fast-dev-gke
       - master
     tags:
       - ci

.github/workflows/tests.yml

@@ -14,12 +14,10 @@
 
 name: "Tests"
 on:
-  schedule:
-    - cron: "45 2 * * *"
+  # schedule:
+  #   - cron: "45 2 * * *"
   pull_request:
     branches:
-      - fast-dev
-      - fast-dev-gke
       - master
     tags:
       - ci

CONTRIBUTING.md

@@ -72,11 +72,13 @@ pytest tests/examples
 Once everything looks good, add/commit any pending changes then push and open a PR on GitHub. We typically enforce a set of design and style conventions, so please make sure you have familiarized yourself with the following sections and implemented them in your code, to avoid lengthy review cycles.
 
 HINT: if you work on high-latency or low-bandwidth network use `TF_PLUGIN_CACHE_DIR` environment variable to dramatically speed up the tests, for example:
+
 ```bash
 TF_PLUGIN_CACHE_DIR=/tmp/tfcache pytest tests
 ```
 
 Or just add into your [terraformrc](https://developer.hashicorp.com/terraform/cli/config/config-file):
+
 ```
 plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"
 ```
@@ -544,6 +546,7 @@ locals {
 #### The `prefix` variable
 
 If you would like to use a "prefix" variable for resource names, please keep its definition consistent across all modules:
+
 ```hcl
 # variables.tf
 variable "prefix" {
@@ -563,6 +566,7 @@ locals {
 ```
 
 For blueprints the prefix is mandatory:
+
 ```hcl
 variable "prefix" {
   description = "Prefix used for resource names."
@@ -608,6 +612,7 @@ The linting workflow tests:
 
 - that the correct copyright boilerplate is present in all files, using `tools/check_boilerplate.py`
 - that all Terraform code is linted via `terraform fmt`
+- that Terraform variables and outputs are sorted alphabetically
 - that all README files have up to date outputs, variables, and files (where relevant) tables, via `tools/check_documentation.py`
 - that all links in README files are syntactically correct and valid if internal, via `tools/check_links.py`
 - that resource names used in FAST stages stay within a length limit, via `tools/check_names.py`
@@ -639,90 +644,123 @@ The test workflow runs test suites in parallel. Refer to the next section for mo
 
 #### Using and writing tests
 
-Our testing approach follows a simple philosophy: we mainly test to ensure code works, and it does not break due to changes to dependencies (modules) or provider resources.
+Our testing approach follows a simple philosophy: we mainly test to ensure code works, and that it does not break due to changes to dependencies (modules) or provider resources.
 
 This makes testing very simple, as a successful `terraform plan` run in a test case is often enough. We only write more specialized tests when we need to check the output of complex transformations in `for` loops.
 
-As our testing needs are very simple, we also wanted to reduce the friction required to write new tests as much as possible: our tests are written in Python, and use `pytest` which is the standard for the language. We adopted this approach instead of others (Inspec/Kitchen, Terratest) as it allows writing simple functions as test units using Python which is simple and widely known.
+As our testing needs are very simple, we also wanted to reduce the friction required to write new tests as much as possible: our tests are written in Python and use `pytest` which is the standard for the language, leveraging our [`tftest`](https://pypi.org/project/tftest/) library, which wraps the Terraform executable and returns familiar data structures for most commands.
 
-The last piece of our testing framework is our [`tftest`](https://pypi.org/project/tftest/) library, which wraps the Terraform executable and returns familiar data structures for most commands.
+Writing `pytest` unit tests to check plan results is really easy, but since wrapping modules and examples in dedicated fixtures and hand-coding checks gets annoying after a while, we developed a thin layer that allows us to use `tfvars` files to run tests, and `yaml` results to check results. In some specific situations you might still want to interact directly with `tftest` via Python, if that's the case skip to the legacy approach below.
 
-##### Testing end-to-end examples
+##### Testing end-to-end examples via `tfvars` and `yaml`
 
-Putting it all together, here is how an end-to-end blueprint test works.
+Our new approach to testing requires you to:
 
-Each example is a Python module in its own directory, and a Terraform fixture that calls the example as a module:
+- create a folder in the right `tests` hierarchy where specific test files will be hosted
+- define `tfvars` files each with a specific variable configuration to test
+- define `yaml` "inventory" files with the plan and output results you want to test
+- declare which of these files need to be run as tests in a `tftest.yaml` file
+
+Let's go through each step in succession, assuming you are testing the new `net-glb` module.
+
+First create a new folder under `tests/modules` replacing any dash in the module name with underscores. You also need to create an empty `__init__.py` file in it, since the folder represents a package from the point of view of `pytest`. Note that if you were testing a blueprint the folder would go in `tests/blueprints`.
 
 ```bash
-tests/blueprints/cloud_operations/iam_delegated_role_grants/
-├── fixture
-│   ├── main.tf
-│   └── variables.tf
-├── __init__.py
-└── test_plan.py
+mkdir tests/modules/net_glb
+touch tests/modules/net_glb/__init__.py
 ```
 
-One point of note is that the folder contains a Python module, so any dash needs to be replaced with underscores to make it importable. The actual test in the `test_plan.py` file looks like this:
+Then define a `tfvars` file with one of the module configurations you want to test. If you have a lot of variables which are shared across different tests, you can group all the common variables in a single `tfvars` file and associate it with each test's specific `tfvars` file (check the [organization module test](./tests/modules/organization/tftest.yaml) for an example).
 
-```python
-def test_resources(e2e_plan_runner):
-  "Test that plan works and the numbers of resources is as expected."
-  modules, resources = e2e_plan_runner()
-  assert len(modules) == 6
-  assert len(resources) == 18
+```hcl
+# file: tests/modules/net_glb/test-simple.tfvars
+name       = "glb-test-0"
+project_id = "my-project"
+backend_buckets_config = {
+  default = {
+    bucket_name = "my-bucket"
+  }
+}
 ```
 
-It uses our pytest `e2e_plan_runner` fixture, which assumes a Terraform test setup is present in the `fixture` folder alongside the test file, runs `plan` on it, and returns the number of modules and resources.
+Next define the corresponding inventory `yaml` file which will be used to assert values from the plan that uses the `tfvars` file above. In the inventory file you have three sections available:
 
-The Terraform fixture is a single block that runs the whole example as a module, and a handful of variables that can be used to test different configurations (not used above so they could be replaced with static strings).
+- `values` is a map of resource indexes (the same ones used by Terraform state) and their attribute name and values; you can define just the attributes you are interested in and the other will be ignored
+- `counts` is a map of resource types (eg `google_compute_engine`) and the number of times each type occurs in the plan; here too just define the ones the need checking
+- `outputs` is a map of outputs and their values; where a value is unknown at plan time use the special `__missing__` token
 
-```hcl
-module "test" {
-  source         = "../../../../../blueprints/cloud-operations/asset-inventory-feed-remediation"
-  project_create = var.project_create
-  project_id     = var.project_id
-}
+```yaml
+# file: tests/modules/net_glb/test-simple.yaml
+values:
+  google_compute_global_forwarding_rule.default:
+    description: Terraform managed.
+    load_balancing_scheme: EXTERNAL
+  google_compute_target_http_proxy.default[0]:
+    name: glb-test-1
+counts:
+  google_compute_backend_bucket: 1
+  google_compute_global_forwarding_rule: 1
+  google_compute_health_check: 1
+  google_compute_target_http_proxy: 1
+  google_compute_url_map: 1
+outputs:
+  address: __missing__
+  backend_service_ids: __missing__
+  forwarding_rule: __missing__
+  group_ids: __missing__
+  health_check_ids: __missing__
+  neg_ids: __missing__
 ```
 
-You can run this test as part of or entire suite of tests, the blueprints suite, or individually:
+Create as many pairs of `tfvars`/`yaml` files as you need to test every scenario and feature, then create the file that triggers our fixture and converts them into `pytest` tests.
 
-```bash
-# run all tests
-pytest
-# only run example tests
-pytest tests/blueprints
-# only run this example tests
-pytest tests/blueprints/cloud_operations/iam_delegated_role_grants/
-# only run a single unit
-pytest tests/blueprints/cloud_operations/iam_delegated_role_grants/test_plan.py::test_resources
+```yaml
+# file: tests/modules/net_glb/tftest.yaml
+module: modules/net-glb
+# if there are variables shared among all tests you can define a common file
+# common_tfvars:
+#   - defaults.tfvars
+tests:
+  test-plan:
+    tfvars:
+      - test-plan.tfvars
+      - test-plan-extra.tfvars
 ```
 
-##### Testing modules
+A good example of tests showing different ways of leveraging our framework is in the [`tests/modules/organization`](./tests/modules/organization) folder.
+
+##### Writing tests in Python (legacy approach)
 
-The same approach used above can also be used for testing modules when a simple plan is enough to validate code. When specific features need to be tested though, the `plan_runner` pytest fixture can be used so that plan resources are returned for inspection.
+Where possible, we recommend using the testing framework described in the previous section. However, if you need it, you can still write tests using Python directly.
 
-The following example from the `project` module leverages variables in the Terraform fixture to define which module resources are returned from plan.
+In general, you should try to use the `plan_summary` fixture, which runs a a terraform plan and returns a `PlanSummary` object. The most important arguments to `plan_summary` are:
+- the path of the Terraform module you want to test, relative to the root of the repository
+- a list of paths representing the tfvars file to pass in to terraform. These paths are relative to the python file defining the test.
 
+If successful, `plan_summary` will return a `PlanSummary` object with the `values`, `counts` and `outputs` attributes following the same semantics described in the previous section. You can use this fields to write your custom tests.
+
+Like before let's imagine we're writing a (python) test for `net-glb` module. First create a new folder under `tests/modules` replacing any dash in the module name with underscores. You also need to create an empty `__init__.py` file in it, to ensure `pytest` discovers you new tests automatically.
+
+```bash
+mkdir tests/modules/net_glb
+touch tests/modules/net_glb/__init__.py
+```
+
+Now create a file containing your tests, e.g. `test_plan.py`:
 ```python
-def test_iam(plan_runner):
-  "Test IAM bindings."
-  iam = (
-      '{"roles/owner" = ["user:[email protected]"],'
-      '"roles/viewer" = ["user:[email protected]", "user:[email protected]"]}'
-  )
-  _, resources = plan_runner(iam=iam)
-  roles = dict((r['values']['role'], r['values']['members'])
-               for r in resources if r['type'] == 'google_project_iam_binding')
-  assert roles == {
-      'roles/owner': ['user:[email protected]'],
-      'roles/viewer': ['user:[email protected]', 'user:[email protected]']}
+def test_name(plan_summary, tfvars_to_yaml, tmp_path):
+  s = plan_summary('modules/net-glb', tf_var_files=['test-plan.tfvars'])
+  address = 'google_compute_url_map.default'
+  assert s.values[address]['project'] == 'my-project'
 ```
 
+For more examples on how to write python tests, the tests for  [`organization`](./tests/modules/organization/test_plan_org_policies.py) and [`net-vpc`](./tests/modules/net_vpc/test_routes.py) modules.
+
 #### Testing documentation examples
 
-Most of our documentation examples are also tested via the `examples` test suite. To enable an example for testing just use the special `tftest` comment as the last line in the example, listing the number of modules and resources tested.
+Most of our documentation examples are also tested via the `examples` test suite. To enable an example for testing just use the special `tftest` comment as the last line in the example, listing the number of modules and resources expected.
 
-A few preset variables are available for use, as shown in this example from the `dns` module documentation.
+A [few preset variables](./tests/examples/variables.tf) are available for use, as shown in this example from the `dns` module documentation.
 
 ```hcl
 module "private-dns" {
@@ -739,6 +777,25 @@ module "private-dns" {
 # tftest modules=1 resources=2
 ```
 
+Note that all HCL code examples in READMEs are automatically tested. To prevent this behavior, include `tftest skip` somewhere in the code.
+
+#### Running tests from a temporary directory
+
+    Most of the time you can run tests using the `pytest` command as described in previous. However, the `plan_summary` fixture allows copying the root module and running the test from a temporary directory.
+
+To enable this option, just define the environment variable `TFTEST_COPY` and any tests using the `plan_summary` fixture will automatically run from a temporary directory.
+
+Running tests from temporary directories is useful if:
+- you're running tests in parallel using `pytest-xdist`. In this case, just run you tests as follows:
+  ```bash
+  TFTEST_COPY=1 pytest -n 4
+  ```
+- you're running tests for the `fast/` directory which contain tfvars and auto.tfvars files (which are read by terraform automatically) making your tests fail. In this case, you can run
+  ```
+  TFTEST_COPY=1 pytest fast/
+  ```
+
+
 #### Fabric tools
 
 The main tool you will interact with in development is `tfdoc`, used to generate file, output and variable tables in README documents.