Add module vars and improved params support (#13)

* It is now possible to specify module-specific vars and var files that are only considered when the corresponding module is run * Params can now also be specified in the config file. Additionally, it is possible to specify and constrain required params that must be specified on the command-line when running gotf
craftypath · Apr 21, 2020 · 1be085d · 1be085d
1 parent 3e7b295
commit 1be085d
Show file tree

Hide file tree

Showing 37 changed files with 757 additions and 252 deletions.
diff --git a/.gitignore b/.gitignore
@@ -4,3 +4,4 @@
 .project
 .settings
 dist
+testdata/gotf*
diff --git a/README.md b/README.md
@@ -47,7 +47,6 @@ Flags:
 
 `gotf` is configured via config file.
 By default, `gotf.yaml` is loaded from the current directory.
-In a real-world scenario, you will probably have a config file per environment.
 Config files support templating as specified below.
 
 ### Parameters
@@ -57,14 +56,36 @@ Config files support templating as specified below.
 Optionally sets a specific Terraform version to use.
 `gotf` will download this version and cache it in `$XDG_CACHE_HOME/gotf/terraform/<version>` verifying GPG signature and SHA256 sum.
 
-#### `varFiles`
+#### `params`
 
-Variable files for Terraform which are added to the Terraform environment via `TF_CLI_ARGS_<command>=-var-file=<file>` for commands that support them.
+Config entries that can be used for templating. See section on templating below for details.
+
+#### `requiredParams`
+
+In addition to specifying `params` in the config file, they may also be specified on the command-line using the `-p|--param` flag.
+Params that are required can be configured here.
+Allowed values for a `param` must be specified as list.
+If no restrictions apply, no value or an empty list must be specified.
+Values must be strings.
+
+#### `globalVarFiles`
+
+A list of variables files which are added to the Terraform environment via `TF_CLI_ARGS_<command>=-var-file=<file>` for commands that support them.
+They are resolved relative to this config file.
+
+#### `moduleVarFiles.<moduleDir>`
+
+A list of module-specific variables files which are added to the Terraform environment if the corresponding module is run via `TF_CLI_ARGS_<command>=-var-file=<file>` for commands that support them.
 They are resolved relative to this config file.
 
-#### `vars`
+#### `globalVars`
+
+Variables which are added to the Terraform environment via `TF_VAR_<var>=value` for commands that support them.
 
-A list of variables that are added to the Terraform environment via `TF_VAR_<var>=value` for commands that support them.
+#### `moduleVars.<moduleDir>`
+
+Module-specific variables which are added to the Terraform environment if the corresponding module is run via `TF_VAR_<var>=value` for commands that support them.
+Module-specific variables override global ones.
 
 #### `envs`
 
@@ -77,12 +98,21 @@ Backend configs are always added as variables (`TF_VAR_backend_<var>=value`) for
 ### Example
 
 ```yaml
-terraformVersion: 0.12.21
+terraformVersion: 0.12.24
+
+requiredParams:
+  environment:
+    - dev
+    - prod
 
-varFiles:
-  - test-{{ .Params.env }}.tfvars
+params:
+  param: myval
 
-vars:
+globalVarFiles:
+  - global-{{ .Params.environment }}.tfvars
+  - global.tfvars
+
+globalVars:
   foo: foovalue
   templated_var: "{{ .Params.param }}"
   mapvar: |-
@@ -96,48 +126,69 @@ vars:
         value2 = false
       }
     }
-  module_path: "{{ .Params.moduleDir }}"
-  module: "{{ base .Params.moduleDir }}"
-  state_key_prefix: '{{ (splitn "_" 2 (base .Params.moduleDir))._1 }}'
+  module_dir: "{{ .Params.moduleDir }}"
+  state_key: '{{ (splitn "_" 2 .Params.moduleDir)._1 }}'
+
+moduleVarFiles:
+  01_networking:
+    - 01_networking/{{ .Params.environment }}.tfvars
+  02_compute:
+    - 02_compute/{{ .Params.environment }}.tfvars
+
+moduleVars:
+  01_networking:
+    myvar: value for networking
+  02_compute:
+    myvar: value for compute
 
 envs:
   BAR: barvalue
   TEMPLATED_ENV: "{{ .Params.param }}"
 
 backendConfigs:
-  key: "{{ .Vars.state_key_prefix }}_{{ .Vars.templated_var }}_{{ .Params.key_suffix }}"
-  storage_account_name: be_storage_account_name_{{ .Vars.foo }}_{{ .Envs.BAR }}
-  resource_group_name: be_resource_group_name_{{ .Vars.foo }}_{{ .Envs.BAR }}
-  container_name: be_container_name_{{ .Vars.foo }}_{{ .Envs.BAR }}
+  key: "{{ .Vars.state_key }}"
+  storage_account_name: mytfstateaccount{{ .Params.environment }}
+  resource_group_name: mytfstate-{{ .Params.environment }}
+  container_name: mytfstate-{{ .Params.environment }}
 ```
 
 ### Templating
 
 Go templating can be used in the config file as follows.
 The [Sprig](https://masterminds.github.io/sprig/) function library is included.
 
-* In the first templating pass, `varFiles`, `vars`, and `envs` are processed.
-  All parameters specified using the `-p|--param` flag are available in the `.Params` object.
-  The module directory passed with the `--module-dir|-m` parameter is available as `module` dir in the `.Params` object.
+* In the first templating pass, `globalVarFiles`, `globalVars`, `moduleVarFiles`, `moduleVars`, and `envs` are processed.
+  All parameters specified under `params` and using the `-p|--param` flag are available in the `.Params` object.
+  CLI params override those specified in the config file.
+  The basename of the module directory passed with the `--module-dir|-m` parameter is available as `moduleDir` dir in the `.Params` object.
 * In the second templating pass, `backendConfigs` are processed.
-  `vars` are available as `.Vars`, `envs` are available as `.Envs` with the results from the first templating pass.
+  `globalVars` and ` moduleVars` are available as `.Vars` and `envs` are available as `.Envs` with the results from the first templating pass.
   Additionally, `.Params` is also available again.
 
 Using the above config file, running `terraform init` could look like this:
 
 ```console
-$ gotf -c example-config.yaml -p param=myval -p key_suffix=mysuffix -m my_modules/01_testmodule init
+$ gotf -c example-config.yaml -p environment=dev -m 01_networking init
 ```
 
 After processing, the config file would look like this:
 
 ```yaml
-terraformVersion: 0.12.21
+terraformVersion: 0.12.24
+
+requiredParams:
+  environment:
+    - dev
+    - prod
+
+params:
+  param: myval
 
-varFiles:
-  - test-prod.tfvars
+globalVarFiles:
+  - global-dev.tfvars
+  - global.tfvars
 
-vars:
+globalVars:
   foo: foovalue
   templated_var: "myval"
   mapvar: |-
@@ -151,19 +202,30 @@ vars:
         value2 = false
       }
     }
-  module_path: "my_modules/01_testmodule"
-  module: "01_testmodule"
-  state_key_prefix: 'testmodule'
+  module_dir: "01_networking"
+  state_key: 'networking'
+
+moduleVarFiles:
+  01_networking:
+    - 01_networking/dev.tfvars
+  02_compute:
+    - 02_compute/dev.tfvars
+
+moduleVars:
+  01_networking:
+    myvar: value for networking
+  02_compute:
+    myvar: value for compute
 
 envs:
   BAR: barvalue
   TEMPLATED_ENV: "myval"
 
 backendConfigs:
-  key: testmodule_myval_mysuffix
-  storage_account_name: be_storage_account_name_foovalue_barvalue
-  resource_group_name: be_resource_group_name_foovalue_barvalue
-  container_name: be_container_name_foovalue_barvalue
+  key: "networking"
+  storage_account_name: mytfstateaccountdev
+  resource_group_name: mytfstate-dev
+  container_name: mytfstate-dev
 ```
 
 ## Debug Output
@@ -172,26 +234,33 @@ Specifying the `--debug` flag produces debug output which is written to stderr.
 For example, the integration test in [cmd/gotf/gotf_test.go](cmd/gotf/gotf_test.go) produces the following debug output before running Terraform:
 
 ```console
-gotf> Loading config file: testdata/test-config-prod.yaml
-gotf> Processing varFiles...
-gotf> Processing vars...
+gotf> Loading config file: testdata/test-config.yaml
+gotf> Processing var files...
+gotf> Processing global var files...
+gotf> Processing module var files...
+gotf> Processing global vars...
+gotf> Processing module vars...
 gotf> Processing envs...
-gotf> Proessing backendConfigs...
-gotf> Using Terraform version 0.12.21
-gotf> Downloading Terraform distro...
-gotf> Downloading SHA256 sums file...
-gotf> Downloading SHA256 sums signature file...
-gotf> Verifying GPG signature...
-gotf> Verifying SHA256 sum...
-gotf> Unzipping distro...
-gotf> Terraform binary: /Users/myuser/Library/Caches/gotf/terraform/0.12.21/terraform
+gotf> Processing backend configs...
+gotf> Using Terraform version 0.12.24
+gotf> Terraform version 0.12.24 already installed.
+gotf> Terraform binary: /Users/myuser/Library/Caches/gotf/terraform/0.12.24/terraform
 gotf>
 gotf> Terraform command-line:
 gotf> -----------------------
-gotf> /Users/myuser/Library/Caches/gotf/terraform/0.12.21/terraform init -no-color
+gotf> /Users/myuser/Library/Caches/gotf/terraform/0.12.24/terraform init -no-color
 gotf>
 gotf> Terraform environment:
 gotf> ----------------------
+gotf> TEMPLATED_ENV=myval
+gotf> TF_CLI_ARGS_destroy=-var-file="../global-prod.tfvars" -var-file="../global.tfvars" -var-file="prod.tfvars"
+gotf> TF_VAR_myvar=value for networking
+gotf> TF_CLI_ARGS_init=-backend-config=path=".terraform/terraform-networking-prod.tfstate"
+gotf> TF_CLI_ARGS_import=-var-file="../global-prod.tfvars" -var-file="../global.tfvars" -var-file="prod.tfvars"
+gotf> TF_VAR_module_dir=01_networking
+gotf> TF_CLI_ARGS_plan=-var-file="../global-prod.tfvars" -var-file="../global.tfvars" -var-file="prod.tfvars"
+gotf> TF_CLI_ARGS_refresh=-var-file="../global-prod.tfvars" -var-file="../global.tfvars" -var-file="prod.tfvars"
+gotf> TF_VAR_templated_var=myval
 gotf> TF_VAR_mapvar={
   entry1 = {
     value1 = testvalue1
@@ -202,15 +271,9 @@ gotf> TF_VAR_mapvar={
     value2 = false
   }
 }
-gotf> TF_VAR_backend_path=.terraform/terraform-testmodule-prod.tfstate
-gotf> TF_CLI_ARGS_init=-backend-config=path=".terraform/terraform-testmodule-prod.tfstate"
-gotf> TF_VAR_state_key=testmodule
-gotf> TF_VAR_foo=42
+gotf> TF_VAR_backend_path=.terraform/terraform-networking-prod.tfstate
 gotf> BAR=barvalue
-gotf> TF_CLI_ARGS_apply=-var-file="../01_testmodule/test-prod.tfvars"
-gotf> TF_CLI_ARGS_destroy=-var-file="../01_testmodule/test-prod.tfvars"
-gotf> TF_CLI_ARGS_plan=-var-file="../01_testmodule/test-prod.tfvars"
-gotf> TF_CLI_ARGS_refresh=-var-file="../01_testmodule/test-prod.tfvars"
-gotf> TF_CLI_ARGS_import=-var-file="../01_testmodule/test-prod.tfvars"
-gotf> TF_VAR_module_dir=01_testmodule
+gotf> TF_CLI_ARGS_apply=-var-file="../global-prod.tfvars" -var-file="../global.tfvars" -var-file="prod.tfvars"
+gotf> TF_VAR_state_key=networking
+gotf> TF_VAR_foo=42
 ```