v1.4.15

what

• Various fixes and improvements
• Better error handling and error messages
• Improve validate stacks command
• Improve describe stacks command

why

• In this configuration

components:
  terraform:
    "test/test-component-override":
      # The `component` attribute specifies that `test/test-component-override` inherits from the `test/test-component` base component,
      # and points to the `test/test-component` Terraform component in the `components/terraform` folder
      component: "test/test-component"

don't require that the base component test/test-component has to be defined in the YAML config (even with an empty vars section) unless it's by itself has its own base YAML component. The base component test/test-component must exist in the components/terraform folder, but not in YAML config.

• Fix this (allow naming YAML components not following the folder structure in the components folder)

components:
  terraform:
    # Note that you can name this YAML component anything that fits your needs.
    # For example, it can be named `test/test-component-override-3` or `test-test-component-override-3`
    # This affects only the `atmos` command you use to provision the component:
    # For example:
    # atmos terraform plan test/test-component-override-3 -s xxx (if you name the YAML component `test/test-component-override-3`)
    # atmos terraform plan test-test-component-override-3 -s xxx (if you name the YAML component `test-test-component-override-3`)
    # The terraform component (terraform code) it points to is defined by using the `metadata.component` attribute
    "test/test-component-override-3":
      vars: {}

• atmos validate stacks command now honors ATMOS_BASE_PATH ENV var (and all other settings from atmos.yaml)

• atmos validate stacks command now validates:

  • all YAML files for any YAML errors and inconsistencies
  • all imports (if they are configured correctly, have correct data types, and if all imports point to existing files)
  • schema - check if all sections in all YAML files are correctly configured and have the correct data types (Note that this is still not a full schema validation, it just check the sections like vars, settings, metadata, env at all levels: global, component type, component)

atmos validate stacks

Validating all YAML files in the 'examples/complete/stacks' folder and all subfolders

Invalid import in the file 'catalog/invalid-yaml-and-schema/invalid-import-1.yaml'
No matches found for the import 'globals/tenant1-globals-does-not-exist'

Invalid import in the file 'catalog/invalid-yaml-and-schema/invalid-import-2.yaml'
The file imports itself in 'catalog/invalid-yaml-and-schema/invalid-import-2'

Invalid import in the file 'catalog/invalid-yaml-and-schema/invalid-import-3.yaml'
The import '[globals/tenant1-globals-invalid]' is not a valid string

Invalid 'import' section in the file 'catalog/invalid-yaml-and-schema/invalid-import-4.yaml'
The 'import' section must be a list of strings

Invalid 'import' section in the file 'catalog/invalid-yaml-and-schema/invalid-import-5.yaml'
The 'import' section must be a list of strings

Invalid 'import' section in the file 'catalog/invalid-yaml-and-schema/invalid-import-6.yaml'
The 'import' section must be a list of strings

Invalid import in the file 'catalog/invalid-yaml-and-schema/invalid-import-7.yaml'
The import is an empty string

Invalid 'import' section in the file 'catalog/invalid-yaml-and-schema/invalid-import-8.yaml'
The 'import' section must be a list of strings

Invalid 'import' section in the file 'catalog/invalid-yaml-and-schema/invalid-import-9.yaml'
The 'import' section must be a list of strings

Invalid 'vars' section in the file 'catalog/invalid-yaml-and-schema/invalid-schema-1'

Invalid 'components.helmfile.bad-component' section in the file 'catalog/invalid-yaml-and-schema/invalid-schema-10'

Invalid 'components.terraform.vpc.vars' section in the file 'catalog/invalid-yaml-and-schema/invalid-schema-11'

Invalid 'components.helmfile.echo-server.vars' section in the file 'catalog/invalid-yaml-and-schema/invalid-schema-12'

Invalid 'components.terraform.vpc.metadata' section in the file 'catalog/invalid-yaml-and-schema/invalid-schema-13'

Invalid 'components.terraform.vpc.settings' section in the file 'catalog/invalid-yaml-and-schema/invalid-schema-14'

Invalid 'components.terraform.vpc.settings.spacelift' section in the file 'catalog/invalid-yaml-and-schema/invalid-schema-15'

Invalid 'terraform.vars' section in the file 'catalog/invalid-yaml-and-schema/invalid-schema-2'

Invalid 'helmfile.vars' section in the file 'catalog/invalid-yaml-and-schema/invalid-schema-3'

Invalid 'components.terraform' section in the file 'catalog/invalid-yaml-and-schema/invalid-schema-4'

Invalid 'components.helmfile' section in the file 'catalog/invalid-yaml-and-schema/invalid-schema-5'

Invalid 'components' section in the file 'catalog/invalid-yaml-and-schema/invalid-schema-6'

Invalid 'settings' section in the file 'catalog/invalid-yaml-and-schema/invalid-schema-7'

Invalid 'env' section in the file 'catalog/invalid-yaml-and-schema/invalid-schema-8'

Invalid 'components.terraform.bad-component' section in the file 'catalog/invalid-yaml-and-schema/invalid-schema-9'

Invalid YAML file 'catalog/invalid-yaml-and-schema/invalid-yaml-1.yaml'
yaml: line 15: found unknown directive name

Invalid YAML file 'catalog/invalid-yaml-and-schema/invalid-yaml-2.yaml'
yaml: line 16: could not find expected ':'

Invalid YAML file 'catalog/invalid-yaml-and-schema/invalid-yaml-3.yaml'
yaml: line 13: did not find expected key

Invalid YAML file 'catalog/invalid-yaml-and-schema/invalid-yaml-4.yaml'
yaml: block sequence entries are not allowed in this context

Invalid YAML file 'catalog/invalid-yaml-and-schema/invalid-yaml-5.yaml'
yaml: mapping values are not allowed in this context

Invalid YAML file 'catalog/invalid-yaml-and-schema/invalid-yaml-6.yaml'
yaml: line 2: block sequence entries are not allowed in this context

Invalid YAML file 'catalog/invalid-yaml-and-schema/invalid-yaml-7.yaml'
yaml: line 4: could not find expected ':'

• Improve describe stacks command:
  • filter out empty stacks (if a query returns stacks w/o any sections, don't include those stacks in the result) - see #141
  • Include derived components when filtering on a base component - see #142

atmos describe stacks --components=test/test-component --sections=none

tenant1/ue2/dev:
  components:
    terraform:
      test/test-component: {}
      test/test-component-override: {}
      test/test-component-override-2: {}
      test/test-component-override-3: {}
tenant1/ue2/prod:
  components:
    terraform:
      test/test-component: {}
      test/test-component-override: {}
      test/test-component-override-2: {}
      test/test-component-override-3: {}
tenant1/ue2/staging:
  components:
    terraform:
      test/test-component: {}
      test/test-component-override: {}
      test/test-component-override-2: {}
      test/test-component-override-3: {}
tenant1/ue2/test1:
  components:
    terraform:
      test/test-component: {}
      test/test-component-override: {}
      test/test-component-override-2: {}
      test/test-component-override-3: {}
tenant2/ue2/dev:
  components:
    terraform:
      test/test-component: {}
      test/test-component-override: {}
      test/test-component-override-2: {}
      test/test-component-override-3: {}
tenant2/ue2/prod:
  components:
    terraform:
      test/test-component: {}
      test/test-component-override: {}
      test/test-component-override-2: {}
      test/test-component-override-3: {}
tenant2/ue2/staging:
  components:
    terraform:
      test/test-component: {}
      test/test-component-override: {}
      test/test-component-override-2: {}
      test/test-component-override-3: {}

NOTE: test/test-component-override-3, test/test-component-override-2, test/test-component-override - all have test/test-component as the base Terraform component (via component or metadata.component attributes)

related

• Closes #141
• Closes #142

v1.4.14

what

• Support terraform two-words commands

why

• Support terraform workspace <subcommand>:
  • terraform workspace list
  • terraform workspace select
  • terraform workspace new
  • terraform workspace delete
  • terraform workspace show

references

• https://www.terraform.io/cli/commands/workspace

test

atmos terraform workspace list test/test-component-override-3 -s tenant1-ue2-dev

Executing command:
/usr/local/bin/terraform workspace list
  default
* test-component-override-3-workspace

atmos terraform workspace show test/test-component-override-3 -s tenant1-ue2-dev

Executing command:
/usr/local/bin/terraform workspace show
test-component-override-3-workspace

atmos terraform workspace select test/test-component-override-3 -s tenant1-ue2-dev

Executing command:
/usr/local/bin/terraform workspace show
test-component-override-3-workspace

atmos terraform workspace new test/test-component-override-3 -s tenant1-ue2-dev

Executing command:
/usr/local/bin/terraform workspace new test-component-override-3-workspace
Created and switched to workspace "test-component-override-3-workspace"!

v1.4.13

what

• Improve error handling and error messages
• Add atmos validate stacks command

why

• Check and validate all YAML files in the stacks folder
• Detect invalid YAML and print the file names and the line numbers

test

atmos validate stacks

Invalid YAML file 'catalog/invalid-yaml/invalid-yaml-1.yaml'
yaml: line 15: found unknown directive name

Invalid YAML file 'catalog/invalid-yaml/invalid-yaml-2.yaml'
yaml: line 16: could not find expected ':'

Invalid YAML file 'catalog/invalid-yaml/invalid-yaml-3.yaml'
yaml: line 13: did not find expected key

Invalid YAML file 'catalog/invalid-yaml/invalid-yaml-4.yaml'
yaml: block sequence entries are not allowed in this context

Invalid YAML file 'catalog/invalid-yaml/invalid-yaml-5.yaml'
yaml: mapping values are not allowed in this context

Invalid YAML file 'catalog/invalid-yaml/invalid-yaml-6.yaml'
yaml: line 2: block sequence entries are not allowed in this context

Invalid YAML file 'catalog/invalid-yaml/invalid-yaml-7.yaml'
yaml: line 4: could not find expected ':'

v1.4.12

what

• Add atmos vendor commands
• Add atmos vendor pull command
• Improve error messages
• Cleanup code

why

• atmos vendor commands are used to manage vendoring for components and stacks
• atmos vendor pull -c <component> command pulls sources and mixins for the specified component
• Support k8s-style YAML config (file component.yaml) to describe component vendoring configuration. The file is placed into the component folder and then the atmos command atmos vendor pull -c <component> is executed to pull the sources and mixins for the component

component.yaml

apiVersion: atmos/v1
kind: ComponentVendorConfig
metadata:
  name: vpc-flow-logs-bucket-vendor-config
  description: Source and mixins config for vendoring of 'vpc-flow-logs-bucket' component
spec:
  source:
    # 'uri' supports all protocols (local files, Git, Mercurial, HTTP, HTTPS, Amazon S3, Google GCP),
    # and all URL and archive formats as described in https://github.com/hashicorp/go-getter
    # In 'uri', Golang templates are supported  https://pkg.go.dev/text/template
    # If 'version' is provided, '{{.Version}}' will be replaced with the 'version' value before pulling the files from 'uri'
    uri: github.com/cloudposse/terraform-aws-components.git//modules/vpc-flow-logs-bucket?ref={{.Version}}
    version: 0.194.0
    # Only include the files that match the 'included_paths' patterns
    # If 'included_paths' is not specified, all files will be matched except those that match the patterns from 'excluded_paths'
    # 'included_paths' support POSIX-style Globs for file names/paths (double-star `**` is supported)
    # https://en.wikipedia.org/wiki/Glob_(programming)
    # https://github.com/bmatcuk/doublestar#patterns
    included_paths:
      - "**/*.tf"
      - "**/*.tfvars"
      - "**/*.md"
    # Exclude the files that match any of the 'excluded_paths' patterns
    # Note that we are excluding 'context.tf' since a newer version of it will be downloaded using 'mixins'
    # 'excluded_paths' support POSIX-style Globs for file names/paths (double-star `**` is supported)
    excluded_paths:
      - "**/context.tf"

  # mixins override files from 'source' with the same 'filename' (e.g. 'context.tf' will override 'context.tf' from the 'source')
  # mixins are processed in the order they are declared in the list
  mixins:
    # https://github.com/hashicorp/go-getter/issues/98
    - uri: https://raw.githubusercontent.com/cloudposse/terraform-null-label/0.25.0/exports/context.tf
      filename: context.tf
    - uri: https://raw.githubusercontent.com/cloudposse/terraform-aws-components/{{.Version}}/modules/datadog-agent/introspection.mixin.tf
      version: 0.194.0
      filename: introspection.mixin.tf

• The URIs (uri) in the vendoring config support all protocols (local files, Git, Mercurial, HTTP, HTTPS, Amazon S3, Google GCP), and all URL and archive formats as described in https://github.com/hashicorp/go-getter

• included_paths and excluded_paths support POSIX-style Globs for file names/paths (double-star ** is supported as well)

test

atmos vendor pull -c infra/vpc-flow-logs-bucket

Pulling sources for the component 'infra/vpc-flow-logs-bucket' 
from 'github.com/cloudposse/terraform-aws-components.git//modules/vpc-flow-logs-bucket?ref=0.194.0' 
and writing to 'examples/complete/components/terraform/infra/vpc-flow-logs-bucket'

Including the file 'README.md' since it matches the '**/*.md' pattern from 'included_paths'
Excluding the file 'context.tf' since it matches the '**/context.tf' pattern from 'excluded_paths'
Including the file 'default.auto.tfvars' since it matches the '**/*.tfvars' pattern from 'included_paths'
Including the file 'main.tf' since it matches the '**/*.tf' pattern from 'included_paths'
Including the file 'outputs.tf' since it matches the '**/*.tf' pattern from 'included_paths'
Including the file 'providers.tf' since it matches the '**/*.tf' pattern from 'included_paths'
Including the file 'variables.tf' since it matches the '**/*.tf' pattern from 'included_paths'
Including the file 'versions.tf' since it matches the '**/*.tf' pattern from 'included_paths'

Pulling the mixin 'https://raw.githubusercontent.com/cloudposse/terraform-null-label/0.25.0/exports/context.tf' 
for the component 'infra/vpc-flow-logs-bucket' 
and writing to 'examples/complete/components/terraform/infra/vpc-flow-logs-bucket'

Pulling the mixin 'https://raw.githubusercontent.com/cloudposse/terraform-aws-components/0.194.0/modules/datadog-agent/introspection.mixin.tf' 
for the component 'infra/vpc-flow-logs-bucket' 
and writing to 'examples/complete/components/terraform/infra/vpc-flow-logs-bucket'

v1.4.11

what

• Support dashes - in the tenant, environment and stage names
• In the examples, add a new stage test-1 and add tests for components, stacks, and spacelift to test having a dash in the stage name (the file name itself being without dashes)

why

• The old atmos supported it (because it was filename-based, not logical stack name based)
• Some clients want to name tenants/environment/stages with dashes in the names (and some already have it, so we need to support that when converting from the old to the new atmos)

test

atmos terraform plan test/test-component-override-3 -s tenant1-ue2-test-1


Found config for the component 'test/test-component-override-3' for the stack 'tenant1-ue2-test-1' 
in the file 'tenant1/ue2/test1'

Variables for the component 'test/test-component-override-3' in the stack 'tenant1-ue2-test-1':

enabled: true
environment: ue2
namespace: eg
region: us-east-2
service_1_list:
- 5
- 6
- 7

v1.4.10

what

• If functions that are used by Terraform providers throw errors, print errors to std.Error

why

• Terraform providers only see errors that are sent to std.Error

v1.4.9

what

• Add atmos aws commands
• Add atmos aws eks update-kubeconfig command

why

• Execute aws CLI commands using atmos context (atmos.yaml CLI config and component/stack configurations)
• Downloading kubeconfig from an EKS cluster using `atmos
• Allow using this functionality in Terraform by implementing a new data source in the terraform-provider-utils provider (this will be added in a PR in https://github.com/cloudposse/terraform-provider-utils repo)

`atmos aws eks update-kubeconfig`  command can be executed in three different ways:

1. If all the required parameters (cluster name and AWS profile/role) are provided on the command-line, 
then 'atmos' executes the command without requiring atmos CLI config and context.
For example: atmos aws eks update-kubeconfig --profile=<profile> --name=<cluster_name>

2. If 'component' and 'stack' are provided on the command-line, 
   then 'atmos' executes the command using atmos CLI config and stack's context by searching for the following settings:
  - 'components.helmfile.cluster_name_pattern' in 'atmos.yaml' CLI config (and calculates the '--name' parameter using the pattern)
  - 'components.helmfile.helm_aws_profile_pattern' in 'atmos.yaml' CLI config (and calculates the '--profile' parameter using the pattern)
  - 'components.helmfile.kubeconfig_path' in 'atmos.yaml' CLI config
  - the variables for the component in the provided stack
  - 'region' from the variables for the component in the stack
For example: atmos aws eks update-kubeconfig <component> -s <stack>

3. Combination of the above. Provide a component and a stack, and override other parameters on the command line:
For example: atmos aws eks update-kubeconfig <component> -s <stack> --kubeconfig=<path_to_kubeconfig> --region=<region>

references

• https://docs.aws.amazon.com/cli/latest/reference/eks/update-kubeconfig.html

v1.4.8

what

• Add metadata.terraform_workspace_pattern

why

• We already have metadata.terraform_workspace using which we can specify/override the terraform workspace for a component
• metadata.terraform_workspace_pattern introduces a pattern to override a terraform workspace for a component

      metadata:
        # Override Terraform workspace
        terraform_workspace: xxxxxxxxxxxxxxx
        terraform_workspace_pattern: "{tenant}-{environment}-{stage}-{component}"

• The following tokens are supported in metadata.terraform_workspace_pattern:

  • {namespace}
  • {tenant}
  • {environment}
  • {stage}
  • {component}
  • {base-component}

• This is useful to when converting YAML stack configs from the old atmos based on variant. In he old atmos, when a component: xxx attribute was specified for a component, it was considered as base component even if it was the same as the YAML component itself, in which case the terraform workspace for the component was calculated as <environment>-<stage>-<component>. The new atmos does not consider the same name for the component: attribute as a base component, which leads to the terraform workspace <environment>-<stage> (and hence Terraform not seeing the old workspace and wanting to re-create the component. To fix it, add terraform_workspace_pattern: "{environment}-{stage}-{component}" to the component's metadata

v1.4.7

what

• Take into account init_run_reconfigure CLI config and --init-run-reconfigure command-line argument when running atmos terraform init ... command

why

• atmos terraform init must behave the same as all other commands that use init_run_reconfigure CLI config and --init-run-reconfigure command-line argument

v1.4.6

what

• If auto_generate_backend_file is true (we are auto-generating backend files), remove backend.tf.json when executing atmos terraform clean command

why

• Useful when using different backends for the same component in different stacks