Releases: cloudposse/atmos
v1.75.0
Improve `atmos validate stacks` and `atmos describe affected` commands @aknysh (#608)
what
- Improve
atmos validate stacks
andatmos describe affected
commands - Update docs
why
-
atmos validate stacks
now detects if the same component in the same stack are defined in more than one Atmos stack manifest files. If such a misconfiguration is detected, the following error is shown:the Atmos component 'vpc' in the stack 'plat-ue2-dev' is defined in more than one top-level stack manifest file: orgs/acme/plat/dev/us-east-2, orgs/acme/plat/dev/us-east-2-extras. Atmos can't decide which stack manifest to use to get configuration for the component in the stack. This is a stack misconfiguration.
-
atmos describe affected
now has better error messages and also executesatmos validate stacks
before detecting the affected components and stacks. This prevents the issue when the same component in the same stack is defined in more than one stack manifest file, and the command used one file from the current local branch and the other file from the target branch, resulting in drift in affected components and stacks. Since this is a stack misconfiguration and is not permitted,atmos describe affected
will display the error and exit -
alias
: Multiple Provider Configuration in Atmos ManifestsAtmos allows you to define multiple configurations for the same provider using a list of provider blocks and the
alias
meta-argument.The generated
providers_override.tf.json
file will have a list of provider configurations, and Terraform will use and override the providers as long as the aliased providers are defined in the Terraform component.For example:
components: terraform: vpc: providers: aws: - region: us-west-2 assume_role: role_arn: "role-1" - region: us-west-2 alias: "account-2" assume_role: role_arn: "role-2"
v1.74.0
Update Atmos logs. Update docs @aknysh (#605)
what
- Update Atmos logs
- Make the logs respect the standard file descriptors like
/dev/stderr
- Update docs
why
Atmos logs are configured in the logs
section in the atmos.yaml
CLI config file:
logs:
# Can also be set using 'ATMOS_LOGS_FILE' ENV var, or '--logs-file' command-line argument
# File or standard file descriptor to write logs to
# Logs can be written to any file or any standard file descriptor, including `/dev/stdout`, `/dev/stderr` and `/dev/null`
file: "/dev/stderr"
# Supported log levels: Trace, Debug, Info, Warning, Off
# Can also be set using 'ATMOS_LOGS_LEVEL' ENV var, or '--logs-level' command-line argument
level: Info
-
logs.file
- the file to write Atmos logs to. Logs can be written to any file or any standard file descriptor, including/dev/stdout
,/dev/stderr
and/dev/null
. If omitted,/dev/stdout
will be used. The environment variableATMOS_LOGS_FILE
can also be used to specify the log file -
logs.level
- Log level. Supported log levels areTrace
,Debug
,Info
,Warning
,Off
. If the log level is set toOff
, Atmos will not log any messages (note that this does not prevent other tools like Terraform from logging). The environment variableATMOS_LOGS_LEVEL
can also be used to specify the log level
To prevent Atmos from logging any messages (except for the outputs of the executed commands), you can do one of the following:
-
Set
logs.file
or the ENV variableATMOS_LOGS_FILE
to/dev/null
-
Set
logs.level
or the ENV variableATMOS_LOGS_LEVEL
toOff
Note that when you set the log level to Debug
or Trace
, Atmos will log additional messages before printing the output of an executed command. For example, let's consider the atmos describe affected
command:
logs:
file: "/dev/stdout"
level: Trace
> atmos describe affected
Checking out Git ref 'refs/remotes/origin/HEAD' ...
Checked out Git ref 'refs/remotes/origin/HEAD'
Current working repo HEAD: ffd2154e1daa32357b75460b9f45d268922b51e1 refs/heads/update-logs
Remote repo HEAD: f7aa382aa8b3d48be8f06cfdb27aad344b89aff4 HEAD
Changed files:
examples/quick-start/Dockerfile
examples/quick-start/atmos.yaml
Affected components and stacks:
[
{
"component": "vpc",
"component_type": "terraform",
"component_path": "examples/quick-start/components/terraform/vpc",
"stack": "plat-uw2-prod",
"stack_slug": "plat-uw2-prod-vpc",
"affected": "stack.vars"
},
{
"component": "vpc",
"component_type": "terraform",
"component_path": "examples/quick-start/components/terraform/vpc",
"stack": "plat-ue2-prod",
"stack_slug": "plat-ue2-prod-vpc",
"affected": "stack.vars"
}
]
With logs.level: Trace
, and logs.file: "/dev/stdout"
, all the messages and the command's JSON output will be printed to the console to the /dev/stdout
standard output.
This behavior might be undesirable when you execute the command atmos describe affected
in CI/CD (e.g. GitHub Actions).
For example, you might want to log all the Atmos messages (by setting logs.level: Trace
) for debugging purposes, and also want to parse the JSON output of the command (e.g. by using jq
) for further processing. In this case, jq
will not be able to parse the JSON output because all the other messages make the output an invalid JSON document.
To deal with that, you can set logs.file
to /dev/stderr
in atmos.yaml
:
logs:
file: "/dev/stderr"
level: Trace
Now when the atmos describe affected
command is executed, the additional messages are printed to /dev/stderr
, but the command's JSON output is printed to /dev/stdout
, allowing jq
to parse it without errors.
> atmos describe affected
# NOTE: These messages are printed to `/dev/stderr`
Checking out Git ref 'refs/remotes/origin/HEAD' ...
Checked out Git ref 'refs/remotes/origin/HEAD'
Current working repo HEAD: ffd2154e1daa32357b75460b9f45d268922b51e1 refs/heads/update-logs
Remote repo HEAD: f7aa382aa8b3d48be8f06cfdb27aad344b89aff4 HEAD
# NOTE: This JSON output is printed to `/dev/stdout`
[
{
"component": "vpc",
"component_type": "terraform",
"component_path": "examples/quick-start/components/terraform/vpc",
"stack": "plat-uw2-prod",
"stack_slug": "plat-uw2-prod-vpc",
"affected": "stack.vars"
},
{
"component": "vpc",
"component_type": "terraform",
"component_path": "examples/quick-start/components/terraform/vpc",
"stack": "plat-ue2-prod",
"stack_slug": "plat-ue2-prod-vpc",
"affected": "stack.vars"
}
]
v1.73.0
Allow `Go` templates in `metadata.component` section. Add `components.terraform.command` section to `atmos.yaml`. Document OpenTofu support @aknysh (#604)
what
- Allow
Go
templates inmetadata.component
section - Add
components.terraform.command
andcomponents.helmfile.command
sections toatmos.yaml
- Update docs
- Document OpenTofu support
why
Go
templates incomponent
andmetadata.component
section are useful when multiple versions of the same Terraform component are used fromcomponents/terraform
section, and a particular version can be defined using templates. UseGo
templates inmetadata.component
section as follows:
components:
terraform:
my-component:
settings:
component: my-tf-component # Point to the Terraform component
metadata:
component: "{{ .settings.component }}"
- The
components.terraform.command
andcomponents.helmfile.command
sections inatmos.yaml
specify the executable to be called byatmos
when running Terraform and Helmfile commands. Allows specifying different versions of OpenTofu executables
components:
terraform:
# Optional `command` specifies the executable to be called by `atmos` when running Terraform commands
# If not defined, `terraform` is used
# Examples:
# command: terraform
# command: /usr/local/bin/terraform
# command: /usr/local/bin/terraform-1.8
# command: tofu
# command: /usr/local/bin/tofu-1.7.1
# Can also be set using 'ATMOS_COMPONENTS_TERRAFORM_COMMAND' ENV var, or '--terraform-command' command-line argument
command: terraform
helmfile:
# Optional `command` specifies the executable to be called by `atmos` when running Helmfile commands
# If not defined, `helmfile` is used
# Examples:
# command: helmfile
# command: /usr/local/bin/helmfile
# Can also be set using 'ATMOS_COMPONENTS_HELMFILE_COMMAND' ENV var, or '--helmfile-command' command-line argument
command: helmfile
Related
- Closes #594
v1.72.0
Update `gomplate` datasources. Add `env` and `evaluations ` sections to `Go` template configurations @aknysh (#599)
what
- Update
gomplate
datasources - Add
env
section toGo
template configurations - Add
evaluations
config toGo
template configurations - Update/improve templating docs
why
- Allow setting environment variables in
Go
templates to access cloud resources fromdatasources
- Allow configuring the number of template evaluations for template processing pipelines
description
Atmos supports configuring the number of evaluations/passes for template processing in atmos.yaml
CLI config file. It effectively allows you to define template processing pipelines.
For example:
templates:
settings:
# Enable `Go` templates in Atmos stack manifests
enabled: true
# Number of evaluations/passes to process `Go` templates
# If not defined, `evaluations` is automatically set to `1`
evaluations: 2
templates.settings.evaluations
- number of evaluations/passes to processGo
templates. If not defined,num_steps
is automatically set to1
Template evaluations are useful in the following scenarios:
- Combining templates from different sections in Atmos stack manifests
- Using templates in the URLs of
datasources
Combining templates from different sections in Atmos stack manifests
You can define more than one step/pass of template processing to use and combine the results from each step.
For example:
templates:
settings:
enabled: true
# Number of evaluations/passes to process `Go` templates
evaluations: 3
settings:
test: "{{ .atmos_component }}"
test2: "{{ .settings.test }}"
components:
terraform:
vpc:
vars:
tags:
tag1: "{{ .settings.test }}-{{ .settings.test2 }}"
tag2: "{{\"{{`{{ .atmos_component }}`}}\"}}"
When executing an Atmos command like atmos terraform plan vpc -s <stack>
, the above template will be processed in three phases:
-
Evaluation 1
settings.test
is set tovpc
settings.test2
is set to{{ .atmos_component }}
vpc.vars.tags.tag1
is set to{{ .atmos_component }}-{{ .settings.test }}
vpc.vars.tags.tag2
is set to{{<backtick>{{ .atmos_component }}<backtick>}}
-
Evaluation 2
settings.test
isvpc
settings.test2
is set tovpc
vpc.vars.tags.tag1
is set tovpc-vpc
vpc.vars.tags.tag2
is set to{{ .atmos_component }}
-
Evaluation 3
settings.test
isvpc
settings.test2
isvpc
vpc.vars.tags.tag1
isvpc-vpc
vpc.vars.tags.tag2
is set tovpc
WARNING:
The above example shows the supported functionality in Atmos templating. You can use it for some use-cases, but it does not mean that you should use it just for the sake of using, since it's not easy to read and understand what data we have after each evaluation step.
The following use-case describes a practical approach to using steps and pipelines in Atmos templates to work
with datasources
.
Using templates in the URLs of datasources
Let's suppose that your company uses a centralized software catalog to consolidate all tags for tagging all the cloud resources. The tags can include tags per account, per team, per service, billing tags, etc.
NOTE: An example of such a centralized software catalog could be https://backstage.io
Let's also suppose that you have a service to read the tags from the centralized catalog and write them into an S3 bucket in one of your accounts. The bucket serves as a cache to not hit the external system's API with too many requests and not to trigger rate limiting.
And finally, let's say that in the bucket, you have folders per account (dev
, prod
, staging
). Each folder has a JSON file with all the tags defined for all the cloud resources in the accounts.
We can then use the Gomplate S3 datasource to read the JSON file with the tags for each account and assign the tags to all cloud resources.
In atmos.yaml
, we can figure 2 evaluation steps of template processing:
templates:
settings:
enabled: true
gomplate:
enabled: true
# Number of evaluations to process `Go` templates
evaluations: 2
In an Atmos stack manifest, we define the environment variables in the env
section (AWS profile with permissions to access the S3 bucket), and the s3-tags
Gomplate datasource.
In the terraform.vars.tags
section, we define all the tags that are returned from the call to the S3 datasource.
import:
# Import the default configuration for all VPCs in the infrastructure
- catalog/vpc/defaults
# Global settings
settings:
templates:
settings:
# Environment variables passed to datasources when evaluating templates
# https://docs.gomplate.ca/functions/aws/#configuring-aws
# https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html
env:
# AWS profile with permissions to access the S3 bucket
AWS_PROFILE: "<AWS profile>"
gomplate:
# Timeout in seconds to execute the datasources
timeout: 5
# https://docs.gomplate.ca/datasources
datasources:
# `s3` datasource
# https://docs.gomplate.ca/datasources/#using-s3-datasources
s3-tags:
# The `url` uses a `Go` template with the delimiters `${ }`,
# which is processed as first step in the template processing pipeline
url: "s3://mybucket/${ .vars.stage }/tags.json"
# Global Terraform config
terraform:
# Global variables that are used by all Atmos components
vars:
tags:
atmos_component: "{{ .atmos_component }}"
atmos_stack: "{{ .atmos_stack }}"
terraform_component: "{{ .component }}"
terraform_workspace: "{{ .workspace }}"
devops_team: '{{`{{ (datasource "s3-tags").tags.devops_team }}`}}'
billing_team: '{{`{{ (datasource "s3-tags").tags.billing_team }}`}}'
service: '{{`{{ (datasource "s3-tags").tags.service }}`}}'
# Atmos component configurations
components:
terraform:
vpc/1:
metadata:
component: vpc # Point to the Terraform component in `components/terraform/vpc` folder
inherits:
# Inherit from the `vpc/defaults` base Atmos component, which defines the default
# configuration for all VPCs in the infrastructure.
# The `vpc/defaults` base component is defined in the `catalog/vpc/defaults`
# manifest (which is imported above).
# This inheritance makes the `vpc/1` Atmos component config DRY.
- "vpc/defaults"
vars:
name: "vpc-1"
When executing an Atmos command like atmos terraform apply vpc/1 -s plat-ue2-dev
, the above template will be processed in two steps:
-
Evaluation 1:
-
datasources.s3-tags.url
is set tos3://mybucket/dev/tags.json
-
the tags that use the
datasource
templates are set to the following:devops_team: '{{ (datasource "s3-tags").tags.devops_team }}' billing_team: '{{ (datasource "s3-tags").tags.billing_team }}' service: '{{ (datasource "s3-tags").tags.service }}'
-
-
Evaluation 2:
- all
s3-tags
datasources get executed, the JSON files3://mybucket/dev/tags.json
with the tags
for thedev
account is downloaded from the S3 bucket, and the tags are parsed and assigned in the
terraform.vars.tags
section
- all
After executing the two evaluation steps, the resulting tags for the Atmos component vpc/1
in the stack plat-ue2-dev
would look like this:
atmos_component: vpc/1
atmos_stack: plat-ue2-dev
terraform_component: vpc
terraform_workspace: plat-ue2-dev-vpc-1
devops_team: dev_networking
billing_team: billing_net
service: net
The tags will be added to all the AWS resources provisioned by the vpc
Terraform component in the plat-ue2-dev
stack.
v1.71.0
Update `atmos describe affected` command. Update docs @aknysh (#590)
what
-
Update
atmos describe affected
command -
Add
--clone-target-ref
flag to theatmos describe affected
command -
Update docs. Add "Remote State Backend" doc
why
-
"Remote State Backend" doc describes how to override Terraform Backend Configuration to access components remote state, and how to do Brownfield development in Atmos
-
Simplify the
atmos describe affected
command to not force it to clone the remote target reference (branch or tag), but instead just check it out (assuming the target reference is already on the local file system) -
Add
--clone-target-ref
flag to theatmos describe affected
command for backwards compatibility. If the flag is passed, the command behaves as the old version (clones the target reference first from the remote origin)
breaking changes
If the atmos describe affected
command was used in a GitHub Action (similar to https://github.com/cloudposse/github-action-atmos-affected-stacks), and the action performed a shallow Git clone (instead of a deep clone), it will break with an error that the target reference (branch) does not exist on the file system. There are a few ways to fix it:
-
Use the flag
--clone-target-ref=true
to force the command to clone the target reference from the remote origin (this flag is addd for backwards compatibility)atmos describe affected --clone-target-ref=true
-
Update the GitHub Action to perform a deep-clone instead of a shallow-clone
- uses: actions/checkout@v4 with: fetch-depth: 0
-
Perform a clone of the target branch into a separate directory and use the
--repo-path=<dir>
command line parameter to specify the path to the already cloned target repository (refer to https://atmos.tools/cli/commands/describe/affected#flags)
description
The atmos describe affected
command uses two different Git commits to produce a list of affected Atmos components and stacks.
For the first commit, the command assumes that the current repo root is a Git checkout. An error will be thrown if the
current repo is not a Git repository (the .git/
folder does not exist or is configured incorrectly).
The second commit can be specified on the command line by using the --ref
(Git References) or --sha
(commit SHA) flags. The --sha
takes precedence over the --ref
flag.
How does it work?
The command performs the following:
-
If the
--repo-path
flag is passed, the command uses it as the path to the already cloned target repo with which to compare the current working branch. I this case, the command will not clone and checkout the target reference, but instead will use the already cloned one to compare the current branch with. In this case, the--ref
,--sha
,--ssh-key
and--ssh-key-password
flags are not used, and an error will be thrown if the--repo-path
flag and any of the--ref
,--sha
,--ssh-key
or--ssh-key-password
flags are provided at the same time -
Otherwise, if the
--clone-target-ref=true
flag is specified, the command clones (into a temp directory) the remote target with which to compare the current working branch. If the--ref
flag or the commit SHA flag--sha
are provided, the command uses them to clone and checkout the remote target. Otherwise, theHEAD
of the remote origin is used (refs/remotes/origin/HEAD
Git ref, usually themain
branch) -
Otherwise, (if the
--repo-path
and--clone-target-ref=true
flags are not passed), the command does not clone anything from the remote origin, but instead just copies the current repo into a temp directory and checks out the target reference with which to compare the current working branch. If the--ref
flag or the commit SHA flag--sha
are provided, the command uses them to check out. Otherwise, theHEAD
of the remote origin is used (refs/remotes/origin/HEAD
Git ref, usually themain
branch). This requires that the target reference is already cloned by Git, and the information about it exists in the.git
directory (in case of using a non-default branch as the target, Git deep clone needs to be executed instead of a shallow clone). This is the recommended way to execute theatmos describe affected
command since it allows working with private repositories without providing the SSH credentials (--ssh-key
and--ssh-key-password
flags), since in this case Atmos does not access the remote origin and instead just checks out the target reference (which is already on the local file system) -
The command deep-merges all stack configurations from both sources: the current working branch and the target reference
-
The command searches for changes in the component directories
-
The command compares each stack manifest section of the stack configurations from both sources looking for differences
-
And finally, the command outputs a JSON or YAML document consisting of a list of the affected components and stacks and what caused it to be affected
v1.70.0
Add `gomplate` datasources to `Go` templates in Atmos stack manifests. Update docs @aknysh (#582)
what
- Add
gomplate
datasources toGo
templates in Atmos stack manifests - Fix an issue with enabling/disabling
Go
templates in imports using thetemplates.settings.enabled
section inatmos.yaml
- Update docs
why
- Allow using Gomplate Datasources in
Go
templates in Atmos stack manifests - The
templates.settings.enabled
section inatmos.yaml
also affected templating in imports, but templating in imports should always be enabled regardless of enabling/disabling templating in Atmos stack manifests
description
Atmos supports Go templates in stack manifests.
Sprig Functions, Gomplate Functions and Gomplate Datasources are supported as well.
Configuration
Templating in Atmos stack manifests can be configured in the following places:
-
In the
templates.settings
section inatmos.yaml
CLI config file -
In the
settings.templates.settings
section in Atmos stack manifests. Thesettings.templates.settings
section can be defined globally per organization, tenant, account, or per component. Atmos deep-merges the configurations from all scopes into the final result using inheritance.
Configuring templating in atmos.yaml
CLI config file
Templating in Atmos stack manifests is configured in the atmos.yaml
CLI config file in the templates.settings
section:
# https://pkg.go.dev/text/template
templates:
settings:
enabled: true
# https://masterminds.github.io/sprig
sprig:
enabled: true
# https://docs.gomplate.ca
# https://docs.gomplate.ca/functions
gomplate:
enabled: true
# Timeout in seconds to execute the datasources
timeout: 5
# https://docs.gomplate.ca/datasources
datasources:
# 'http' datasource
# https://docs.gomplate.ca/datasources/#using-file-datasources
ip:
url: "https://api.ipify.org?format=json"
# https://docs.gomplate.ca/datasources/#sending-http-headers
# https://docs.gomplate.ca/usage/#--datasource-header-h
headers:
accept:
- "application/json"
# 'file' datasources
# https://docs.gomplate.ca/datasources/#using-file-datasources
config-1:
url: "./config1.json"
config-2:
url: "file:///config2.json"
# `aws+smp` AWS Systems Manager Parameter Store datasource
# https://docs.gomplate.ca/datasources/#using-awssmp-datasources
secret-1:
url: "aws+smp:///path/to/secret"
# `aws+sm` AWS Secrets Manager datasource
# https://docs.gomplate.ca/datasources/#using-awssm-datasource
secret-2:
url: "aws+sm:///path/to/secret"
# `s3` datasource
# https://docs.gomplate.ca/datasources/#using-s3-datasources
s3-config:
url: "s3://mybucket/config/config.json"
-
templates.settings.enabled
- a boolean flag to enable/disable the processing ofGo
templates in Atmos stack manifests. If set tofalse
, Atmos will not processGo
templates in stack manifests -
templates.settings.sprig.enabled
- a boolean flag to enable/disable the Sprig Functions in Atmos stack manifests -
templates.settings.gomplate.enabled
- a boolean flag to enable/disable the Gomplate Functions and Gomplate Datasources in Atmos stack manifests -
templates.settings.gomplate.timeout
- timeout in seconds to execute Gomplate Datasources -
templates.settings.gomplate.datasources
- a map of Gomplate Datasource definitions:-
The keys of the map are the datasource names, which are used in
Go
templates in Atmos stack manifests.
For example:terraform: vars: tags: provisioned_by_ip: '{{ (datasource "ip").ip }}' config1_tag: '{{ (datasource "config-1").tag }}' config2_service_name: '{{ (datasource "config-2").service.name }}'
-
The values of the map are the datasource definitions with the following schema:
-
url
- the Datasource URL -
headers
- a map of HTTP request headers for thehttp
datasource. The keys of the map are the header names. The values of the map are lists of values for the header.The following configuration will result in the
accept: application/json
HTTP header being sent with the HTTP request to the datasource:headers: accept: - "application/json"
-
-
Configuring templating in Atmos stack manifests
The settings.templates.settings
section can be defined globally per organization, tenant, account, or per component. Atmos deep-merges the configurations from all scopes into the final result using inheritance.
For example, define Gomplate Datasources for the entire organization in the stacks/orgs/acme/_defaults.yaml
stack manifest:
settings:
templates:
settings:
gomplate:
# 7 seconds timeout to execute the datasources
timeout: 7
# https://docs.gomplate.ca/datasources
datasources:
# 'file' datasources
# https://docs.gomplate.ca/datasources/#using-file-datasources
config-1:
url: "./my-config1.json"
config-3:
url: "file:///config3.json"
Atmos deep-merges the configurations from the settings.templates.settings
section in Atmos stack manifests with the templates.settings
section in atmos.yaml
CLI config file using inheritance.
The settings.templates.settings
section in Atmos stack manifests takes precedence over the templates.settings
section in atmos.yaml
CLI config file, allowing you to define the global datasources
in atmos.yaml
and then add or override datasources
in Atmos stack manifests for the entire organization, tenant, account, or per component.
For example, taking into account the configurations described above in atmos.yaml
CLI config file and in the stacks/orgs/acme/_defaults.yaml
stack manifest, the final datasources
map will look like this:
gomplate:
timeout: 7
datasources:
ip:
url: "https://api.ipify.org?format=json"
headers:
accept:
- "application/json"
config-1:
url: "./my-config1.json"
config-2:
url: "file:///config2.json"
config-3:
url: "file:///config3.json"
Note that the config-1
datasource from atmos.yaml
was overridden with the config-1
datasource from the stacks/orgs/acme/_defaults.yaml
stack manifest. The timeout
attribute was overridden as well.
You can now use the datasources
in Go
templates in all Atmos sections that support Go
templates.
Atmos sections supporting Go
templates
You can use Go
templates in the following Atmos sections to refer to values in the same or other sections:
vars
settings
env
metadata
providers
overrides
backend
backend_type
For example, let's say we have the following component configuration using Go
templates:
component:
terraform:
vpc:
settings:
setting1: 1
setting2: 2
setting3: "{{ .vars.var3 }}"
setting4: "{{ .settings.setting1 }}"
component: vpc
backend_type: s3
region: "us-east-2"
assume_role: "<role-arn>"
backend_type: "{{ .settings.backend_type }}"
metadata:
component: "{{ .settings.component }}"
providers:
aws:
region: "{{ .settings.region }}"
assume_role: "{{ .settings.assume_role }}"
env:
ENV1: e1
ENV2: "{{ .settings.setting1 }}-{{ .settings.setting2 }}"
vars:
var1: "{{ .settings.setting1 }}"
var2: "{{ .settings.setting2 }}"
var3: 3
# Add the tags to all the resources provisioned by this Atmos component
tags:
atmos_component: "{{ .atmos_component }}"
atmos_stack: "{{ .atmos_stack }}"
atmos_manifest: "{{ .atmos_stack_file }}"
region: "{{ .vars.region }}"
terraform_workspace: "{{ .workspace }}"
assumed_role: "{{ .providers.aws.assume_role }}"
description: "{{ .atmos_component }} component provisioned in {{ .atmos_stack }} stack by assuming IAM role {{ .providers.aws.assume_role }}"
# Examples of using the Sprig and Gom...
v1.69.0
In Atmos v1.55 (PR #515) we switched to using the TF_WORKSPACE
environment variable for selecting Terraform workspaces when issuing Terraform commands. This had the unintended consequence that the workspace was no longer being set for use outside of Atmos as a side effect. This version reverses that change, once again setting the workspace via terraform workspace select
so that it remains selected after Atmos finishes. Expand the details below for an extended explanation.
Use `terraform workspace select` to select workspace @Nuru (#580)
what
- Use
terraform workspace select
to select the workspace when needed - Do not ask for Read permissions for output files
- Only attempt
terraform workspace new
whenselect
returns with exit code 1; report any other kind of error
why
- Restores previous behavior, which includes fixing #574, and leaving the active workspace selected for use by
terraform
commands outside of Atmos - Prevents users running under
atmos terraform shell
from having issues runningterraform workspace
commands or having the workspace remain selected after changing directories - Atmos does not need Read permission on outputs, and redirected outputs may not be readable, causing unnecessary failures
- Avoid hiding unexpected errors
discussion
While Atmos is a powerful tool and for some there is a desire to make Atmos fully capable of replacing the terraform
CLI entirely, we are not there yet. For now, there is a need to be able to use terraform
alongside with atmos
in some edge cases. Also, for some, there is a desire to bypass some of Atmos' overhead when running repeated tasks (especially failing tasks) by invoking terraform
commands directly.
The issue of selecting a Terraform "workspace" highlights the kind of difficulty one runs into when designing a solution where Atmos and Terraform can work side by side. Terraform has the concept of, for each component, a "current workspace", which is a segregated Terraform state. Cloud Posse uses a different workspace for each deployment of a component (account and region and, where the component is deployed multiple times in the same account and region, each separate deployment).
The problem with a workspace selection is that it is stateful, modal, and mostly hidden. When you run a terraform
command, the component it applies to is determined by the current working directory (one form of mode/state), and the workspace it uses is determined by either:
- The most recent invocation of
terraform workspace select
in that directory - The value of the environment variable
TF_WORKSPACE
with the environment variable taking precedence.
To further complicate matters, if you run terraform workspace select foo
and the workspace foo
has not been created, you get an error. Likewise, if you run TF_WORKSPACE=foo terraform init -reconfigure
and the workspace foo
does not exist, you get an error. But if you run TF_WORKSPACE=foo terraform plan
and the workspace foo
does not exist, it will silently and automatically be created for you (so beware of typos).
Whichever mechanism you choose, you are setting some kind of persistent state which affects what terraform
commands affect, and it becomes easy to get lost as to what you are actually working on. Atmos handles this by always setting the state for its Terraform commands, which is admirable, and it can be more helpful to people using terraform
directly if it helps those users manage this hidden state effectively.
Prior to Atmos version 1.55, its behavior was that if it ran any Terraform command that applied to a specific workspace (anything other than init
), it selected the workspace, creating it if needed, and left the workspace selected. This means that if you working in the component directory, then any terraform
commands like terraform state show
would work on the component in the workspace of the stack you most recently used for that component. After running atmos terraform plan <component> -s <stack>
, you could do everything else using plain Terraform commands if you wanted to (the only trick being that, for some commands, you would need to supply the -var-file
that Atmos generated). While many users would never do things this way, in some use cases it can be particularly time saving. For example, you may want to run the same set of Terraform commands repeatedly on the same component, using different stacks, as can happen when migrating a component to a new version requires some manual steps.
As of Atmos 1.55, Atmos changed how it selects a workspace. It started using the environment variable TF_WORKSPACE
instead of terraform workspace select
. While this works for Atmos, it changes the behavior for running the bare terraform
commands. No longer did Atmos set the workspace for you, nor would atmos terraform workspace <component> -s <stack>
set the workspace for you, either. In fact, there was no longer any direct way to find out the correct workspace name to use. Your plain terraform
commands worked on the default workspace, or whatever one was selected the last time you used an old version of Atmos.
So in summary, the change made using terraform
harder to use, without appreciably making atmos
easier to use. The change did eliminate a bug that was causing problems, and the consequences were not obvious (nor universally considered negative), so it was made. This PR fixes the bug a different way and restores the behavior of v1.54 and earlier.
In the long run, we are seeking ways to make these kinds of behaviors optional and easier to choose. For now, we are just reverting to the "tried-and-true" behavior to avoid further surprises.
references
v1.68.0
Add `gomplate` templating engine to Atmos stack manifests. Update docs @aknysh (#578)
Breaking changes
If you used Go
templates in Atmos stack manifest before, they were enabled by default.
Starting with this release, you have to explicitly enable Go
templating in atmos.yaml
and enable either Sprig or Gomplate (or both).
See the description below.
what
- Add
gomplate
templating engine to Atmos stack manifests - Update docs:
why
-
Allow using the rich collection of the Gomplate Functions in Atmos stack manifest
-
NOTE: The Gomplate Datasources, including configuring the datasources in Atmos stack manifests and using Inheritance for datasources, will be added in the follow up PR
Description
Configuration
Templating in Atmos stack manifests is configured in the atmos.yaml
CLI config file in the templates
section:
# https://pkg.go.dev/text/template
templates:
settings:
enabled: true
# https://masterminds.github.io/sprig
sprig:
enabled: true
# https://docs.gomplate.ca
gomplate:
enabled: true
-
templates.settings.enabled
- a boolean flag to enable/disable the processing ofGo
templates in Atmos stack manifests. If set
tofalse
, Atmos will not processGo
templates in stack manifests -
templates.settings.sprig.enabled
- a boolean flag to enable/disable the Sprig Functions
in Atmos stack manifests -
templates.settings.gomplate.enabled
- a boolean flag to enable/disable the Gomplate Functions
in Atmos stack manifests
Atmos sections supporting Go
templates
You can use Go
templates in the following Atmos sections to refer to values in the same or other sections:
vars
settings
env
metadata
providers
overrides
backend
backend_type
For example, let's say we have the following component configuration using Go
templates:
component:
terraform:
vpc:
settings:
setting1: 1
setting2: 2
setting3: "{{ .vars.var3 }}"
setting4: "{{ .settings.setting1 }}"
component: vpc
backend_type: s3
region: "us-east-2"
assume_role: "<role-arn>"
backend_type: "{{ .settings.backend_type }}"
metadata:
component: "{{ .settings.component }}"
providers:
aws:
region: "{{ .settings.region }}"
assume_role: "{{ .settings.assume_role }}"
env:
ENV1: e1
ENV2: "{{ .settings.setting1 }}-{{ .settings.setting2 }}"
vars:
var1: "{{ .settings.setting1 }}"
var2: "{{ .settings.setting2 }}"
var3: 3
# Add the tags to all the resources provisioned by this Atmos component
tags:
atmos_component: "{{ .atmos_component }}"
atmos_stack: "{{ .atmos_stack }}"
atmos_manifest: "{{ .atmos_stack_file }}"
region: "{{ .vars.region }}"
terraform_workspace: "{{ .workspace }}"
assumed_role: "{{ .providers.aws.assume_role }}"
description: "{{ .atmos_component }} component provisioned in {{ .atmos_stack }} stack by assuming IAM role {{ .providers.aws.assume_role }}"
# Examples of using the Gomplate and Sprig functions
# https://docs.gomplate.ca/functions/strings
atmos_component_description: "{{ strings.Title .atmos_component }} component {{ .vars.name | strings.Quote }} provisioned in the stack {{ .atmos_stack | strings.Quote }}"
# https://masterminds.github.io/sprig/os.html
provisioned_by_user: '{{ env "USER" }}'
When executing Atmos commands like atmos describe component
and atmos terraform plan/apply
, Atmos processes all the template tokens in the manifest and generates the final configuration for the component in the stack:
settings:
setting1: 1
setting2: 2
setting3: 3
setting4: 1
component: vpc
backend_type: s3
region: us-east-2
assume_role: <role-arn>
backend_type: s3
metadata:
component: vpc
providers:
aws:
region: us-east-2
assume_role: <role-arn>
env:
ENV1: e1
ENV2: 1-2
vars:
var1: 1
var2: 2
var3: 3
tags:
assumed_role: <role-arn>
atmos_component: vpc
atmos_component_description: Vpc component "common" provisioned in the stack "plat-ue2-dev"
atmos_manifest: orgs/acme/plat/dev/us-east-2
atmos_stack: plat-ue2-dev
description: vpc component provisioned in plat-ue2-dev stack by assuming IAM role <role-arn>
provisioned_by_user: <user>
region: us-east-2
terraform_workspace: plat-ue2-dev
Use-cases
While Go
templates in Atmos stack manifests offer great flexibility for various use-cases, one of the obvious use-cases is to add a standard set of tags to all the resources in the infrastructure.
For example, by adding this configuration to the stacks/orgs/acme/_defaults.yaml
Org-level stack manifest:
terraform:
vars:
tags:
atmos_component: "{{ .atmos_component }}"
atmos_stack: "{{ .atmos_stack }}"
atmos_manifest: "{{ .atmos_stack_file }}"
terraform_workspace: "{{ .workspace }}"
# Examples of using the Gomplate and Sprig functions
# https://docs.gomplate.ca/functions/strings
atmos_component_description: "{{ strings.Title .atmos_component }} component {{ .vars.name | strings.Quote }} provisioned in the stack {{ .atmos_stack | strings.Quote }}"
# https://masterminds.github.io/sprig/os.html
provisioned_by_user: '{{ env "USER" }}'
The tags will be processed and automatically added to all the resources provisioned in the infrastructure.
Excluding templates from processing by Atmos
If you need to provide Go
templates to external systems (e.g. ArgoCD or Datadog) verbatim and prevent Atmos from processing the templates, use double curly braces + backtick + double curly braces instead of just double curly braces:
{{`{{ instead of {{
}}`}} instead of }}
For example:
components:
terraform:
eks/argocd:
metadata:
component: "eks/argocd"
vars:
enabled: true
name: "argocd"
chart_repository: "https://argoproj.github.io/argo-helm"
chart_version: 5.46.0
chart_values:
template-github-commit-status:
message: |
Application {{`{{ .app.metadata.name }}`}} is now running new version.
webhook:
github-commit-status:
method: POST
path: "/repos/{{`{{ call .repo.FullNameByRepoURL .app.metadata.annotations.app_repository }}`}}/statuses/{{`{{ .app.metadata.annotations.app_commit }}`}}"
body: |
{
{{`{{ if eq .app.status.operationState.phase "Running" }}`}} "state": "pending"{{`{{end}}`}}
{{`{{ if eq .app.status.operationState.phase "Succeeded" }}`}} "state": "success"{{`{{end}}`}}
{{`{{ if eq .app.status.operationState.phase "Error" }}`}} "state": "error"{{`{{end}}`}}
{{`{{ if eq .app.status.operationState.phase "Failed" }}`}} "state": "error"{{`{{end}}`}},
"description": "ArgoCD",
"target_url": "{{`{{ .context.argocdUrl }}`}}/applications/{{`{{ .app.metadata.name }}`}}",
"context": "continuous-delivery/{{`{{ .app.metadata.name }}`}}"
}
When Atmos processes the templates in the manifest shown above, it renders them as raw strings allowing sending the templates to the external system for processing:
chart_values:
template-github-commit-status:
message: |
Application {{ .app.metadata.name }} is now running new version.
webhook:
github-commit-status:
method: POST
path: "/repos/{{ call .repo.FullNameByRepoURL .app.metadata.annotations.app_repository }}/statuses/{{ .app.metadata.annotations.app_commit }}"
body: |
{
{{ if eq .app.status.operationState.phase "Running" }} "state": "pending"{{end}}
{{ if eq .app.status.operationState.phase "Succeeded" }} "state": "success"{{end}}
{{ if eq .app.status.operationState.phase "Error" }} "state": "error"{{end}}
{{ if eq .app.status.operationState.phase "Failed" }} "state": "error"{{end}},
"description": "ArgoCD",
"target_url": "{{ .context.argocdUrl }}/applications/{{ .app.metadata.name }}",
"context": "continuous-delivery/{{ .app.metadata.name }}"
}
The printf
template function is also supported and can be used instead of double curly braces + backtick + double curly braces.
The following examples produce the same result:
chart_values:
template-github-commit-status:
message: >-
Application {{`{{ .app.metadata.name }}`}} is now running new version.
chart_values:
template-github-commit-status:
message: "Application {{`{{ .app.metadata.name }}`}} is now running new version."
chart_values:
template-github-commit-status:
message: >-
{{ printf "Application {{ .app.metadata.name }} is now running new version." ...
v1.67.0
Add Terraform Cloud backend. Add/update docs @aknysh (#572)
what
- Add Terraform Cloud backend
- Add docs:
- Terraform Workspaces in Atmos
- Terraform Backends. Describes how to configure backends for AWS S3, Azure Blob Storage, Google Cloud Storage, and Terraform Cloud
why
- Allow configuring and using the Terraform Cloud backend
- Document existing and new Atmos features
docs
Terraform Cloud backend uses a cloud
block to specify which organization and workspace(s) to use.
To configure the Terraform Cloud backend in Atmos, add the following config to an Atmos manifest in _defaults.yaml
:
terraform:
backend_type: cloud
backend:
cloud:
organization: "my-org"
hostname: "app.terraform.io"
workspaces:
# Parameters for workspaces
For each component, you can optionally specify the workspaces.name
parameter similar to the following:
components:
terraform:
my-component:
# Optional backend configuration for the component
backend:
cloud:
workspaces:
name: "my-component-workspace"
If auto_generate_backend_file
is set to true
in the atmos.yaml
CLI config file in the components.terraform
section,
Atmos will deep-merge the backend configurations from the _defaults.yaml
manifests and from the component itself, and will generate a backend config JSON file backend.tf.json
in the component's folder, similar to the following example:
{
"terraform": {
"cloud": {
"hostname": "app.terraform.io",
"organization": "my-org",
"workspaces": {
"name": "my-component-workspace"
}
}
}
}
Instead of specifying the workspaces.name
parameter for each component in the component manifests, you can use
the {terraform_workspace}
token in the cloud
backend config in the _defaults.yaml
manifest.
The token {terraform_workspace}
will be automatically replaced by Atmos with the Terraform workspace for each component. This will make the entire configuration DRY.
terraform:
backend_type: cloud
backend:
cloud:
organization: "my-org"
hostname: "app.terraform.io"
workspaces:
# The token `{terraform_workspace}` will be automatically replaced with the
# Terraform workspace for each Atmos component
name: "{terraform_workspace}"
v1.66.0
Add `stacks.name_template` section to `atmos.yaml`. Add `Go` templating to Atmos stack manifests @aknysh (#560)
what
- Add
stacks.name_template
section toatmos.yaml
- Add
Go
templating to Atmos stack manifests - Update docs
why
-
Add
stacks.name_template
section toatmos.yaml
-stacks.name_template
section allows greater flexibility in defining the naming convention for the top-level Atmos stacks-
stacks.name_pattern
configures the name pattern for the top-level Atmos stacks using the context variablesnamespace
,tenant
,environment
andstage
as the tokens. Depending on the structure of your organization, OUs, accounts and regions, setstacks.name_pattern
to the following:-
name_pattern: {stage}
- if you use just one region and a few accounts (stages) in just one organization and one OU. In this case, the top-level Atmos stacks will use just thestage
(account) in their names, and to provision the Atmos components in the top-level stacks, you will be executing Atmos commands likeatmos terraform apply <component> --stack dev
,atmos terraform apply <component> --stack staging
andatmos terraform apply <component> --stack prod
-
name_pattern: {environment}-{stage}
- if you have multiple regions and accounts (stages) in just one organization and one OU. In this case, the top-level Atmos stacks will use theenvironment
(region) andstage
(account) in their names, and to provision the Atmos components in the top-level stacks, you will be executing Atmos commands likeatmos terraform apply <component> --stack ue2-dev
,atmos terraform apply <component> --stack uw2-staging
andatmos terraform apply <component> --stack ue1-prod
. Note that thename_pattern
can also be defined as{stage}-{environment}
, in which case the Atmos commands will look likeatmos terraform apply <component> --stack dev-ue2
-
name_pattern: {tenant}-{environment}-{stage}
- if you have multiple regions, OUs (tenants) and accounts (stages) in just one organization. In this case, the top-level Atmos stacks will use thetenant
,environment
(region) andstage
(account) in their names, and to provision the Atmos components in the top-level stacks, you will be executing Atmos commands likeatmos terraform apply <component> --stack plat-ue2-dev
,atmos terraform apply <component> --stack core-uw2-staging
andatmos terraform apply <component> --stack plat-ue1-prod
, whereplat
andcore
are the OUs/tenants in your organization -
name_pattern: {namespace}-{tenant}-{environment}-{stage}
- if you have a multi-org, multi-tenant, multi-account and multi-region architecture. In this case, the top-level Atmos stacks will use thenamespace
,tenant
,environment
(region) andstage
(account) in their names, and to provision the Atmos components in the top-level stacks, you will be executing Atmos commands likeatmos terraform apply <component> --stack org1-plat-ue2-dev
,atmos terraform apply <component> --stack org2-core-uw2-staging
andatmos terraform apply <component> --stack org2-plat-ue1-prod
, whereorg1
andorg2
are the organization names (defined asnamespace
in the corresponding_defaults.yaml
config files for the organizations)
-
-
stacks.name_template
serves the same purpose asstacks.name_pattern
(defines the naming convention for the top-level Atmos stacks), but provides much more functionality. Instead of using the predefined context variables as tokens, it uses Go templates. Sprig Functions are supported as well-
For the
Go
template tokens, and you can use any Atmos sections (e.g.vars
,providers
,settings
)
that the Atmos commandatmos describe component <component> -s <stack>
generates for a component in a stack. -
name_template: "{{.vars.tenant}}-{{.vars.environment}}-{{.vars.stage}}"
defines the same name pattern for the top-level Atmos stacks asname_pattern: "{tenant}-{environment}-{stage}"
does -
Since
stacks.name_template
allows using any variables form thevars
section (and other sections), you can define
your own naming convention for your organization or for different clouds (AWS, Azure, GCP). For example, in the
corresponding_defaults.yaml
stack manifests, you can use the following variables:org
instead ofnamespace
division
instead oftenant
region
instead ofenvironment
account
instead ofstage
Then define the following
stacks.name_template
inatmos.yaml
:stacks: name_template: "{{.vars.division}}-{{.vars.account}}-{{.vars.region}}"
You will be able to execute all Atmos commands using the newly defined naming convention:
atmos terraform plan <component> -s <division-account-region> atmos terraform apply <component> -s <division-account-region> atmos describe component <component> -s <division-account-region>
NOTE:
Use eitherstacks.name_pattern
orstacks.name_template
to define the naming convention for the top-level Atmos stacks.stacks.name_template
has higher priority. Ifstacks.name_template
is specified,stacks.name_pattern
will be ignored.
-
-
-
Add
Go
templating to Atmos stack manifestsAtmos supports Go templates in stack manifests. Sprig Functions are supported as well.
You can use
Go
templates in the following Atmos section to refer to values in the same or other sections:vars
settings
env
metadata
providers
overrides
backend
backend_type
NOTE:
In the template tokens, you can refer to any value in any section that the Atmos commandatmos describe component <component> -s <stack>
generates.For example, let's say we have the following component configuration using
Go
templates:component: terraform: vpc: settings: setting1: 1 setting2: 2 setting3: "{{ .vars.var3 }}" setting4: "{{ .settings.setting1 }}" component: vpc backend_type: s3 region: "us-east-2" assume_role: "<role-arn>" backend_type: "{{ .settings.backend_type }}" metadata: component: "{{ .settings.component }}" providers: aws: region: "{{ .settings.region }}" assume_role: "{{ .settings.assume_role }}" env: ENV1: e1 ENV2: "{{ .settings.setting1 }}-{{ .settings.setting2 }}" vars: var1: "{{ .settings.setting1 }}" var2: "{{ .settings.setting2 }}" var3: 3 # Add the tags to all the resources provisioned by this Atmos component tags: atmos_component: "{{ .atmos_component }}" atmos_stack: "{{ .atmos_stack }}" atmos_manifest: "{{ .atmos_stack_file }}" region: "{{ .vars.region }}" terraform_workspace: "{{ .workspace }}" assumed_role: "{{ .providers.aws.assume_role }}" description: "{{ .atmos_component }} component provisioned in {{ .atmos_stack }} stack by assuming IAM role {{ .providers.aws.assume_role }}" # `provisioned_at` uses the Sprig functions # https://masterminds.github.io/sprig/date.html # https://pkg.go.dev/time#pkg-constants provisioned_at: '{{ dateInZone "2006-01-02T15:04:05Z07:00" (now) "UTC" }}'
When executing Atmos commands like
atmos describe component
andatmos terraform plan/apply
, Atmos processes all the template tokens in the manifest and generates the final configuration for the component in the stack:settings: setting1: 1 setting2: 2 setting3: 3 setting4: 1 component: vpc backend_type: s3 region: us-east-2 assume_role: <role-arn> backend_type: s3 metadata: component: vpc providers: aws: region: us-east-2 assume_role: <role-arn> env: ENV1: e1 ENV2: 1-2 vars: var1: 1 var2: 2 var3: 3 tags: assumed_role: <role-arn> atmos_component: vpc atmos_manifest: orgs/acme/plat/dev/us-east-2 atmos_stack: plat-ue2-dev description: vpc component provisioned in plat-ue2-dev stack by assuming IAM role <role-arn> provisioned_at: "2024-03-12T16:18:24Z" region: us-east-2 terraform_workspace: plat-ue2-dev
While
Go
templates in Atmos stack manifests offer great flexibility for various use-cases, one of the obvious use-cases
is to add a standard set of tags to all the resources in the infrastructure.For example, by adding this configuration to the
stacks/orgs/acme/_defaults.yaml
Org-level stack manifest:terraform: vars: tags: atmos_component: "{{ .atmos_component }}" atmos_stack: "{{ .atmos_stack }}" atmos_manifest: "{{ .atmos_stack_file }}" terraform_workspace: "{{ .workspace }}" provisioned_at: '{{ dateInZone "2006-01-02T15:04:05Z07:00" (now) "UTC" }}'
the tags will be processed and automatically added to all the resources provisioned in the infrastructure.