v1.84.0

Add Atmos Pro integration to `atmos.yaml`. Add caching to `atmos.Component` template function. Implement `atmos.GomplateDatasource` template function @aknysh (#647)

## what

• Add Atmos Pro integration to atmos.yaml
• Add caching to atmos.Component template function
• Implement atmos.GomplateDatasource template function
• Update docs
  • atmos.Component
  • atmos.GomplateDatasource

why

• Add Atmos Pro integration to atmos.yaml. This is in addition to the functionality added in Add --upload flag to atmos describe affected command. If the Atmos Pro configuration is present in the integrations.pro section in atmos.yaml, it will be added in the config section when executing the atmos describe affected --upload=true command for further processing on the server

  {
       "base_sha": "6746ba4df9e87690c33297fe740011e5ccefc1f9",
       "head_sha": "5360d911d9bac669095eee1ca1888c3ef5291084",
       "owner": "cloudposse",
       "repo": "atmos",
       "config": {
         "timeout": 3,
          "events": [
             "pull_request": [
                {
                  "on": ["open", "synchronize", "reopen"],
                  "workflow": "atmos-plan.yml",
                  "dispatch_only_top_level_stacks": true
                },               
                {
                  "on": ["merged"],
                  "workflow": "atmos-apply.yaml",
                },               
              ],
              "release": [
              ]
           ]
       }
       "stacks": [
          {
            "component": "vpc",
            "component_type": "terraform",
            "component_path": "components/terraform/vpc",
            "stack": "plat-ue2-dev",
            "stack_slug": "plat-ue2-dev-vpc",
            "affected": "stack.vars",
            "included_in_dependents": false,
            "dependents": []
          }
      ]
   }

• Add caching to atmos.Component template function

  Atmos caches (in memory) the results of atmos.Component template function execution. If you call the function for the same component in a stack more than once, the first call will produce the result and cache it, and all the consecutive calls will just use the cached data. This is useful when you use the atmos.Component function for the same component in a stack in multiple places in Atmos stack manifests. It will speed up the function execution and stack processing.

  For example:

  components:
    terraform:
      test2:
        vars:
          tags:
            test: '{{ (atmos.Component "test" .stack).outputs.id }}'
            test2: '{{ (atmos.Component "test" .stack).outputs.id }}'
            test3: '{{ (atmos.Component "test" .stack).outputs.id }}'

  In the example, the test2 Atmos component uses the outputs (remote state) of the test Atmos component from the same stack. The template function {{ atmos.Component "test" .stack }} is executed three times (once for each tag).

  After the first execution, Atmos caches the result in memory (all the component sections, including the outputs), and reuses it in the next two calls to the function. The caching makes the stack processing about three times faster in this particular example. In a production environment where many components are used, the speedup can be even more significant.

• Implement atmos.GomplateDatasource template function

  The atmos.GomplateDatasource template function wraps the Gomplate Datasources and caches the results, allowing executing the same datasource many times without calling the external endpoint multiple times. It speeds up the datasource execution and stack processing, and can eliminate other issues with calling an external endpoint, e.g. timeouts and rate limiting.

  Usage

    {{ (atmos.GomplateDatasource "<alias>").<attribute> }}

  Caching the result of atmos.GomplateDatasource function

  Atmos caches (in memory) the results of atmos.GomplateDatasource template function execution. If you execute the function for the same datasource alias more than once, the first execution will call the external endpoint, produce the result and cache it. All the consecutive calls will just use the cached data. This is useful when you use the atmos.GomplateDatasource function for the same datasource alias in multiple places in Atmos stack manifests. It will speed up the function execution and stack processing.

  For example:

  settings:
    templates:
      settings:
        gomplate:
          timeout: 5
          datasources:
            ip:
              url: "https://api.ipify.org?format=json"
              headers:
                accept:
                  - "application/json"
  components:
    terraform:
      test:
        vars:
          tags:
            test1: '{{ (datasource "ip").ip }}'
            test2: '{{ (atmos.GomplateDatasource "ip").ip }}'
            test3: '{{ (atmos.GomplateDatasource "ip").ip }}'
            test4: '{{ (atmos.GomplateDatasource "ip").ip }}'

  In the example, we define a gomplate datasource ip and specify an external endpoint in the url parameter.

  We use the Gomplate datasource function in the tag test1, and the atmos.GomplateDatasource wrapper for the same datasource alias ip in the other tags. The atmos.GomplateDatasource wrapper will call the same external endpoint, but will cache the result and reuse it between the datasource invocations.

  When processing the component test from the above example, Atmos does the following:

  • Executes the {{ (datasource "ip").ip }} template. It calls the external endpoint using the HTTP protocol and assign the ip attribute from the result to the tag test1

  • Executes the {{ (atmos.GomplateDatasource "ip").ip }} template. It calls the external endpoint again, caches the result in memory, and assigns the ip attribute from the result to the tag test2

  • Executes the {{ (atmos.GomplateDatasource "ip").ip }} two more times for the tags test3 and test4. It detects that the result for the same datasource alias ip is already presend in the memory cache and reuses it without calling the external endpoint two more times

  The datasource result caching makes the stack processing much faster and significantly reduces the load on external endpoints, preventing such issues as timeouts and rate limiting.