Investigate Terraform Plan terraform-advanced-analytics-workspaces-infrastructure repo

[jacek-dudek]

Plan Available at: https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/terraform-advanced-analytics-workspaces-infrastructure/-/jobs/1375144

1. Investigate the plan changes
2. Break changes down into categories: breaking changes, non-breaking changes, intended changes
   Some Notes:

• The LRS -> ZRS changes in the storage accounts should be investigated to not be breaking for the storage accounts. Let's make sure that these don't destroy and recreate anything.
• The resource tagging changes for:
  ~ "PrimaryProjectContact" = "[email protected]" -> "[email protected]"
  ~ "PrimaryTechnicalContact" = "[email protected]" -> "[email protected]"
  is non breaking, but how can we get rid of these from the Terraform plan because they add a bunch of annoying noise.
• For changes like, at the minimum, catalogue them if they are hard to understand and we could possibly review together https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/terraform-advanced-analytics-workspaces-infrastructure/-/jobs/1375144#L3674
• Take note of random metadata changes

3. For any breaking changes, create a follow up ticket, provide context in follow up ticket and we can investigate later.

[jacek-dudek]

This terraform plan is using this terraform version and these providers:

  Terraform v1.4.6
  on linux_amd64
  + provider registry.terraform.io/hashicorp/azurerm v3.35.0
  + provider registry.terraform.io/hashicorp/dns v3.3.2
  + provider registry.terraform.io/hashicorp/helm v2.10.1
  + provider registry.terraform.io/hashicorp/kubernetes v2.22.0
  + provider registry.terraform.io/hashicorp/local v2.4.0
  + provider registry.terraform.io/hashicorp/null v3.2.1
  + provider registry.terraform.io/hashicorp/random v3.5.1

[jacek-dudek]

The first thing that's done is downloading all the terraform module dependencies that are being referenced directly or indirectly by the dev_cc_00.tf file.

Upgrading modules...
Downloading git::https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/modules/terraform-azure-statcan-aaw-region-environment.git?ref=v2.0.3 for aaw_common_cc...
- aaw_common_cc in .terraform/modules/aaw_common_cc
Downloading git::https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/modules/terraform-azure-statcan-aaw-environment.git?ref=v2.3.18 for aaw_dev_cc_00...
- aaw_dev_cc_00 in .terraform/modules/aaw_dev_cc_00
Downloading git::https://github.com/statcan/terraform-azurerm-kubernetes-cluster-nodepool.git?ref=v1.0.3 for aaw_dev_cc_00.cloud_main_system_node_pool...
- aaw_dev_cc_00.cloud_main_system_node_pool in .terraform/modules/aaw_dev_cc_00.cloud_main_system_node_pool
...

[jacek-dudek]

Next there are a bunch of statements starting with module. or data. that indicate that something is being read (: Reading...) or (Read complete after <n>s) or has its state updated (: Refreshing state...)

Ok, apparently data blocks are for producing data that may change over time and are sourced from other entities not managed by terraform modules or variables.

I think the (: Refreshing state...) statements are explained by this section in terraform docs:
In Terraform, refreshing your state file updates Terraform's knowledge of your infrastructure, as represented in your state file, with the actual state of your infrastructure. Terraform plan and apply operations run an implicit in-memory refresh as part of their functionality, reconciling any drift from your state file before suggesting infrastructure changes. You can also update your state file without making modifications to your infrastructure using the -refresh-only flag for plan and apply operations.

[jacek-dudek]

I thought this statement in the terraform plan output (line 19) might have been the option regarding how far we go up in terms of terraform module dependencies:

Getting source from Git repository 00:01
* Fetching changes with git depth set to 50... *

But it's something else, it's a fetch command option that limits the retrieval of commit history from a remote repository, effectively creating a shallow copy with a specified number of commits. See explanation in git-book: https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---depthltdepthgt

[jacek-dudek]

Interpreting the plan changes:
(+, green) means adding a resource
(-, red) means removing a resource
(-/+, red,green) means replacing a resource
(~, orange) means modifying a resource
(lhs -> rhs, orange) means changing an attribute from lhs value to rhs value)

[jacek-dudek]

The first three plan changes are to resources of type "azurerm_private_dns_zone".
They're all changes to the value assigned to the tag named "PrimaryProjectContact".
Probably non-breaking changes.

Private DNS Zone can be used when you wish to use a custom domain name for your Azure resources rather than using the Azure provided domain names.

Link to resource type: https://learn.microsoft.com/en-us/azure/dns/dns-private-zone-terraform?tabs=azure-cli
Link to source file: https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/terraform-advanced-analytics-workspaces-infrastructure/-/blob/main/dns.tf?ref_type=heads

Next three plan changes are to resources of type "azurerm_private_dns_zone_virtual_network_link".
Same as above, they're changes to the value assigned to the tag named "PrimaryProjectContact".

Next two plan changes are to resources of type "azurerm_resource_group".
Again, they're tag value modifications.

Going to ignore the tag modifications from now on.

[jacek-dudek]

Summarizing the first bunch of resources that necessitate replacements:

resource "azurerm_advanced_threat_protection" "audit"
A change in target_id value.
Forces replacement.

resource "azurerm_monitor_diagnostic_setting" "kubernetes_audit"
A bunch of log entries removed and new ones added.

resource "azurerm_storage_account" "audit"
Change of account_replication_type: "LRS" -> "ZRS"
Forces replacement.

resource "azurerm_storage_account_network_rules" "audit"
Change of storage_account_id value.
Forces replacement.

Two resources of type "azurerm_virtual_network"
Remove ddos_protection_plan argument block.

resource "azurerm_advanced_threat_protection" "velero"
Change of target_resource_id value.
Forces replacement.

resource "azurerm_role_assignment" "velero_storage_key_operator"
Change of scope argument.
Forces replacement.

resource "azurerm_storage_account" "velero"
Change of account_replication_type: "LRS" -> "RAGZRS"
Forces replacement.

resource "azurerm_storage_account" "workflows"
Change of account_replication_type: "LRS" -> "ZRS"
Forces replacement.

resource "azurerm_storage_account_network_rules" "velero"
Change of storage_account_id.
Forces replacement.

[jacek-dudek]

Looking into how the terraform plan output could be made to ignore certain changes in state.
(Namely the changes in values of certain tags, like PrimaryProjectContact.)

Found there is a meta-argument in terraform named lifecycle.
Link to docs: https://developer.hashicorp.com/terraform/language/meta-arguments/lifecycle

It has a sub-argument named ignore-changes that specifies resource attributes that Terraform should ignore when planning updates.

So in our case I was hoping we could write something like this:

lifecycle {
  ignore-changes {
    tags [PrimaryProjectContact] 
  }
}

I was looking to apply it at the provider block level and have it effect all the resources from that provider.
However, terraform documentation states that it can be applied to resource blocks.
This is an open issue with people commenting along the same lines as what I was intending to do:
hashicorp/terraform#27360

I will try writing it at the provider level and see if the terraform code gets validated, maybe it will actually work...