Concurrent execution of bigip_ssl_key_cert fails

[MasakariDOR]

Environment

• TMOS/Bigip Version: BIG-IP 14.1.5.5
• Terraform Version: 1.5.7
• Terraform bigip provider Version: 1.22.0

Summary

Using the resource bigip_ssl_key_cert concurrently produces transaction errors. This can occur on multiple concurrent creates, updates or deletes.

Steps To Reproduce

Steps to reproduce the behavior:

1. Provide terraform resource config which you are facing trouble along with the output of it.
   We can create 4x certificate / key pairs on the bigip using the same source pem files. If using different pem files for each, we also encounter errors that keys/certs don't match!

resource "bigip_ssl_key_cert" "certificate" {
  lifecycle {
    create_before_destroy = true
  }
  provider = bigip.EDEV_ISD_DEV_LTM_RCC
  for_each = {
    "cert1" : {}
    "cert2" : {}
    "cert3" : {}
    "cert4" : {}
  }
  cert_name    = each.key
  cert_content = file("./cert.pem")
  key_name     = format("%s.key", each.key)
  key_content  = file("./key.pem")
  partition    = "Common"
}

Produces the following output

bigip_ssl_key_cert.certificate["cert2"]: Creating...
bigip_ssl_key_cert.certificate["cert1"]: Creating...
bigip_ssl_key_cert.certificate["cert3"]: Creating...
bigip_ssl_key_cert.certificate["cert4"]: Creating...
bigip_ssl_key_cert.certificate["cert2"]: Creation complete after 1s [id=cert2.key_cert2]
bigip_ssl_key_cert.certificate["cert1"]: Creation complete after 1s [id=cert1.key_cert1]
bigip_ssl_key_cert.certificate["cert3"]: Creation complete after 2s [id=cert3.key_cert3]
╷
│ Error: error while starting transaction: error encountered while starting transaction: Transaction CREATE operation is not allowed to be added to transaction.
│
│   with bigip_ssl_key_cert.certificate["cert4"],
│   on test1.tf line 1, in resource "bigip_ssl_key_cert" "certificate":
│    1: resource "bigip_ssl_key_cert" "certificate" {
│
╵

3. To get to know more about the issue, provide terraform debug logs
   terraform.log

4. To capture debug logs, export TF_LOG variable with debug ( export TF_LOG= DEBUG ) before
   runnning terraform apply/plan
   Attached above

5. As3/DO json along with the resource config( for AS3/DO resource issues )
   N/A

Expected Behavior

Parallel execution must be supported for TF resources. We cannot execute with parallelism=1 to cater for this resource as it will slow down the execution too much.

Actual Behavior

I have found that multiple (more likely for more resources) triggers an issue error while starting transaction: error encountered while starting transaction: Transaction CREATE operation is not allowed to be added to transaction.
I have also seen transactions attempt to use mismatching keys/certs and in one case, created the object on the bigip with entirely the wrong cert!
This has also caused unrecoverable issues where TF creates / deletes the resources but fails to update the state. In these cases, resources needed to be re-imported to state or deleted from the device to recover.

[RavinderReddyF5]

Hi @MasakariDOR , is this reproducible ? can you please share reproduction steps, we didn't see any issue in our testing:

Config:

resource "bigip_ssl_key_cert" "testkeycert" {
  lifecycle {
    create_before_destroy = true
  }
  for_each = {
    "cert1" : {}
    "cert2" : {}
    "cert3" : {}
    "cert4" : {}
  }
  partition    = "Common"
  cert_name    = each.key
  # cert_name    = format("%s.crt", each.key)
  cert_content = file("server1.crt")
  key_name     = format("%s.key", each.key)
  key_content  = file("server1.key")
}

TF Apply:

# bigip_ssl_key_cert.testkeycert["cert1"] will be created
  + resource "bigip_ssl_key_cert" "testkeycert" {
      + cert_content   = (sensitive value)
      + cert_full_path = (known after apply)
      + cert_name      = "cert1"
      + id             = (known after apply)
      + key_content    = (sensitive value)
      + key_full_path  = (known after apply)
      + key_name       = "cert1.key"
      + partition      = "Common"
    }

  # bigip_ssl_key_cert.testkeycert["cert2"] will be created
  + resource "bigip_ssl_key_cert" "testkeycert" {
      + cert_content   = (sensitive value)
      + cert_full_path = (known after apply)
      + cert_name      = "cert2"
      + id             = (known after apply)
      + key_content    = (sensitive value)
      + key_full_path  = (known after apply)
      + key_name       = "cert2.key"
      + partition      = "Common"
    }

  # bigip_ssl_key_cert.testkeycert["cert3"] will be created
  + resource "bigip_ssl_key_cert" "testkeycert" {
      + cert_content   = (sensitive value)
      + cert_full_path = (known after apply)
      + cert_name      = "cert3"
      + id             = (known after apply)
      + key_content    = (sensitive value)
      + key_full_path  = (known after apply)
      + key_name       = "cert3.key"
      + partition      = "Common"
    }

  # bigip_ssl_key_cert.testkeycert["cert4"] will be created
  + resource "bigip_ssl_key_cert" "testkeycert" {
      + cert_content   = (sensitive value)
      + cert_full_path = (known after apply)
      + cert_name      = "cert4"
      + id             = (known after apply)
      + key_content    = (sensitive value)
      + key_full_path  = (known after apply)
      + key_name       = "cert4.key"
      + partition      = "Common"
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

bigip_ssl_key_cert.testkeycert["cert2"]: Creating...
bigip_ssl_key_cert.testkeycert["cert4"]: Creating...
bigip_ssl_key_cert.testkeycert["cert3"]: Creating...
bigip_ssl_key_cert.testkeycert["cert1"]: Creating...
bigip_ssl_key_cert.testkeycert["cert3"]: Creation complete after 5s [id=cert3.key_cert3]
bigip_ssl_key_cert.testkeycert["cert4"]: Creation complete after 6s [id=cert4.key_cert4]
bigip_ssl_key_cert.testkeycert["cert2"]: Creation complete after 6s [id=cert2.key_cert2]
bigip_ssl_key_cert.testkeycert["cert1"]: Creation complete after 6s [id=cert1.key_cert1]

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.