Provider wants to replace existing kafka_acls when a new entry/ creating new resource is being added.

[thennati]

Hi @Mongey ,

thank you for amazing tool. we're seeing an issue when we try to create a new resource, or update the resource its trying to destroying existing acls and creating new acls. is there any fix you can provide us. Really appreciate your help on this.
im using 0.6.0 version, using Amazon Managed Kafka. running terraform in an EC2
Terraform v1.5.7
on linux_amd64
output:
build 11-Apr-2024 09:45:06 �[1m # kafka_acl.acl[61]�[0m will be created
build 11-Apr-2024 09:45:06 �[0m �[32m+�[0m�[0m resource "kafka_acl" "acl" {
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_host = ""
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_operation = "Read"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_permission_type = "Allow"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_principal = "User:CN=test.user"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m id = (known after apply)
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m resource_name = "test_dev"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m resource_pattern_type_filter = "Literal"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m resource_type = "Group"
build 11-Apr-2024 09:45:06 }
build 11-Apr-2024 09:45:06
build 11-Apr-2024 09:45:06 �[1m # kafka_acl.acl[62]�[0m will be created
build 11-Apr-2024 09:45:06 �[0m �[32m+�[0m�[0m resource "kafka_acl" "acl" {
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_host = ""
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_operation = "Read"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_permission_type = "Allow"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_principal = "User:CN=test.user"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m id = (known after apply)
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m resource_name = "kafka_topic"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m resource_pattern_type_filter = "Prefixed"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m resource_type = "Topic"
build 11-Apr-2024 09:45:06 }
build 11-Apr-2024 09:45:06
build 11-Apr-2024 09:45:06 �[1mPlan:�[0m 58 to add, 0 to change, 55 to destroy.

few open and closed issues
#285
#55 (comment)

[Mongey]

Hi @thennati can you provide the full terraform you are using, and what version of Kafka this is on.

Thanks!

[thennati]

MSK kafka version : 3.5.1

provider.tf :

terraform {
  required_version = ">=1.1.5"
  required_providers {
    kafka = {
      source = "Mongey/kafka"
    }
  }
}
provider "kafka" {
  bootstrap_servers = [var.bootstrap_servers]
  ca_cert           = file("${path.cwd}/../certs/ca-chain.crt")
  client_cert       = file("${path.cwd}/../certs/msk-acls.pem")
  client_key        = file("${path.cwd}/../certs/private.key")
  tls_enabled       = true
}

terraform {
  backend "s3" {
  }
}

main.tf :

locals {
  acl_pairs = flatten([
    for acl_pair in var.kafka_acl : [
      for name in acl_pair.resource_name : {
        resource_name = name
        resource_type = acl_pair.resource_type
        acl_principal = acl_pair.acl_principal
        acl_operation = acl_pair.acl_operation
        acl_permission_type = acl_pair.acl_permission_type
        resource_pattern_type_filter = acl_pair.resource_pattern_type_filter
      }
    ]
  ])
}
resource "kafka_acl" "acl" {
    count = length(local.acl_pairs)
    resource_name       = local.acl_pairs[count.index].resource_name
    resource_type       = local.acl_pairs[count.index].resource_type
    acl_principal       = local.acl_pairs[count.index].acl_principal
    acl_host            = "*"
    acl_operation       = local.acl_pairs[count.index].acl_operation
    acl_permission_type = local.acl_pairs[count.index].acl_permission_type
    resource_pattern_type_filter = local.acl_pairs[count.index].resource_pattern_type_filter
}

variables.tf :

variable "kafka_acl" {
  type = map(object({
    resource_name       = list(string)
    resource_type       = string
    acl_principal       = string
    acl_operation       = string
    acl_permission_type = string
    resource_pattern_type_filter = string
  }))
  default = {
    "Consumer_1" = {
      resource_name       = ["syslog"]
      resource_type       = "Topic"
      acl_principal       = "User:Alice"
      acl_operation       = "Write"
      acl_permission_type = "Deny"
      resource_pattern_type_filter = "Literal"
    }
    "Consumer_2" = {
      resource_name       = ["syslog2"]
      resource_type       = "Topic"
      acl_principal       = "User:Alice"
      acl_operation       = "Read"
      acl_permission_type = "Deny"
      resource_pattern_type_filter = "Prefixed"
    }
  }

}
variable "bootstrap_servers" {
  type    = string
  default = "broker.kafka.ap-southeast-2.amazonaws.com:9094"
}

variable "ca-chain"{
  type = string
  default = "../certs/ca-chain.crt"
}

variable "cert"{
  type = string
  default = "../certs/msk-acls.pem"
}

variable "private_key" {
  type = string
  default = "../certs/private.key"
}

terrform.acls.tfvars :

  "kafka_ui_topic"= {
    resource_name       = ["*"]
    resource_type       = "Topic"
    acl_principal       = "User:CN=test_user_2"
    acl_operation       = "All"
    acl_permission_type = "Allow"
    resource_pattern_type_filter = "Literal"
  }
  "relay_acl_write"= {
    resource_name       = ["*"] #relay to access all the topics avaialble in the cluster
    resource_type       = "Topic"
    acl_principal       = "User:CN=test_user_1" 
    acl_operation       = "Write"
    acl_permission_type = "Allow"
    resource_pattern_type_filter = "Literal"
  }
  "relay_acl_read"= {
    resource_name       = ["*"] #relay to access all the topics avaialble in the cluster
    resource_type       = "Topic"
    acl_principal       = "User:CN=test_user"
    acl_operation       = "Read"
    acl_permission_type = "Allow"
    resource_pattern_type_filter = "Literal"
  }

[thennati]

please let us know if you still require any further info on this thanks @Mongey

[thennati]

@Mongey any update on this please ?

[Mongey]

@thennati I haven't had time to look at this yet

[thennati]

@Mongey can you please have a look at this reply from sarama
IBM/sarama#2885

[joaocc]

@Mongey could you please kindly confirm if this is planned to be addressed? If not, is the recommendation to use IAM (assuming it is fixed in 0.7.x)? Thanks