[ISSUE] Issue with databricks_quality_monitor resource

[pietern]

Configuration

terraform {
  required_providers {
    databricks = {
      source = "databricks/databricks"
    }
  }
}

data "databricks_current_user" "me" {
  // This data source is used to get the current user's username
}

resource "databricks_quality_monitor" "this" {
    //
    // CREATE TABLE main.pieter_noordhuis.quality_monitor_issue AS (
    //   SELECT * FROM RANGE(1000)
    // );
    //
    table_name = "main.pieter_noordhuis.quality_monitor_issue"
    output_schema_name = "main.pieter_noordhuis"
    assets_dir = "/Workspace/Users/${data.databricks_current_user.me.user_name}/quality_monitor_issue"

    snapshot {}
}

Expected Behavior

It should create the monitor.

Actual Behavior

First apply:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # databricks_quality_monitor.this will be created
  + resource "databricks_quality_monitor" "this" {
      + assets_dir                 = "/Workspace/Users/[email protected]/quality_monitor_issue"
      + dashboard_id               = (known after apply)
      + drift_metrics_table_name   = (known after apply)
      + id                         = (known after apply)
      + monitor_version            = (known after apply)
      + output_schema_name         = "main.pieter_noordhuis"
      + profile_metrics_table_name = (known after apply)
      + status                     = (known after apply)
      + table_name                 = "main.pieter_noordhuis.quality_monitor_issue"

      + snapshot {}
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

databricks_quality_monitor.this: Creating...
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to databricks_quality_monitor.this, provider "provider[\"registry.terraform.io/databricks/databricks\"]" produced an unexpected new value:
│ .data_classification_config: block count changed from 0 to 1.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵

Subsequent apply:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # databricks_quality_monitor.this will be updated in-place
  ~ resource "databricks_quality_monitor" "this" {
      ~ dashboard_id               = "01efa293c24a154a96a5ed9909008c7a" -> (known after apply)
      ~ drift_metrics_table_name   = "main.pieter_noordhuis.quality_monitor_issue_drift_metrics" -> (known after apply)
      ~ id                         = "main.pieter_noordhuis.quality_monitor_issue" -> (known after apply)
      ~ monitor_version            = "2" -> (known after apply)
      ~ profile_metrics_table_name = "main.pieter_noordhuis.quality_monitor_issue_profile_metrics" -> (known after apply)
      ~ status                     = "MONITOR_STATUS_ACTIVE" -> (known after apply)
        # (3 unchanged attributes hidden)

      - data_classification_config {
          - enabled = false -> null
        }

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

databricks_quality_monitor.this: Modifying... [id=main.pieter_noordhuis.quality_monitor_issue]
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to databricks_quality_monitor.this, provider "provider[\"registry.terraform.io/databricks/databricks\"]" produced an unexpected new value:
│ .data_classification_config: block count changed from 0 to 1.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Steps to Reproduce

1. Create a table in UC to reference
2. Update the ref in the repro example above
3. Run terraform apply

Terraform and provider versions

Provider v1.58.0.

Is it a regression?

Yes, this worked in v1.54.0.

[micheledaddetta-databricks]

Hello @pietern , I had the same issue. I was able to solve it setting the USE_SDK_V2_RESOURCES using the following command export USE_SDK_V2_RESOURCES="databricks_quality_monitor".
This issue is related to the quality monitor resource migration from sdkv2 to plugin framework.
However, it would be great to avoid this kind of setting, since in certain case the ones who manage the Terraform code are not the sames that manage the CICD pipeline.

[alexott]

@mgyucht will #4284 help here?