google_bigquery_analytics_hub_listing_subscription: does not work if exchange / listing is in a different project from the subscriber project

[agonzalezandino]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.11.3
on amd64

• provider registry.terraform.io/hashicorp/google v6.28.0
• provider registry.terraform.io/hashicorp/google-beta v6.28.0

Affected Resource(s)

google_bigquery_analytics_hub_listing_subscription

Terraform Configuration

Unfortunately the issue is still present in version 6.28.0.

I've created the three resources (exchange, listing and dataset) to set up the subscription in project A by following the official documentation:

resource "google_bigquery_analytics_hub_data_exchange" "subscription" {
  project = "project-a"
  location         = "US"
  data_exchange_id = "my_data_exchange"
  display_name     = "my_data_exchange"
  description      = ""
}

resource "google_bigquery_dataset" "subscription" {
  project = "project-a"
  dataset_id                  = "my_dataset"
  friendly_name               = "my_dataset"
  description                 = ""
  location                    = "US"
}

resource "google_bigquery_analytics_hub_listing" "subscription" {
  project = "project-a"
  location         = "US"
  data_exchange_id = google_bigquery_analytics_hub_data_exchange.subscription.data_exchange_id
  listing_id       = "my_listing"
  display_name     = "my_listing"
  description      = ""

  bigquery_dataset {
    dataset = google_bigquery_dataset.subscription.id
  }
}

Then I've tried to create the subscription in project B, as follows:

resource "google_bigquery_analytics_hub_listing_subscription" "subscription" {
  location = "US"
  data_exchange_id = google_bigquery_analytics_hub_data_exchange.subscription.data_exchange_id
  listing_id       = google_bigquery_analytics_hub_listing.subscription.listing_id
  destination_dataset {
    description = "A test subscription"
    friendly_name = "👋"
    labels = {
      testing = "123"
    }
    location = "US"
    dataset_reference {
      dataset_id = "destination_dataset"
      project_id = google_bigquery_dataset.subscription.project
    }
  }
}

As a result, I get this error when running terraform apply:

google_bigquery_analytics_hub_listing_subscription.subscription: Creating...
╷
│ Error: Error creating ListingSubscription: googleapi: Error 404: Failed to find data exchange in projects/1099313479020: dataExchanges/my_data_exchange
│ 
│   with google_bigquery_analytics_hub_listing_subscription.subscription,
│   on main.tf line 30, in resource "google_bigquery_analytics_hub_listing_subscription" "subscription":
│   30: resource "google_bigquery_analytics_hub_listing_subscription" "subscription" {
│ 
╵

If I repeat the same process but this time creating the subscription under project A, everything works as expected.

I've also tried to replace the references to google_bigquery_analytics_hub_data_exchange.subscription.data_exchange_id and google_bigquery_analytics_hub_listing.subscription.listing_id with the full resource id projects/{{project}}/locations/{{location}}/dataExchanges/{{data_exchange_id}} and projects/{{project}}/locations/{{location}}/dataExchanges/{{data_exchange_id}}/listings/{{listing_id}}, respectively, but that also returns an error:

google_bigquery_analytics_hub_listing_subscription.subscription: Creating...
╷
│ Error: Error creating ListingSubscription: googleapi: got HTTP response code 404 with body: 
│ 
│   with google_bigquery_analytics_hub_listing_subscription.subscription,
│   on main.tf line 30, in resource "google_bigquery_analytics_hub_listing_subscription" "subscription":
│   30: resource "google_bigquery_analytics_hub_listing_subscription" "subscription" {
│ 
╵

Debug Output

No response

Expected Behavior

The subscription is successfully created under project B when the exchange, listing and dataset belong to project A.

Actual Behavior

Creating the subscription fails.

Steps to reproduce

Use the Terraform configuration above pointing to your testing GCP projects with required permissions and run terraform apply.

Important Factoids

No response

References

#21402

b/420674656

[c2thorn]

The top level project field directly refers to the project of the data exchange. It contributes to the URL of the HTTP request which needs a data exchange to point to.

The destination_dataset.dataset_reference.project_id is the project of the dataset. This is where the actual subscription object is created.

Unless I'm missing something, #21402 referred to when these two projects differ. I don't believe there is an option in the API to make the subscription itself separate from both the data exchange and dataset. @agonzalezandino does this make sense?

Our documentation should be corrected to reflect this distinction.

[jorgemur]

I dont think this is just a Documentation issue. there are still some issues regarding how Terraform manages this:

1. When you revoke this subcription from the UI , terraform keeps saying the subscription is still there.
2. When you delete the dataset that this resource creates and you start a plan it will say "No changes Your infrastructure matches the configuration"

[c2thorn]

I dont think this is just a Documentation issue. there are still some issues regarding how Terraform manages this:

1. When you revoke this subcription from the UI , terraform keeps saying the subscription is still there.
2. When you delete the dataset that this resource creates and you start a plan it will say "No changes Your infrastructure matches the configuration"

Is this specifically for cross-project subscriptions?

[jongoings]

I'm experiencing similar issues trying to add subscription from a separate project. Getting the following error on the terraform apply.
Error: Error creating ListingSubscription: googleapi: Error 400: Source dataset not found. This could mean that the source dataset was deleted and recreated during the lifetime of this request.

My terraform plan succeeds without issue.

[melinath]

Note from triage: @c2thorn could you take another look at this?

[c2thorn]

Note from triage: [@c2thorn](https://github.com/c2thorn) could you take another look at this?

I'll look into it this week. It is still my understanding summarized in #22222 (comment) that the original setup of this issue is not a valid use case for the resource.

However I'll look into #22222 (comment) and #22222 (comment) to see if they all have the same underlying issue.

@jongoings do you have a sample config that shows which projects you are using between google_bigquery_analytics_hub_data_exchange, google_bigquery_dataset, and google_bigquery_analytics_hub_listing?