Error when reading or editing GCS service account not found

[natea]

When I try to run the Terraform command terraform -chdir=terraform apply -auto-approve, the following errors occur:

│ Error: Error when reading or editing Project "616355888868": Get "https://cloudresourcemanager.googleapis.com/v1/projects/616355888868?alt=json&prettyPrint=false": oauth2: cannot fetch token: 400 Bad Request
│ Response: {
│   "error": "invalid_grant",
│   "error_description": "Bad Request"
│ }
│
│   with module.pdf_redactor.data.google_project.project,
│   on pdf-redactor/workflow-trigger.tf line 33, in data "google_project" "project":
│   33: data "google_project" "project" {
│
╵
╷
│ Error: Error when reading or editing GCS service account not found: Get "https://storage.googleapis.com/storage/v1/projects/616355888868/serviceAccount?alt=json&prettyPrint=false": oauth2: cannot fetch token: 400 Bad Request
│ Response: {
│   "error": "invalid_grant",
│   "error_description": "Bad Request"
│ }
│
│   with module.pdf_redactor.data.google_storage_project_service_account.gcs_account,
│   on pdf-redactor/workflow-trigger.tf line 35, in data "google_storage_project_service_account" "gcs_account":
│   35: data "google_storage_project_service_account" "gcs_account" {
│
╵

[natea]

I googled the error, and came across this answer on Stackoverflow:
https://stackoverflow.com/questions/70807862/how-to-solve-error-when-reading-or-editing-project-service-foo-container-google

After running the command recommended in that answer gcloud auth application-default login, and re-running the command terraform -chdir=terraform apply -auto-approve, I got further in the process, but then got new errors.

[natea]

The new error is:

│ Error: Error creating InspectTemplate: googleapi: Error 403: Your application is authenticating by using local Application Default Credentials. The dlp.googleapis.com API requires a quota project, which is not set by default. To learn how to set your quota project, see https://cloud.google.com/docs/authentication/adc-troubleshooting/user-creds .

I went to that page https://cloud.google.com/docs/authentication/troubleshoot-adc#user-creds-client-based and there it recommended running this command gcloud auth application-default set-quota-project $PROJECT

But even after running that command, and re-running the terraform -chdir=terraform apply -auto-approve command, I'm still getting the following errors:

╷
│ Error: Error creating Service: googleapi: Error 409: Resource 'dlp-runner-3067' already exists.
│
│   with module.pdf_redactor.google_cloud_run_v2_service.dlp_runner,
│   on pdf-redactor/dlp-runner.tf line 39, in resource "google_cloud_run_v2_service" "dlp_runner":
│   39: resource "google_cloud_run_v2_service" "dlp_runner" {
│
╵
╷
│ Error: Error creating InspectTemplate: googleapi: Error 403: Your application is authenticating by using local Application Default Credentials. The dlp.googleapis.com API requires a quota project, which is not set by default. To learn how to set your quota project, see https://cloud.google.com/docs/authentication/adc-troubleshooting/user-creds .
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/764086051850",
│       "service": "dlp.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│
│   with module.pdf_redactor.google_data_loss_prevention_inspect_template.dlp_pdf_template,
│   on pdf-redactor/dlp.tf line 15, in resource "google_data_loss_prevention_inspect_template" "dlp_pdf_template":
│   15: resource "google_data_loss_prevention_inspect_template" "dlp_pdf_template" {
│
╵
╷
│ Error: Error creating Service: googleapi: Error 409: Resource 'findings-writer-3067' already exists.
│
│   with module.pdf_redactor.google_cloud_run_v2_service.findings_writer,
│   on pdf-redactor/findings-writer.tf line 32, in resource "google_cloud_run_v2_service" "findings_writer":
│   32: resource "google_cloud_run_v2_service" "findings_writer" {
│
╵
╷
│ Error: Error creating Service: googleapi: Error 409: Resource 'pdf-merger-3067' already exists.
│
│   with module.pdf_redactor.google_cloud_run_v2_service.pdf_merger,
│   on pdf-redactor/pdf-merger.tf line 26, in resource "google_cloud_run_v2_service" "pdf_merger":
│   26: resource "google_cloud_run_v2_service" "pdf_merger" {
│
╵
╷
│ Error: Error creating Service: googleapi: Error 409: Resource 'pdf-splitter-3067' already exists.
│
│   with module.pdf_redactor.google_cloud_run_v2_service.pdf_splitter,
│   on pdf-redactor/pdf-splitter.tf line 26, in resource "google_cloud_run_v2_service" "pdf_splitter":
│   26: resource "google_cloud_run_v2_service" "pdf_splitter" {
│
╵

Can these errors be safely ignored since they're saying that the resource has already been created, and presumably Terraform is trying to create it again?

After running that

[felimartina]

Hey @natea - I'm the creator of this solution. I'm sorry you encountered these many issues when trying to deploy this solution.

Multiple terraform runs can lead to these duplicated resources, did you try running terraform -chdir=terraform destroy ? that should help remove all existing resources. Then you can start from scratch. You might also want to delete the terraofmr state and re-run terraform init. That will force the creation of a new random id for the resources (-3067 in your case).

Let me know if any of these tips help, while I take some time to review the deployment process. It seems that Google Cloud changed some of the default permissions and the behaviour of some gcloud calls and that broke the seamless installation.

It will take me a 2-3 weeks because I'm going ooo next week, though. Please bare with me.

If you find a solution or workaround please post it in here or submit a pull request.

[felimartina]

@natea - I made a few updates to the deployment process. Can you try running the deployment process from scratch on a new project?

Also make sure you are using a project with a linked Billing Account, since some of the APIs (DLP) might require you to have Billing Accounts on the project (I think that's related to the problem you are facing on quota).

[natea]

Hi @felimartina , thanks for your reply. I did as you suggested and removed all the existing resources and re-created them. But I'm still getting this error when running the terraform apply command:

│ Error: Error creating InspectTemplate: googleapi: Error 403: Your application is authenticating by using local Application Default Credentials. The dlp.googleapis.com API requires a quota project, which is not set by default. To learn how to set your quota project, see https://cloud.google.com/docs/authentication/adc-troubleshooting/user-creds .
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/764086051850",
│       "service": "dlp.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│
│   with module.pdf_redactor.google_data_loss_prevention_inspect_template.dlp_pdf_template,
│   on pdf-redactor/dlp.tf line 15, in resource "google_data_loss_prevention_inspect_template" "dlp_pdf_template":
│   15: resource "google_data_loss_prevention_inspect_template" "dlp_pdf_template" {
│
╵

[felimartina]

Hey @natea - #28 aims to fix some issues we've noticed with CloudStorage access.
Can you try again with a fresh project and let me know if you still face issues?