Terraform test: Add AWS persistent

[def-]

Test run: https://buildkite.com/materialize/qa-canary/builds/338

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

[def-]

@bobbyiliev The test works locally, but fails in CI: https://buildkite.com/materialize/qa-canary/builds/338#0193bc0c-9a96-41f3-a136-927eba3f5477

An error occurred (AccessDeniedException) when calling the DescribeCluster operation: User: arn:aws:sts::834237029485:assumed-role/buildkite-aarch64-small-d306b64-Role/i-0346d14d0e83e2d8c is not authorized to perform: eks:DescribeCluster on resource: arn:aws:eks:us-east-1:834237029485:cluster/aws-persistent-cluster

Can I easily tell terraform that anyone in the Materialize org can have access to this cluster?

[def-]

@bobbyiliev There seems to be another problem. I think the security group and subnet are not using a prefix. Now that I added the persistent aws setup in this PR the temporary aws terraform setup is failing: https://buildkite.com/materialize/nightly/builds/10674#0193bc9c-f90a-45eb-8494-11b525a7a606

│ Error: deleting Security Group (sg-0dfe7eb420640183b): operation error EC2: DeleteSecurityGroup, https response error StatusCode: 400, RequestID: eba72284-122d-4bdb-8c17-fdb6cca811ea, api error DependencyViolation: resource sg-0dfe7eb420640183b has a dependent object
│ Error: deleting EC2 Subnet (subnet-04a826bd3eab3b285): operation error EC2: DeleteSubnet, https response error StatusCode: 400, RequestID: 332610d1-f9a5-4df5-ae4c-cd430fd29596, api error DependencyViolation: The subnet 'subnet-04a826bd3eab3b285' has dependencies and cannot be deleted.

Can you take a look please?

[def-]

I also tried granting the CI role permissions to the EKS cluster, but still seeing the same:

An error occurred (AccessDeniedException) when calling the DescribeCluster operation: User: arn:aws:sts::834237029485:assumed-role/buildkite-aarch64-small-d306b64-Role/i-00838a09531c9f213 is not authorized to perform: eks:DescribeCluster on resource: arn:aws:eks:us-east-1:834237029485:cluster/aws-persistent-cluster

[bobbyiliev]

@bobbyiliev There seems to be another problem. I think the security group and subnet are not using a prefix. Now that I added the persistent aws setup in this PR the temporary aws terraform setup is failing: https://buildkite.com/materialize/nightly/builds/10674#0193bc9c-f90a-45eb-8494-11b525a7a606

Just submitted a PR to update a hardcoded prefix.

I also tried granting the CI role permissions to the EKS cluster, but still seeing the same:

Is there an easy way for us to get the CI role during the run itself? I think that we could extend the terraform module to accept an extra parameter and add the role to the cluster dynamically.