BigQuery cloud test for Iceberg catalogs

[rpdevmp]

Summary:

• Added BigQuery cloud test for Iceberg catalogs
• Currently it is using "key_value" Iceberg mode
• It supports using AWS S3 storage or GS GCP storage
• Tested multiple times with Ducktape on GCP cloud clusters
• Also, tested with bq tool on GCP and AWS clusters
• Running simple queries to verify Iceberg catalog data

Backports Required

• [x ] none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v24.3.x
• v24.2.x
• v24.1.x

Release Notes

• none

[rpdevmp]

• Ducktape tests were done on GCP BYOC cluster, with Iceberg enabled, key_value iceberg mode
• Need to add google-cloud-bigquery to Ducktape requirements, so its available on Ducktape node. As a workaround, used this command before running tests:

sudo /opt/.ducktape-venv/bin/python3 -m pip install google-cloud-bigquery

But this is NOT a blocker for this PR. We should be able to review and merge it as tests work

As a follow up, we could extend it with other supported modes, more topics, different data formats, etc. But those enhancements are NOT part of this PR.

[rpdevmp]

@andrewhsu @andrwng
More details cloud be found at:
https://redpandadata.atlassian.net/wiki/spaces/DEVPROD/pages/1004535815/BigQuery+with+Iceberg+Catalogs+on+Redpanda+cloud+clusters

[rpdevmp]

@travisdowns, Adding you as a reviewer as well, since you're familiar with cloud tests and this should make sense to you. We can go over details of this whenever you have time.

[clee]

Doesn't work on Azure? Or just hasn't been tested there yet? (Also does Bigquery work on AWS? I thought it was GCP-only)

[rpdevmp]

It should work on Azure, but requires cross cloud policy setup, which is not yet ready and not tested with Azure storage, but in theory should work the same. We can add Azure storage once this gets merged and tested on other clouds.

Bigquery itself is on GCP, but when we create Datasets and external tables, it could query "cross-cloud". Details are documented in the Conflucne page that I shared. Requres one-time setup to allow BigQuery to connect to Redpanda storage buckets on another cloud