Separate billing project & destination project

[enzoapu]

Hi~ Thanks for developing and maintaining this repo🙏

I have questions about cross-project billing when using the BigQuery sink connector.

I have two projects, Project A and Project B. I want to write data to the dataset of Project A through the connector (upsertEnabled=True). However, I want to use the slots of Project B.

Therefore, I created the service account B-user under Project B and assigned necessary permissions to B-user in Project A.
Then in the connector configuration(properties), set:

project = projectA_id
keyfile = B-user.json

I expected that this would use query slots from Project B, but when I actually queried from the INFORMATION_SCHEMA.JOBS_BY_PROJECT of Project B, I could not see the query record related to B-user. Instead, it was in the INFORMATION_SCHEMA JOBS_BY_PROJECT of Project A has related MERGE query records of B-user.

SELECT
    *
FROM
    `projectB_id.region-us.INFORMATION_SCHEMA.JOBS_BY_PROJECT`
WHERE
    DATE(creation_time) >= CURRENT_DATE() - 7
    AND user_email = 'B-user@projectB_id.iam.gserviceaccount.com'

-
> There is no data to display. 

I think the source code assigned the project in the connector's properties as query billing account, instead of the project_id in the key.json.

My question is is there a way to separate billing from destination projects?

This will be of great help, thank you!!!

Enzo Chang

[b-goyal]

That's an interesting case @enzoapu . The connector only supports one project id which it accepts as configuration. It does not parse the keyfile an sends it directly to the client library. So, from the connector side I am not aware of any ways to separate the billing from the configured destination project.

[enzochang17]

Hello @b-goyal ,
Thank you for your explanation. Is it possible that this will become a future development feature?
I think this can help many companies who want to separate their billing accounts!