fix bigquery autodetect

[sh-rp]

Description

When using autodetect_schema on bigquery, merge and replace with "insert-from-staging" will fail, because dlt is:

1. Trying to truncate non-existing tables (this is solveable fairly easy, see first commit)
2. On the first load trying to insert data into a non existing table (also easily solvable by creating the correct final table from the staging table before the merge is run)

Problems: The suggested approach in this ticket will not work if the schema of the table changes, because bigquery has no commands to update the schema of an existing table to match the schema of another existing table. So the staging table will auto evolve but it's not easy to have the main table follow this without directly inserting data there.

Possible solutions:

• Disallow schema evolutions when merge and autodetect is enabled. The users would have to manually do this.
• Write a bit of code that can update the main table to match the staging table by reflecting the information schema. This probably is fairly time consuming to do on every load. We'd have to insert some update commands between loading the staging jobs which change the schema and executing the merge sql
• Update the schemas of all tables, staging and final, by inserting data from a parquet file and selecting a partition that does not exist with auto schema updates enabled. This way, according to my research, the schema will be updated but no data is loaded.

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	f515820
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/67338091793bbe000833d88d

[sh-rp]

The proposed quick fix for this would be to enable merging and staging replace for bigquery auto schema detection based on cloning the table schema from staging to final table in bigquery. Merges with schema updates will fail though, either we somehow freeze the schema for these tables or we add a very big note in the docs about this. Maybe we can amend the exception message for a fail like this with some more info about why this might have failed for bigquery tables with autoschema.

[rudolfix]

this fixes the case without schema evolution. it is a good first attempt (btw. I think autodetect schema cannot really evolve existing tables so we are fine!
https://www.reddit.com/r/bigquery/comments/1c02azc/autodetecting_updated_schema_when/?rdt=38122)

[rudolfix]

this code should be executed at the end of each copy job - and only if autodetect schema is enabled. this way we can just run our merge job as usual.

we could also migrate columns from staging table to destination tables but maybe we can start with just cloning

[sh-rp]

I have prepended it so the bigquery merge jobs for autodetect tables now. The other solution would be much more complicated with two consecutive followupjobs. I can do it that way too, but I think my solution is fairly clean...

[rudolfix]

should we do it only when autodetect schema is on?

[sh-rp]

done