Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BigQuery Partitioning Improvements #887

Merged
merged 42 commits into from
Jan 18, 2024
Merged

BigQuery Partitioning Improvements #887

merged 42 commits into from
Jan 18, 2024

Conversation

Pipboyguy
Copy link
Collaborator

@Pipboyguy Pipboyguy commented Jan 11, 2024 •
edited
Loading

Adds support to partition on datetime, date and bigint types.

Changes

  • Control flow to test each data type and the appropriate partition logic to apply.
  • Added tests for each type.
  • My research suggests bigquery can't partition on FLOAT64, hence I removed this claim from the docs.
  • Minor grammar fixes, (subjective) code cleanup

Caveats

It is worth nothing that in order to partition on INT64 we need to know the bounds upfront. Please see the workaround and limitations that come with it.

Fixes #852

Pipboyguy and others added 29 commits January 7, 2024 01:16
@Pipboyguy
Add tests (#852)
f8402ca 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Merge remote-tracking branch 'origin/devel' into 852-cannot-create-pa…
b47feca 
…rtitioned-table-in-bigquery-destination
@Pipboyguy
Optimise imports (#852)
0822d32 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Minor changes (#852)
17971ce 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Amend for clarity, spelling etc (#852)
0a95700 
Signed-off-by: Marcel Coetzee <[email protected]>
@sultaniman @Pipboyguy
Update field type coercion with more specific examples
fb32ada
@sultaniman @Pipboyguy
Fix typo
7e039cc
@sultaniman @Pipboyguy
Adjust explanation
c9f9335
@sultaniman @Pipboyguy
Add encouragement for experimentation
e3bf10f
@sultaniman @burnash @Pipboyguy
Update docs/website/docs/general-usage/schema.md
9004f9c 
Co-authored-by: Anton Burnashev <[email protected]>
@sultaniman @Pipboyguy
Update table column title naming
1d5d4d2
@sultaniman @Pipboyguy
Adjust data to avoid naming confusion
1b751b2
@sultaniman @Pipboyguy
Adjust explanation phrasing
4612906
@sultaniman @Pipboyguy
Revert column name
618659d
@Pipboyguy
Refactor and improve documentation grammar (#814)
5ec87a6 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Refactor BigQuery and tests for clarity and correctness (#852)
393f631 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Add tests (#852)
41f0bb1 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Optimise imports (#852)
8b927c3 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Minor changes (#852)
5cf9436 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Amend for clarity, spelling etc (#852)
c028b6b 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Refactor and improve documentation grammar (#814)
da2bd46 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Refactor BigQuery and tests for clarity and correctness (#852)
c1e312e 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
IDE inspection disable on function (#852)
1109410 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Merge (#852
b2ad1da 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Merge branch 'devel' of github.com:dlt-hub/dlt into 852-cannot-create…
4577176 
…-partitioned-table-in-bigquery-destination
@Pipboyguy
Add bigquery tests for time, date and integer partitioning (#852)
74a831f 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Add control flow to check typing and partition accordingly #852
2f1791e 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Merge branch 'devel' of github.com:dlt-hub/dlt into 852-cannot-create…
ce082c0 
…-partitioned-table-in-bigquery-destination
@Pipboyguy
Remove partitioning on support from BQ docs #852
8c5d825 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy Pipboyguy requested a review from sh-rp January 11, 2024 20:20
@Pipboyguy Pipboyguy requested a review from rudolfix January 11, 2024 20:20
@Pipboyguy Pipboyguy self-assigned this Jan 11, 2024
@Pipboyguy Pipboyguy added the bug Something isn't working label Jan 11, 2024
@Pipboyguy Pipboyguy linked an issue Jan 11, 2024 that may be closed by this pull request
Cannot create partitioned table in BigQuery destination #852
Closed
@netlify Netlify
Copy link

netlify bot commented Jan 11, 2024 •
edited
Loading

✅ Deploy Preview for dlt-hub-docs ready!

Name Link
🔨 Latest commit be60ecf
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/65a7f3d9fc58420009abb502
😎 Deploy Preview https://deploy-preview-887--dlt-hub-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@Pipboyguy
Format and fix grammar/spelling in docs #852
07bb5f6 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Merge branch 'devel' of github.com:dlt-hub/dlt into 852-cannot-create…
1f6578f 
…-partitioned-table-in-bigquery-destination
@Pipboyguy
Merge branches '852-cannot-create-partitioned-table-in-bigquery-desti…
a1fd3df 
…nation' and 'devel' of github.com:dlt-hub/dlt into 852-cannot-create-partitioned-table-in-bigquery-destination
@Pipboyguy Pipboyguy marked this pull request as draft January 15, 2024 11:33
@Pipboyguy
Relocate BigQuery tests to the appropriate folder (#852)
c9b8c17 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Relocate partitioning tests to the table builder tests (#852)
3be6413 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Merge branch 'devel' of github.com:dlt-hub/dlt into 852-cannot-create…
b24b435 
…-partitioned-table-in-bigquery-destination
@Pipboyguy
Merge
f3bd0d5 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Remove
1130b9a 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Update BigQuery documentation and comments #852
358806f 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Amend bq docs #852
92221a6 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Merge branch 'devel' of github.com:dlt-hub/dlt into 852-cannot-create…
a1ad4b4 
…-partitioned-table-in-bigquery-destination
@Pipboyguy
Add negative partition tests #852
2847d27 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy Pipboyguy marked this pull request as ready for review January 17, 2024 15:20
@Pipboyguy Pipboyguy requested a review from sh-rp January 17, 2024 15:20
@Pipboyguy
Add init file #852
be60ecf 
Signed-off-by: Marcel Coetzee <[email protected]>
sh-rp
sh-rp approved these changes Jan 18, 2024
View reviewed changes
Copy link
Collaborator

@sh-rp sh-rp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@sh-rp sh-rp merged commit 3921d61 into devel Jan 18, 2024
56 checks passed
@sh-rp sh-rp deleted the 852-cannot-create-partitioned-table-in-bigquery-destination branch January 18, 2024 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sh-rp sh-rp sh-rp approved these changes

@rudolfix rudolfix Awaiting requested review from rudolfix

@burnash burnash Awaiting requested review from burnash

@AstrakhantsevaAA AstrakhantsevaAA Awaiting requested review from AstrakhantsevaAA

Assignees

@Pipboyguy Pipboyguy

Labels
bug Something isn't working
Projects
dlt core library
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Cannot create partitioned table in BigQuery destination
3 participants
@Pipboyguy @sh-rp @sultaniman