0.4.5 release master merge #1006
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* WIP(gdrive): implement basic version of gdrive source * add gdrive bucket into tests * skip destination tests * add temporary fixture for gdrive testing * return env preservation * refactors google drive fsspec * documents and cleanups code * add test retries * implements fs.info to be consistent --------- Co-authored-by: Marcin Rudolf <[email protected]>
add some docs links
…ng tools article (#956) Authored-by: Michal Hauzírek <> @mhauzirek
* feat(airflow): add the "parallel" mode
update pipeline links
add update pipeline links script
…980) * Add more test cases for extract_inner_type * mark dataclasses as complex type * Fix annotation reconstruction and better union type checks * Add more checks to typing tests and more complex pydantic tests * Ignore mypy warning * Adjust mypy warning ignore reason * Move complex tests for extract_inner_type to libs * Upgrade typing-extensions to 4.9.0 and mypy to 1.8.0 * Ignore mypy warning for syntax error * Pass tuple to pass annotation metadata to Annotated * Remove generic base for test model * Remove unused mypy error code * Add more test cases * Fix linting errors * Fix linting errors * Format code * Format code * Adjust tests for python<3.10 * Use assert_type instead of simple assert * Revert * Remove unused import * Define UUID4 and remove created file * Revert mypy and typing-extensions update * Fix linting issues
* Stub adapter function #855 Signed-off-by: Marcel Coetzee <[email protected]> * Lift duplicated generator function to test utils #855 Signed-off-by: Marcel Coetzee <[email protected]> * Grammar #855 Signed-off-by: Marcel Coetzee <[email protected]> * Grammar #855 Signed-off-by: Marcel Coetzee <[email protected]> * Update BigQuery adapter and related tests #855 Signed-off-by: Marcel Coetzee <[email protected]> * Refactor BigQuery adapter and modify tests #855 Signed-off-by: Marcel Coetzee <[email protected]> * Update tests #855 Signed-off-by: Marcel Coetzee <[email protected]> * Rename apply_hints table parameter #855 Signed-off-by: Marcel Coetzee <[email protected]> * Rename apply_hints table parameter #855 Signed-off-by: Marcel Coetzee <[email protected]> * Implement BigQuery adapter hint tests #855 Signed-off-by: Marcel Coetzee <[email protected]> * Add table hints as property to hints #855 Signed-off-by: Marcel Coetzee <[email protected]> * Lift additional table hints to top level hints dictionary #855 Signed-off-by: Marcel Coetzee <[email protected]> * #855 Add explicit sqlglot dependency Signed-off-by: Marcel Coetzee <[email protected]> * #855 Add test Signed-off-by: Marcel Coetzee <[email protected]> * #855 Format Signed-off-by: Marcel Coetzee <[email protected]> * #855 Use SQLglot ast diffs in tests Signed-off-by: Marcel Coetzee <[email protected]> * #855 Ast diff test finalized with no adapter Signed-off-by: Marcel Coetzee <[email protected]> * #855 Finish test_adapter_hints_partitioning Signed-off-by: Marcel Coetzee <[email protected]> * #855 test_adapter_hints_round_half_away_from_zero Signed-off-by: Marcel Coetzee <[email protected]> * #855 Finalize test_adapter_hints_round_half_away_from_zero Signed-off-by: Marcel Coetzee <[email protected]> * #855 Update table expiration parsing and add tests Signed-off-by: Marcel Coetzee <[email protected]> * #855 Qualify x-bigquery-column hints fully Signed-off-by: Marcel Coetzee <[email protected]> * #855 Refactor bq tests Signed-off-by: Marcel Coetzee <[email protected]> * Implement adapter rounding hints #855 Signed-off-by: Marcel Coetzee <[email protected]> * Lock file Signed-off-by: Marcel Coetzee <[email protected]> * #855 Add test for clustering on multiple columns. Signed-off-by: Marcel Coetzee <[email protected]> * #855 Set escape flag in tests Signed-off-by: Marcel Coetzee <[email protected]> * #855 Use ensure_resource in bigquery_adapter.py Signed-off-by: Marcel Coetzee <[email protected]> * #855 Minor refactor Signed-off-by: Marcel Coetzee <[email protected]> * #855 Fix multiple partition test Signed-off-by: Marcel Coetzee <[email protected]> * #855 Partition ordering Signed-off-by: Marcel Coetzee <[email protected]> * #855 Remove sqlglot dependency and tests Signed-off-by: Marcel Coetzee <[email protected]> * #855 Remove pycharm editor flags Signed-off-by: Marcel Coetzee <[email protected]> * #855 Fix Signed-off-by: Marcel Coetzee <[email protected]> * #855 Remove (git conflict?) Signed-off-by: Marcel Coetzee <[email protected]> * #855 Fix spacing in PARTITION BY clause Signed-off-by: Marcel Coetzee <[email protected]> * #855 Refactor assertions in bigquery table tests Signed-off-by: Marcel Coetzee <[email protected]> * #855 Minor adapter refactor Signed-off-by: Marcel Coetzee <[email protected]> * #855 Rename tests for clarity Signed-off-by: Marcel Coetzee <[email protected]> * #855 Grammar, minor housecleaning Signed-off-by: Marcel Coetzee <[email protected]> * #855 Pass current tests Signed-off-by: Marcel Coetzee <[email protected]> * #855 Format Signed-off-by: Marcel Coetzee <[email protected]> * Refactor table hint tests and fix conditional block #855 Signed-off-by: Marcel Coetzee <[email protected]> * Finish Unit tests #855 Signed-off-by: Marcel Coetzee <[email protected]> * Improve formatting and readability in BigQuery docs #855 Signed-off-by: Marcel Coetzee <[email protected]> * Update documentation heading capitalization #855 Signed-off-by: Marcel Coetzee <[email protected]> * Update tests #855 Signed-off-by: Marcel Coetzee <[email protected]> * Format and fix linting issues #855 Signed-off-by: Marcel Coetzee <[email protected]> --------- Signed-off-by: Marcel Coetzee <[email protected]>
* Use BIGNUMERIC for large decimals in bigquery * Handle dest without decimal type
* Normalize pipeline name for google secrets provider * Resolve test root as abs path, format conftest * Strip punctuation manually and strip whitespace
* black formatting * remove unused exception * add initial support for replicate write disposition * add hard_delete hint and sorted deduplication for merge * undo config change * undo unintentional changes * refactor hard_delete handling and introduce dedup_sort hint * update docstring * replace dialect-specific SQL * add parentheses to ensure proper clause evaluation order * add escape defaults and temp tables for non-primary key case * exclude destinations that don't support merge from test * correct typo * extend docstring * remove redundant copies for immutable strings * simplify boolean logic * add more test cases for hard_delete and dedup_sort hints * refactor table chain resolution * marks tables that seen data in normalizer, skips empty jobs if never seen data * ignores tables that didn't seen data when loading, tests edge cases * add sort order configuration option * bumps schema engine to v9, adds migrations * filters tables without data properly in load * converts seen-data to boolean, fixes tests * disables filesystem tests config due to merge present * add docs for hard_delete and dedup_sort column hints * fixes extending table chains in load * refactors load and adds unit tests with dummy --------- Co-authored-by: Jorrit Sandbrink <[email protected]> Co-authored-by: Marcin Rudolf <[email protected]>
* allows to set custom typed json encoding pua start * replaces . in direct encoding * sets new PUA start, fallbacks to original string on decoding error * tests pipeline pua encoded state migration, moved to original PUA start
* docs(filesystem): include Google Drive into filesystem tutorial * fix typos * fix typo * fix API name
* feat(airflow): implement parallel-isolated mode * emit a warning for incremental args in resources * in parallel-isolated mode use pipeline names * runs isolated-parallel in separate pipelines, parallel in single --------- Co-authored-by: Marcin Rudolf <[email protected]>
* allows to decorate async function with dlt.source * adds pytest-async and updates pytest to 7.x * fixes forked teardown issue 7.x * bumps deps for py 3.12 * adds py 12 common tests * fixes typings after deps bump * bumps airflow, yanks duckdb to 0.9.2 * adds right hand pipe operator * adds docs on async sources and resources * enables jsonl on athena * forces staging for databricks * skips useless warnings for automatic staging in Pipeline * passes correct loader file format in pipeline tests * fixes pendulum cursor in incremental on arrow * fixes parquet tests
✅ Deploy Preview for dlt-hub-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Merges devel to master for 0.4.5 release