-
Notifications
You must be signed in to change notification settings - Fork 197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parametrized destinations #746
Merged
+978
−593
Merged
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit
Hold shift + click to select a range
3cdab60
Move destination modules to subfolder
steinitzu 3b53861
Mockup destination factory
steinitzu 4b449e3
Destination factory replacing reference and dest __init__
steinitzu f195fa7
Update factories
steinitzu 340f637
Defer duckdb credentials resolving in pipeline context
steinitzu 4ae4ceb
Simplify destination config resolution
steinitzu bccfa9d
capabilities are callable
steinitzu 3bde2f4
bigquery, athena factories
steinitzu 0d59d51
Add rest of factories
steinitzu 3ee262d
Cleanup
steinitzu 9243b2f
Destination type vars
steinitzu 9ad561c
Cleanup
steinitzu 007d7a3
Fix test
steinitzu 62b5a57
Create initial config from non-defaults only
steinitzu d77b54a
Update naming convention path
steinitzu 0d4ad49
Fix config in bigquery location test
steinitzu f657071
Only keep non-default config args in factory
steinitzu 56e922a
Resolve duckdb credentials in pipeline context
steinitzu 08a8fdc
Cleanup
steinitzu 2567ca0
Union credentials arguments
steinitzu 3537c03
Common tests without dest dependencies
steinitzu 25a937a
Forward all athena arguments
steinitzu 97f1afc
Delete commented code
steinitzu fe24e14
Reference docstrings
steinitzu 481a7cb
Add deprecation warning for credentials argument
steinitzu d91402c
Init docstrings for destination factories
steinitzu fc92929
Fix tests
steinitzu 13ec6fb
Destination name in output
steinitzu 1f16c8f
Correct exception in unknown destination test
steinitzu 2541d49
Merge branch 'devel' into sthor/parametrized-destination
rudolfix File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading status checks…
Resolve duckdb credentials in pipeline context
commit 56e922a02ec3662b213461e63a52c2fb91579f99
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -96,25 +96,35 @@ class DuckDbCredentials(DuckDbBaseCredentials): | |
|
||
__config_gen_annotations__: ClassVar[List[str]] = [] | ||
|
||
def _database_path(self) -> str: | ||
def is_partial(self) -> bool: | ||
partial = super().is_partial() | ||
if partial: | ||
return True | ||
# Wait until pipeline context is set up before resolving | ||
return self.database == ":pipeline:" | ||
|
||
def on_resolved(self) -> None: | ||
# do not set any paths for external database | ||
if self.database == ":external:": | ||
return | ||
# try the pipeline context | ||
is_default_path = False | ||
if self.database == ":pipeline:": | ||
return self._path_in_pipeline(DEFAULT_DUCK_DB_NAME) | ||
self.database = self._path_in_pipeline(DEFAULT_DUCK_DB_NAME) | ||
else: | ||
# maybe get database | ||
maybe_database, maybe_is_default_path = self._path_from_pipeline(DEFAULT_DUCK_DB_NAME) | ||
# if pipeline context was not present or database was not set | ||
if not self.database or not maybe_is_default_path: | ||
# create database locally | ||
is_default_path = maybe_is_default_path | ||
path = maybe_database | ||
else: | ||
path = self.database | ||
self.database = maybe_database | ||
|
||
path = os.path.abspath(path) | ||
# always make database an abs path | ||
self.database = os.path.abspath(self.database) | ||
# do not save the default path into pipeline's local state | ||
if not is_default_path: | ||
self._path_to_pipeline(path) | ||
return path | ||
self._path_to_pipeline(self.database) | ||
|
||
def _path_in_pipeline(self, rel_path: str) -> str: | ||
from dlt.common.configuration.container import Container | ||
|
@@ -123,9 +133,7 @@ def _path_in_pipeline(self, rel_path: str) -> str: | |
context = Container()[PipelineContext] | ||
if context.is_active(): | ||
# pipeline is active, get the working directory | ||
abs_path = os.path.abspath(os.path.join(context.pipeline().working_dir, rel_path)) | ||
context.pipeline().set_local_state_val(LOCAL_STATE_KEY, abs_path) | ||
return abs_path | ||
return os.path.join(context.pipeline().working_dir, rel_path) | ||
raise RuntimeError("Attempting to use special duckdb database :pipeline: outside of pipeline context.") | ||
|
||
def _path_to_pipeline(self, abspath: str) -> None: | ||
|
@@ -173,7 +181,7 @@ def _path_from_pipeline(self, default_path: str) -> Tuple[str, bool]: | |
return default_path, True | ||
|
||
def _conn_str(self) -> str: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. do we still need it? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Seems just for motherduck compatiblity it's needed, uses the same method. |
||
return self._database_path() | ||
return self.database | ||
|
||
|
||
@configspec | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you should not call super(). config resolver will do that for you
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Huh? How does that work? :/