Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extend destination factory to allow for custom names and config sections #750

Closed
rudolfix opened this issue Nov 10, 2023 · 0 comments
Closed
Assignees

Comments

@rudolfix
Copy link
Collaborator

rudolfix commented Nov 10, 2023

Background
With #746 destinations can be used like sources (imported and then parametrized). Here we add more flexibility on how to configure them,

*Tasks

    • in DestinationClientConfiguration rename destination_name to destination_type: it is a Final (non configurable) field (we can convert it into class var as well)
    • add name field which defaults to destination_type if not provided.
    • use the ability of with_config to accept section as lambda that takes the arguments of the function to generate dynamic config section for a destination: [destination.<name>], please fix with_config signature and add unit test for it
    • add environment (str) config field with default None
    • each destination factory must accept the base fields of DestinationClientConfiguration (name, environment, credentials) and expose them as attributes

Implementation
This probably impacts #746 - we should get rid of current destination reference and most of the init of implementation in favor of destination factory.

relevant part of with_config that deals with dynamic sections

# if section derivation function was provided then call it
                if section_f:
                    curr_sections: Tuple[str, ...] = (section_f(bound_args.arguments), )
                    # sections may be a string
                elif isinstance(sections, str):
                    curr_sections = (sections,)
                else:
                    curr_sections = sections
@rudolfix rudolfix moved this from Todo to Planned in dlt core library Nov 10, 2023
@rudolfix rudolfix moved this from Planned to Done in dlt core library Nov 13, 2023
@rudolfix rudolfix moved this from Done to In Progress in dlt core library Nov 13, 2023
@rudolfix rudolfix moved this from In Progress to Planned in dlt core library Nov 13, 2023
@rudolfix rudolfix moved this from Planned to Done in dlt core library Nov 19, 2023
@rudolfix rudolfix moved this from Done to In Progress in dlt core library Nov 19, 2023
@rudolfix rudolfix assigned rudolfix and sh-rp and unassigned steinitzu and rudolfix Nov 21, 2023
@rudolfix rudolfix moved this from In Progress to Done in dlt core library Dec 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

No branches or pull requests

3 participants