Extend layout placeholder params for filesystem destinations

[sultaniman]

This PR addresses #930

Requirements:

• Make sure you can place something after the extension,
• Should be able to pass custom placeholders dictionary to fs and merge w/ existing placeholders,
  • if placeholder is not known then show a warning and skip,
  • add validation for placeholders if they are defined with user friendly messages,
  • support custom current date formatting placeholders (interpolation options on top of pendulum or ISO standard)
• Placeholders in dictionary can be values or callbacks,
• Make configurable via config toml,
• We would like to have overridable current date to filesystem arguments

@sh-rp recommends to test with staging destination
you’ll have to test it, write a test where you run the destination settings that use staging and add a map function for the filename

TODO

• Tests,
• Docs update

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	a0cad82
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/660d6395d2b8d80009b75363

[sultaniman]

I am passing these three arguments via kwargs so here on resolve I reconcile and can override the values from config.toml.
It doesn't feel right however is it fine @sh-rp @rudolfix ?

[rudolfix]

OK factory is tricky and its main goal is to delay instantiation, population of the configs until runtime (ie. load step) while the UX is like you can create destination at any moment.

look at base init

def __init__(self, **kwargs: Any) -> None:
        # Create initial unresolved destination config
        # Argument defaults are filtered out here because we only want arguments passed explicitly
        # to supersede config from the environment or pipeline args
        sig = inspect.signature(self.__class__.__init__)
        params = sig.parameters
        self.config_params = {
            k: v for k, v in kwargs.items() if k not in params or v != params[k].default
        }

it will store all unknown paramters in config_params
and then here

def configuration(self, initial_config: TDestinationConfig) -> TDestinationConfig:
        """Get a fully resolved destination config from the initial config"""
        config = resolve_configuration(
            initial_config,
            sections=(known_sections.DESTINATION, self.destination_name),
            # Already populated values will supersede resolved env config
            explicit_value=self.config_params,
        )
        return config

whenever configuration is created, we use those params as explicit initials

outcome: you do not need any special code to handle that. just pass additional stuff to super() init and it will be pushed to config during resolution

[sultaniman]

Done.

[rudolfix]

OK factory is tricky and its main goal is to delay instantiation, population of the configs until runtime (ie. load step) while the UX is like you can create destination at any moment.

look at base init

def __init__(self, **kwargs: Any) -> None:
        # Create initial unresolved destination config
        # Argument defaults are filtered out here because we only want arguments passed explicitly
        # to supersede config from the environment or pipeline args
        sig = inspect.signature(self.__class__.__init__)
        params = sig.parameters
        self.config_params = {
            k: v for k, v in kwargs.items() if k not in params or v != params[k].default
        }

it will store all unknown paramters in config_params
and then here

def configuration(self, initial_config: TDestinationConfig) -> TDestinationConfig:
        """Get a fully resolved destination config from the initial config"""
        config = resolve_configuration(
            initial_config,
            sections=(known_sections.DESTINATION, self.destination_name),
            # Already populated values will supersede resolved env config
            explicit_value=self.config_params,
        )
        return config

whenever configuration is created, we use those params as explicit initials

outcome: you do not need any special code to handle that. just pass additional stuff to super() init and it will be pushed to config during resolution

[rudolfix]

you do not need this method at all. config will be populated automatically

[rudolfix]

OK you need it to validate the layout. at this moment you know all the config values. some come from providers some from factory,

so here you need to validate the layout. see if all placeholders are known. if not we need a really good explanation which are missing etc.

[rudolfix]

don't do that. fsspec does that in put_file. this code is not needed. what was your reason to add it?
OFC this smells like race conditions that's why we try to "create" dirs in initialize. btw. buckets do not have dirs really

[sultaniman]

@rudolfix the strange thing was that ffspec didn't create directory that's why I added it, I will remove this and see if test pipeline will work out.

[rudolfix]

hopefully this will merge into one nice path_utils or layout.py :)

[sultaniman]

Yes layout.py is temporary thing to keep things clean and move things to path_utils before merging.

[rudolfix]

what's wrong with using path_utils method?

[rudolfix]

here you must catch TypeError which indicates a wrong signature and explain what the correct signature is. otherwise people will be confused

ideally you could inspect all the signatures when validating placeholders

[rudolfix]

also add
{YY}/{MM}/{DD}/{table_name}/... and see where it fails

[rudolfix]

I'll comment on that in next review. this is a bigger chunk. we want to support all paths in append-only and require old layout only when we replace data (we can't guarantee replacement otherwise)

[sultaniman]

Closing this in favor of #1182