Apply hints for nested tables

[steinitzu]

Description

Draft of nested table hints implementation:

apply_hints(path=['a', 'b', 'c'], columns=...)

Is working so far but there are some bugs and tests needed.

Related Issues

• Resolves Simplify schema modification of child tables #1647

Additional Context

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	e51bd25
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/679a7aff51c87d000948a3af

[rudolfix]

Please see my suggestion how to deal with naming convention. Docs requirements are in the ticket.

[rudolfix]

compute_table_chain must take NamingConvention instance that has a method to join path and we do not need to hardcode the "__".

overall this is a weakness of dlt that it relies on such separator and stores only normalized names in the schema. we lose a little bit of lineage information but right now we can't really avoid that without a big rewrite

[zilto]

The current implementation adds the tables to the schema (as tested), but it doesn't affect how the data is loaded.

For example, the hints will appear in

pipeline.default_schema.tables.keys()
# ignoring the dlt tables
# 'nested_data', 'override_child_outer1', 'override_child_outer1_innerfoo','nested_data__outer1', 'nested_data__outer1__innerfoo'

Whereas the normalizer row counts show no ingested data for the tables

pipeline.last_trace.last_normalize_info.row_counts
# 'nested_data': 2, 'nested_data__outer1': 2, 'nested_data__outer1__innerfoo': 2

I believe changes need to be made to Extractor._write_to_dynamic_table() and _get_dynamic_table_name() to push data to the right table. (Extractor._write_to_static_table() should rely on the explicitly provided table name).

The extractor would need to hold some mapping, but it could be more appropriate to move the logic to dlt.common.normalizers.json.helpers or to a Schema method?

[rudolfix]

Relational normalizer follows its logic of creating nested tables and column names. it comes only from the data. there's no mechanism to rename those, except the root table name which the user must set.

dlt is data first, not schema first. it is counterintuitive if you chose to start your work with schema, not data.

I assume that in example you are giving, you used a custom table name for nested table. If this is not the case ping me on slack. maybe there's a bug somewhere

in the ticket above, there's a note:

You still may allow users to specify table_name on the nested hint. If you do so, you'll need to modify the normalizer so it maps paths to those names. IMO this is for another ticket and bigger overhaul of the schema
prevent following to be set on nested table:
parent_table_name: TTableHintTemplate[str] = None,
incremental: TIncrementalConfig = None,

so I'd say we block setting table name on nested hints (also parent name and incremental do make sense)