Behavior: Get column info from information_schema Part I #808
+271
−59
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
benc-db
requested review from
andrefurlan-db and
rcypher-databricks
as code owners
benc-db
commented
Sep 25, 2024
| @@ -116,26 +119,6 @@ class DatabricksConfig(AdapterConfig): | |||
| merge_with_schema_evolution: Optional[bool] = None | |||
|
|
|||
|
|
|||
| def check_not_found_error(errmsg: str) -> bool: | |||
There was a problem hiding this comment.
Moved to prevent circular dependency.
benc-db
commented
Sep 25, 2024
| @@ -175,6 +158,19 @@ class DatabricksAdapter(SparkAdapter): | |||
| } | |||
| ) | |||
|
|
|||
| get_column_behavior: GetColumnsBehavior | |||
|
|
|||
| def __init__(self, config: Any, mp_context: SpawnContext) -> None: | |||
There was a problem hiding this comment.
Although I don't control the invocation of init, and thus can't pass the behaviors in, doing this during init ensures we only check the behavior flag once. I tried to make this a little more functional but couldn't figure out how to override an existing function definition that is inherited from a parent, hence the goofy class-based strategy.
benc-db
commented
Sep 25, 2024
| @@ -28,6 +28,25 @@ | |||
| {% do return(load_result('get_columns_comments').table) %} | |||
| {% endmacro %} | |||
|
|
|||
| {% macro get_columns_comments_via_information_schema(relation) -%} | |||
| {% call statement('repair_table', fetch_result=False) -%} | |||
| REPAIR TABLE {{ relation|lower }} SYNC METADATA | |||
There was a problem hiding this comment.
Ensure information_schema is up to date prior to using this method.
benc-db
had a problem deploying
to
azure-prod
GitHub Actions
Failure
benc-db
had a problem deploying
to
azure-prod
GitHub Actions
Failure
benc-db
had a problem deploying
to
azure-prod
GitHub Actions
Failure
benc-db
had a problem deploying
to
azure-prod
GitHub Actions
Failure
benc-db
had a problem deploying
to
azure-prod
GitHub Actions
Failure
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Partial fix for #779
Description
This is my second attempt at addressing the issue that describe extended can truncate complex types. With the release of dbt-core 1.8.7, we can now process behavior flags; this PR introduces the choice of using information_schema for grabbing column information in get_columns_for_relation. I'm hiding behind a behavior flag because given the current state of UC information_schema, we have to run repair to trust that columns of recently created or altered table will be present in the information_schema, which adds overhead. Furthermore, this trick only works for Delta tables at this time. It is hoped that in time the sync issue with information_schema will be solved, but in the mean time, users can use this flag when they have complex types that describe extended truncates.
Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.