Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove unused rows_affected column from test_executions #195

Open
jaypeedevlin opened this issue Sep 7, 2022 · 1 comment
Open

Remove unused rows_affected column from test_executions #195

jaypeedevlin opened this issue Sep 7, 2022 · 1 comment
Labels
bug Something isn't working

Comments

@jaypeedevlin
Copy link
Contributor

This looks like it may have been copied over from model_executions in error.

Removing it is a non-trivial task because of the migration path — our insert statements don't enumerate the column names, so just removing all mentions of the column will cause inserts for existing projects to fail. We'll need to find a way to get only the column names that exist in the current incremental source model and use those.

This will be generically useful any time we need to drop a column, which we're likely to need to do again.

@glsdown glsdown added the bug Something isn't working label Mar 31, 2023
@glsdown glsdown mentioned this issue Sep 13, 2023
13 tasks
@glsdown glsdown linked a pull request Sep 13, 2023 that will close this issue
13 tasks
glsdown pushed a commit that referenced this issue Sep 14, 2023
@glsdown glsdown removed a link to a pull request Sep 14, 2023
13 tasks
@glsdown
Copy link
Contributor

glsdown commented Sep 14, 2023

This should be possible once #381 has been merged, however, I initially included it in there, and found that DBX is having issues when a field gets removed as it gives an error that it's unable to find the field.

Runtime Error in model test_executions (models/sources/test_executions.sql)
Cannot resolve rows_affected in UPDATE clause given columns DBT_INTERNAL_SOURCE.command_invocation_id, DBT_INTERNAL_SOURCE.node_id, DBT_INTERNAL_SOURCE.run_started_at, DBT_INTERNAL_SOURCE.was_full_refresh, DBT_INTERNAL_SOURCE.thread_id, DBT_INTERNAL_SOURCE.status, DBT_INTERNAL_SOURCE.compile_started_at, DBT_INTERNAL_SOURCE.query_completed_at, DBT_INTERNAL_SOURCE.total_node_runtime, DBT_INTERNAL_SOURCE.failures, DBT_INTERNAL_SOURCE.message, DBT_INTERNAL_SOURCE.adapter_response.; line 11 pos 2

When I have some time I will look into why removing the field causes this error (Snowflake and BQ are fine), but recording it here for future.

glsdown added a commit that referenced this issue Sep 18, 2023
* Add named columns

* Remove unused `rows_affected` column in `test_executions` #195

* Reorganise macro folder

* Add documentation for macros

* Consolidate logic in upload results

* Simplify how nodes are extracted

* Generalise identifying objects to load further

* Have all results included in the loop

* Split out get_dataset_content

* Move the get table content into its own macro

* Bug fixing

* Move relation definition to insert macro to avoid quoting issues

* Revert "Remove unused `rows_affected` column in `test_executions` #195"

This reverts commit 9182fdc.

* Remove ---debug left in databricks tox

* Apply suggestions from code review

Co-authored-by: Jared Rimmer <[email protected]>

---------

Co-authored-by: Jared Rimmer <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants