Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V1.1.9 tmc #14

Draft
wants to merge 6 commits into
base: v1.1.8-tmc
Choose a base branch
from
Draft

V1.1.9 tmc #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ce0ddc5
remove indivisiable commitee from tests
willyraedy Oct 8, 2024
634c890
typo
willyraedy Oct 8, 2024
17a9556
remove test filter from lookup tables
willyraedy Oct 8, 2024
6eee82a
add distincts
willyraedy Oct 8, 2024
fb8a928
remove rows with missing primary key
willyraedy Oct 8, 2024
a8e25cc
typo
willyraedy Oct 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 36 additions & 12 deletions models/staging/ngpvan/_staging_ngpvan_models.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,9 @@ models:
description: "Surrogate primary key uniquely identifying this record across all members data."
tests:
- not_null
- unique
- unique:
config:
where: source_schema != 'raw_bonterra__tmc_corrected' and committee_id != 68149

- name: stg_ngpvan__campaigns
config:
Expand Down Expand Up @@ -66,7 +68,9 @@ models:
description: "Surrogate primary key uniquely identifying this record across all members data."
tests:
- not_null
- unique
- unique:
config:
where: source_schema != 'raw_bonterra__tmc_corrected' and committee_id != 68149

- name: stg_ngpvan__codes
config:
Expand Down Expand Up @@ -103,7 +107,9 @@ models:
description: "Surrogate primary key uniquely identifying this record across all members data."
tests:
- not_null
- unique
- unique:
config:
where: source_schema != 'raw_bonterra__tmc_corrected' and committee_id != 68149

- name: stg_ngpvan__code_types
config:
Expand Down Expand Up @@ -152,7 +158,9 @@ models:
description: "Surrogate primary key uniquely identifying this record across all members data."
tests:
- not_null
- unique
- unique:
config:
where: source_schema != 'raw_bonterra__tmc_corrected' and committee_id != 68149

- name: stg_ngpvan__contacts_activist_codes
config:
Expand Down Expand Up @@ -195,7 +203,9 @@ models:
description: "Surrogate primary key uniquely identifying this record across all members data."
tests:
- not_null
- unique
- unique:
config:
where: source_schema != 'raw_bonterra__tmc_corrected' and committee_id != 68149

- name: stg_ngpvan__contacts_codes
config:
Expand Down Expand Up @@ -230,7 +240,9 @@ models:
description: "Surrogate primary key uniquely identifying this record across all members data."
tests:
- not_null
- unique
- unique:
config:
where: source_schema != 'raw_bonterra__tmc_corrected' and committee_id != 68149

- name: stg_ngpvan__contacts_contacts
config:
Expand Down Expand Up @@ -277,7 +289,9 @@ models:
description: "Surrogate primary key uniquely identifying this record across all members data."
tests:
- not_null
- unique
- unique:
config:
where: source_schema != 'raw_bonterra__tmc_corrected' and committee_id != 68149

- name: stg_ngpvan__contacts_notes
config:
Expand Down Expand Up @@ -320,7 +334,9 @@ models:
description: "Surrogate primary key uniquely identifying this record across all members data."
tests:
- not_null
- unique
- unique:
config:
where: source_schema != 'raw_bonterra__tmc_corrected' and committee_id != 68149

- name: stg_ngpvan__contacts_survey_responses
config:
Expand Down Expand Up @@ -365,7 +381,9 @@ models:
description: "Surrogate primary key uniquely identifying this record across all members data."
tests:
- not_null
- unique
- unique:
config:
where: source_schema != 'raw_bonterra__tmc_corrected' and committee_id != 68149 and contacts_survey_response_id is not null
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

primary key is just straight up missing for like 14k records spanning all years


- name: stg_ngpvan__contact_types
config:
Expand Down Expand Up @@ -458,7 +476,9 @@ models:
description: "Surrogate primary key uniquely identifying this record across all members data."
tests:
- not_null
- unique
- unique:
config:
where: source_schema != 'raw_bonterra__tmc_corrected' and committee_id != 68149

- name: stg_ngpvan__survey_responses
config:
Expand Down Expand Up @@ -488,7 +508,9 @@ models:
description: "Surrogate primary key uniquely identifying this record across all members data."
tests:
- not_null
- unique
- unique:
config:
where: source_schema != 'raw_bonterra__tmc_corrected' and committee_id != 68149

- name: stg_ngpvan__users
config:
Expand Down Expand Up @@ -537,7 +559,9 @@ models:
description: "Surrogate primary key uniquely identifying this record across all members data."
tests:
- not_null
- unique
- unique:
config:
where: source_schema != 'raw_bonterra__tmc_corrected' and committee_id != 68149

- name: stg_ngpvan__users_user_groups
config:
Expand Down
2 changes: 1 addition & 1 deletion models/staging/ngpvan/core/stg_ngpvan__activist_codes.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,6 @@ WITH
)

SELECT
*
DISTINCT *
FROM renamed

2 changes: 1 addition & 1 deletion models/staging/ngpvan/core/stg_ngpvan__committees.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,6 @@ WITH
)

SELECT
*
DISTINCT *
FROM renamed

2 changes: 1 addition & 1 deletion models/staging/ngpvan/core/stg_ngpvan__survey_responses.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,4 +32,4 @@ WITH
LEFT JOIN surveyquestions USING (surveyquestionid)
)

SELECT * FROM responses
SELECT DISTINCT * FROM responses