Merge pull request #45 from fivetran/release/v0.7.0

Release/v0.7.0

.buildkite/pipeline.yml

@@ -1,4 +1,19 @@
 steps:
+  - label: ":postgres: Run Tests - Postgres"
+    key: "run-dbt-postgres"
+    plugins:
+      - docker#v3.13.0:
+          image: "python:3.8"
+          shell: [ "/bin/bash", "-e", "-c" ]
+          environment:
+            - "BASH_ENV=/tmp/.bashrc"
+            - "CI_POSTGRES_DBT_DBNAME"
+            - "CI_POSTGRES_DBT_HOST"
+            - "CI_POSTGRES_DBT_PASS"
+            - "CI_POSTGRES_DBT_USER"
+    commands: |
+      bash .buildkite/scripts/run_models.sh postgres
+      
   - label: ":snowflake-db: Run Tests - Snowflake"
     key: "run_dbt_snowflake"
     plugins:
@@ -43,7 +58,7 @@ steps:
     commands: |
       bash .buildkite/scripts/run_models.sh redshift
 
-  - label: ":bricks: Run Tests - Databricks"
+  - label: ":databricks: Run Tests - Databricks"
     key: "run_dbt_databricks"
     plugins:
       - docker#v3.13.0:

CHANGELOG.md

@@ -1,3 +1,23 @@
+# dbt_github v0.7.0
+
+## 🚨 Breaking Change 🚨
+- Updated the following models to aggregate at the `repository` grain in addition to their time period grain. ([#42](https://github.com/fivetran/dbt_github/pull/42), [#43](https://github.com/fivetran/dbt_github/pull/43))
+  - `github__daily_metrics`
+  - `github__weekly_metrics`
+  - `github__monthly_metrics`
+  - `github__quarterly_metrics`
+## 🎉 Features
+- Added column `requested_reviewers` to provide a list of users that were requested to review on a pull request. This is to supplement the column `reviewers`, which provides a list of users that have submitted a reivew, whether or not they were requested to. ([#44](https://github.com/fivetran/dbt_github/pull/44))
+- PostgreSQL compatibility! ([#44](https://github.com/fivetran/dbt_github/pull/44))
+## 🔧 Bug Fix
+- Updated model `int_github__pull_request_reviewers` so that the list of reviewers generated does not contain duplicate usernames. ([#44](https://github.com/fivetran/dbt_github/pull/44))
+## 🚘 Under the Hood
+- For the metrics models that were updated, added unique-combination-of-column tests for the combination of the time period and repository. ([#44](https://github.com/fivetran/dbt_github/pull/44))
+- Removed uniqueness tests on time period in metrics models in favor of the combo test. ([#44](https://github.com/fivetran/dbt_github/pull/44))
+- Removed ordering in metrics models to improve efficiency. ([#44](https://github.com/fivetran/dbt_github/pull/44))
+## 📝 Contributors 
+- @onimsha ([#42](https://github.com/fivetran/dbt_github/pull/42))
+
 # dbt_github v0.6.0
 [PR #35](https://github.com/fivetran/dbt_github/pull/35) includes the following breaking changes:
 ## 🚨 Breaking Changes 🚨:

README.md

@@ -27,18 +27,18 @@ The following table provides a detailed list of all models materialized within t
 | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | [github__issues](https://fivetran.github.io/dbt_github/#!/model/model.github.github__issues)     | Each record represents a GitHub issue, enriched with data about its assignees, milestones, and time comparisons.                                             |
 | [github__pull_requests](https://fivetran.github.io/dbt_github/#!/model/model.github.github__pull_requests)     | Each record represents a GitHub pull request, enriched with data about its repository, reviewers, and durations between review requests, merges and reviews. |
-| [github__daily_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__daily_metrics)     | Each record represents a single day, enriched with metrics about PRs and issues that were created and closed during that period.                              |
-| [github__weekly_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__weekly_metrics)    | Each record represents a single week, enriched with metrics about PRs and issues that were created and closed during that period.                             |
-| [github__monthly_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__monthly_metrics)   | Each record represents a single month, enriched with metrics about PRs and issues that were created and closed during that period.                            |
-| [github__quarterly_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__quarterly_metrics) | Each record represents a single quarter, enriched with metrics about PRs and issues that were created and closed during that period.                          |
+| [github__daily_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__daily_metrics)     | Each record represents a single day and repository, enriched with metrics about PRs and issues that were created and closed during that period.                              |
+| [github__weekly_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__weekly_metrics)    | Each record represents a single week and repository, enriched with metrics about PRs and issues that were created and closed during that period.                             |
+| [github__monthly_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__monthly_metrics)   | Each record represents a single month and repository, enriched with metrics about PRs and issues that were created and closed during that period.                            |
+| [github__quarterly_metrics](https://fivetran.github.io/dbt_github/#!/model/model.github.github__quarterly_metrics) | Each record represents a single quarter and repository, enriched with metrics about PRs and issues that were created and closed during that period.                          |
 <!--section-end-->
 
 # 🎯 How do I use the dbt package?
 ## Step 1: Prerequisites
 To use this dbt package, you must have the following:
 
 - At least one Fivetran Github connector syncing data into your destination.
-- A **BigQuery**, **Snowflake**, **Redshift**, or **Databricks** destination.
+- A **BigQuery**, **Snowflake**, **Redshift**, **PostgreSQL**, or **Databricks** destination.
 
 ### Databricks Dispatch Configuration
 If you are using a Databricks destination with this package you will need to add the below (or a variation of the below) dispatch configuration within your `dbt_project.yml`. This is required in order for the package to accurately search for macros within the `dbt-labs/spark_utils` then the `dbt-labs/dbt_utils` packages respectively.
@@ -55,7 +55,7 @@ Include the following github package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/github
-    version: [">=0.6.0", "<0.7.0"]
+    version: [">=0.7.0", "<0.8.0"]
 ```
 
 ## Step 3: Define database and schema variables
@@ -125,7 +125,7 @@ packages:
       version: [">=1.0.0", "<2.0.0"]
 
     - package: fivetran/github_source
-      version: [">=0.6.0", "<0.7.0"]
+      version: [">=0.7.0", "<0.8.0"]
     
     - package: dbt-labs/spark_utils
       version: [">=0.3.0", "<0.4.0"]

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'github'
-version: '0.6.0'
+version: '0.7.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
   github:

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'github_integration_tests'
-version: '0.6.0'
+version: '0.7.0'
 config-version: 2
 profile: 'integration_tests'
 vars:

models/github.yml

@@ -174,17 +174,24 @@ models:
       - name: reviewers
         description: List of Users who reviewed the pull request
 
+      - name: requested_reviewers
+        description: List of Users who were requested to review the pull request
+
       - name: number_of_reviews
         description: Number of times a pull request was reviewed
 
   - name: github__daily_metrics
     description: >
       Summary numbers for issues and pull requests by day
+    tests:
+      - dbt_utils.unique_combination_of_columns:
+          combination_of_columns:
+            - repository
+            - day
     columns:
       - name: day
         description: The reporting day
         tests:
-          - unique
           - not_null
       - name: number_issues_opened
         description: The total number of issues created during this time period
@@ -207,6 +214,11 @@ models:
       - name: number_prs_closed_without_merge
         description: The total number of pull requests closed without a merge during this time period
 
+      - name: repository
+        description: The name of the repository
+        tests:
+          - not_null
+
       - name: sum_days_pr_open
         description: The total number of days a pull request opened during this time period was open
 
@@ -217,11 +229,15 @@ models:
   - name: github__weekly_metrics
     description: >
       Summary numbers for issues and pull requests by week
+    tests:
+      - dbt_utils.unique_combination_of_columns:
+          combination_of_columns:
+            - repository
+            - week
     columns:
       - name: week
         description: The reporting week
         tests:
-          - unique
           - not_null
       - name: number_issues_opened
         description: The total number of issues created during this time period
@@ -249,15 +265,24 @@ models:
 
       - name: longest_days_pr_open
         description: The longest number of days a pull request opened during this time period was open
+
+      - name: repository
+        description: The name of the repository
+        tests:
+          - not_null
 
   - name: github__monthly_metrics
     description: >
       Summary numbers for issues and pull requests by month
+    tests:
+      - dbt_utils.unique_combination_of_columns:
+          combination_of_columns:
+            - repository
+            - month
     columns:
       - name: month
         description: The reporting month
         tests:
-          - unique
           - not_null
       - name: number_issues_opened
         description: The total number of issues created during this time period
@@ -285,15 +310,24 @@ models:
 
       - name: longest_days_pr_open
         description: The longest number of days a pull request opened during this time period was open
+
+      - name: repository
+        description: The name of the repository
+        tests:
+          - not_null
 
   - name: github__quarterly_metrics
     description: >
       Summary numbers for issues and pull requests by quarter
+    tests:
+      - dbt_utils.unique_combination_of_columns:
+          combination_of_columns:
+            - repository
+            - quarter
     columns:
       - name: quarter
         description: The reporting quarter
         tests:
-          - unique
           - not_null
       - name: number_issues_opened
         description: The total number of issues created during this time period
@@ -321,4 +355,8 @@ models:
 
       - name: longest_days_pr_open
         description: The longest number of days a pull request opened during this time period was open
-
+
+      - name: repository
+        description: The name of the repository
+        tests:
+          - not_null

models/github__daily_metrics.sql

@@ -9,64 +9,80 @@ pull_requests as (
 ), 
 
 issues_opened_per_day as (
-   select 
-      {{ dbt.date_trunc('day', 'created_at') }} as day, 
+    select 
+      {{ dbt.date_trunc('day', 'created_at') }} as day,
+      repository as repository,
       count(*) as number_issues_opened,
       sum(days_issue_open) as sum_days_issue_open,
       max(days_issue_open) as longest_days_issue_open
     from github_issues
-    group by 1
+    group by 
+      1,2
 ), 
 
 issues_closed_per_day as (
-   select 
-      {{ dbt.date_trunc('day', 'closed_at') }} as day, 
+    select 
+      {{ dbt.date_trunc('day', 'closed_at') }} as day,
+      repository as repository,
       count(*) as number_issues_closed
     from github_issues
     where closed_at is not null
-    group by 1
+    group by 
+      1,2
 ), 
 
 prs_opened_per_day as (
-   select 
-      {{ dbt.date_trunc('day', 'created_at') }} as day, 
+    select 
+      {{ dbt.date_trunc('day', 'created_at') }} as day,
+      repository as repository,
       count(*) as number_prs_opened,
       sum(days_issue_open) as sum_days_pr_open,
       max(days_issue_open) as longest_days_pr_open
     from pull_requests
-    group by 1
+    group by 
+      1,2
 ), 
 
 prs_merged_per_day as (
-   select 
+    select 
       {{ dbt.date_trunc('day', 'merged_at') }} as day, 
+      repository as repository,
       count(*) as number_prs_merged
     from pull_requests
     where merged_at is not null
-    group by 1
+    group by
+      1,2
 ), 
 
 prs_closed_without_merge_per_day as (
-   select 
-      {{ dbt.date_trunc('day', 'closed_at') }} as day, 
+    select 
+      {{ dbt.date_trunc('day', 'closed_at') }} as day,
+      repository as repository,
       count(*) as number_prs_closed_without_merge
     from pull_requests
     where closed_at is not null
       and merged_at is null
-    group by 1
+    group by
+      1,2
 ), 
 
 issues_per_day as (
     select 
       coalesce(issues_opened_per_day.day, 
         issues_closed_per_day.day
       ) as day,
+      coalesce(issues_opened_per_day.repository, 
+        issues_closed_per_day.repository
+      ) as repository,
       number_issues_opened,
       number_issues_closed,      
       sum_days_issue_open,
       longest_days_issue_open
     from issues_opened_per_day
-    full outer join issues_closed_per_day on issues_opened_per_day.day = issues_closed_per_day.day
+    full outer join issues_closed_per_day
+    on 
+      issues_opened_per_day.day = issues_closed_per_day.day
+      and issues_opened_per_day.repository = issues_closed_per_day.repository
 ), 
 
 prs_per_day as (
@@ -75,18 +91,29 @@ prs_per_day as (
         prs_merged_per_day.day,
         prs_closed_without_merge_per_day.day
       ) as day,
+      coalesce(prs_opened_per_day.repository, 
+        prs_merged_per_day.repository,
+        prs_closed_without_merge_per_day.repository
+      ) as repository,
       number_prs_opened,
       number_prs_merged,
       number_prs_closed_without_merge,
       sum_days_pr_open,
       longest_days_pr_open
     from prs_opened_per_day
-    full outer join prs_merged_per_day on prs_opened_per_day.day = prs_merged_per_day.day
-    full outer join prs_closed_without_merge_per_day on coalesce(prs_opened_per_day.day, prs_merged_per_day.day) = prs_closed_without_merge_per_day.day
+    full outer join prs_merged_per_day 
+    on
+      prs_opened_per_day.day = prs_merged_per_day.day
+      and prs_opened_per_day.repository = prs_merged_per_day.repository
+    full outer join prs_closed_without_merge_per_day 
+    on
+      coalesce(prs_opened_per_day.day, prs_merged_per_day.day) = prs_closed_without_merge_per_day.day
+      and coalesce(prs_opened_per_day.repository, prs_merged_per_day.repository) = prs_closed_without_merge_per_day.repository
 )
 
 select 
   coalesce(issues_per_day.day, prs_per_day.day) as day,
+  coalesce(issues_per_day.repository, prs_per_day.repository) as repository,
   coalesce(number_issues_opened, 0) as number_issues_opened,
   coalesce(number_issues_closed, 0) as number_issues_closed,
   sum_days_issue_open,
@@ -97,5 +124,7 @@ select
   sum_days_pr_open,
   longest_days_pr_open
 from issues_per_day 
-full outer join prs_per_day on issues_per_day.day = prs_per_day.day
-order by day desc
+full outer join prs_per_day 
+on
+  issues_per_day.day = prs_per_day.day
+  and issues_per_day.repository = prs_per_day.repository

models/github__monthly_metrics.sql

@@ -5,6 +5,7 @@ with daily_metrics as (
 
 select 
   {{ dbt.date_trunc('month', 'day') }} as month, 
+  repository as repository,
   sum(number_issues_opened) as number_issues_opened,
   sum(number_issues_closed) as number_issues_closed,
   sum(sum_days_issue_open) / sum(number_issues_opened) as avg_days_issue_open,
@@ -15,5 +16,4 @@ select
   sum(sum_days_pr_open) / sum(number_prs_opened) as avg_days_pr_open,
   max(longest_days_pr_open) as longest_days_pr_open
 from daily_metrics 
-group by 1
-order by 1 desc
+group by 1,2

models/github__quarterly_metrics.sql

@@ -5,6 +5,7 @@ with daily_metrics as (
 
 select 
   {{ dbt.date_trunc('quarter', 'day') }} as quarter, 
+  repository as repository,
   sum(number_issues_opened) as number_issues_opened,
   sum(number_issues_closed) as number_issues_closed,
   sum(sum_days_issue_open) / sum(number_issues_opened) as avg_days_issue_open,
@@ -16,5 +17,4 @@ select
   max(longest_days_pr_open) as longest_days_pr_open
 
 from daily_metrics 
-group by 1
-order by 1 desc
+group by 1,2