Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added ability to exclude files based on subfolder name #82

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

vvvito
Copy link
Contributor

@vvvito vvvito commented Oct 29, 2024

Resolves #81.

Adding an additional argument called model_path_exclusion_filter which is a list of strings that holds glob-style patterns to parse through table original_file_path in order to exclude tables from the doc compute function.

Here is my dbt folder structure

📁models 
    📁dim
    - dim_core_dates.sql
    - dim_core_forms.sql
    - dim_core_patients.sql
        📁stg
        - dim_stg_dates.sql
        - dim_stg_forms.sql
        - dim_stg_patients.sql
      

Example:

# baseline
dbt-coverage compute doc --cov-format markdown --model-path-filter models/dim

Returns the following:

# Coverage report (doc)
| Model | Columns Covered | % |
|:------|----------------:|:-:|
| dim.dim_core_clinic_programs                                           |     0/7     |   0.0% |
| dim.dim_core_dates                                                     |     0/72    |   0.0% |
| dim.dim_core_episodes                                                  |    18/43    |  41.9% |
| dim.dim_core_form_details                                              |    11/15    |  73.3% |
| dim.dim_core_forms                                                     |    13/15    |  86.7% |
| dim.dim_core_locations                                                 |     0/26    |   0.0% |
| dim.dim_core_orders                                                    |    21/22    |  95.5% |
| dim.dim_core_patient_current_identifiers                               |     8/11    |  72.7% |
| dim.dim_core_patients                                                  |    25/39    |  64.1% |
| dim.dim_core_providers                                                 |     0/15    |   0.0% |
| dim.dim_core_time_hourly                                               |     0/4     |   0.0% |
| dim.dim_core_time_periods                                              |     0/13    |   0.0% |
| dim.dim_core_unit_programs                                             |     4/4     | 100.0% |
| dim_stg.stg_core_appointment_types                                     |     0/22    |   0.0% |
| dim_stg.stg_core_code_value_mappings                                   |     0/10    |   0.0% |
| dim_stg.stg_core_code_values                                           |     0/11    |   0.0% |
| dim_stg.stg_core_episodes                                              |     0/49    |   0.0% |
| dim_stg.stg_core_form_details                                          |     0/15    |   0.0% |
| dim_stg.stg_core_forms                                                 |     0/12    |   0.0% |
| dim_stg.stg_core_locations                                             |     0/26    |   0.0% |
| dim_stg.stg_core_orders                                                |     0/20    |   0.0% |
| dim_stg.stg_core_pathology_results                                     |     0/25    |   0.0% |
| dim_stg.stg_core_patient_identifiers                                   |     0/9     |   0.0% |
| dim_stg.stg_core_patients                                              |     0/40    |   0.0% |
| dim_stg.stg_core_providers                                             |     0/15    |   0.0% |
| dim_stg.stg_core_radiology_results                                     |     0/25    |   0.0% |
| Total                                                                  |   100/565   |  17.7% |

All the models that are are prefixed with a 'stg' are staging models, and should not be included in the coverage compute.

Instead

# compute doc coverage for models found in the `dim` folder but exclude models found in the `stg` sub-folder
dbt-coverage compute doc --cov-format markdown --model-path-exclusion-filter */stg --model-path-filter models/dim

Returns the following:

# Coverage report (doc)
| Model | Columns Covered | % |
|:------|----------------:|:-:|
| dim.dim_core_clinic_programs                                           |     0/7     |   0.0% |
| dim.dim_core_dates                                                     |     0/72    |   0.0% |
| dim.dim_core_episodes                                                  |    18/43    |  41.9% |
| dim.dim_core_form_details                                              |    11/15    |  73.3% |
| dim.dim_core_forms                                                     |    13/15    |  86.7% |
| dim.dim_core_locations                                                 |     0/26    |   0.0% |
| dim.dim_core_orders                                                    |    21/22    |  95.5% |
| dim.dim_core_patient_current_identifiers                               |     8/11    |  72.7% |
| dim.dim_core_patients                                                  |    25/39    |  64.1% |
| dim.dim_core_providers                                                 |     0/15    |   0.0% |
| dim.dim_core_time_hourly                                               |     0/4     |   0.0% |
| dim.dim_core_time_periods                                              |     0/13    |   0.0% |
| dim.dim_core_unit_programs                                             |     4/4     | 100.0% |
| Total                                                                  |   100/286   |  35.0% |

How to use

Pass in as many --model-path-exclusion-filter arguments you wish to exclude any number of subfolders from being computed

Example:

# This will exclude tables found under a subfolder called 'bronze' or 'silver' any level deep, or a 'stg' subfolder, found within a 'mock' subfolder, any level deep.
dbt-coverage compute doc --model-path-exclusion-filter */bronze --model-path-exclusion-filter */silver --model-path-exclusion-filter */mock/stg

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add the ability to exclude subfolders from computing doc coverage
1 participant