convert cent based fields to major currency unit based

.buildkite/scripts/run_models.sh

@@ -19,7 +19,7 @@ dbt deps
 dbt seed --target "$db" --full-refresh
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"
-dbt run --vars '{stripe__using_invoice_line_sub_filter: false, stripe__using_credit_notes: true, stripe__using_price: false, stripe__using_subscription_history: false}' --full-refresh --target "$db"
+dbt run --vars '{stripe__using_invoice_line_sub_filter: false, stripe__using_credit_notes: true, stripe__using_price: false, stripe__using_subscription_history: false, stripe__convert_values: true}' --full-refresh --target "$db"
 dbt test --target "$db"
 
 dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"

CHANGELOG.md

@@ -1,6 +1,25 @@
-# dbt_stripe_source version.version
+# dbt_stripe_source v0.12.2
+[PR [#87](https://github.com/fivetran/dbt_stripe_source/pull/87)] includes the following updates:
+
+## Feature Updates: Optionally convert Amounts to Major Units
+- Stripe passes amount-based fields, such as `amount`, `net`, and `fee`, in the smallest denomination as raw form. This means, if your currency has minor and major units such as USD, 100 represents 100 cents, the minor unit, or 1 USD, the major unit. Alternatively, if your currency doesn't use minor units such as JPY, 100 represents 100 JPY. 
+
+- This PR introduces a variable `stripe__convert_values` (disabled by default) which allows users the option to divide all amount-based fields by 100.
+  - For information on how to enable the division, refer to the [README]((https://github.com/fivetran/dbt_stripe_source?tab=readme-ov-file#enabling-cent-to-dollar-conversion)) on configuring the `stripe__convert_values` variable.
+  - Otherwise, amount-based fields will be brought through in their raw form.
+  - Examples of currencies using minor units (in which enabling `stripe__convert_values` is relevant) include USD, Euro, and CAD.
+  - Examples of currencies NOT using minor units (in which is makes more sense to keep the amount-based fields in raw form) include Japanese Yen (JPY), Indonesian Rupiah (IDR), and Korean Won (KRW).
+
+## Under the Hood
+- Introduces the `convert_values` macro that holds the logic for enabling the division by 100, referenced by all amount-based fields.
+- Updates the `run_models.sh` script to test for when `stripe__convert_values` is set to True.
+
+## Notes
+- Note that in a future release, the division by 100 will be enabled by default instead.
+- Currently this package does not support multiple currencies, but we have created a [feature flag to support multiple currencies](https://github.com/fivetran/dbt_stripe/issues/102) where you are welcome to provide feedback or contribute to the discussion.
 
 ## Documentation
+- Updates the descriptions for all amount-based fields to specify the grain of the values and add information about the `stripe__convert_values` variable.
 - Corrected references to connectors and connections in the README. ([#88](https://github.com/fivetran/dbt_stripe_source/pull/88))
 
 # dbt_stripe_source v0.12.1

README.md

@@ -1,4 +1,6 @@
-<p align="center">
+# Stripe Source dbt Package ([Docs](https://fivetran.github.io/dbt_stripe_source/))
+
+<p align="left">
     <a alt="License"
         href="https://github.com/fivetran/dbt_stripe_source/blob/main/LICENSE">
         <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a>
@@ -13,7 +15,6 @@
         <img src="https://img.shields.io/badge/Fivetran_Quickstart_Compatible%3F-yes-green.svg" /></a>
 </p>
 
-# Stripe Source dbt Package ([Docs](https://fivetran.github.io/dbt_stripe_source/))
 ## What does this dbt package do?
 <!--section="stripe_source_model"-->
 - Materializes [Stripe staging tables](https://fivetran.github.io/dbt_stripe_source/#!/overview/stripe_source/models/?g_v=1&g_e=seeds) which leverage data in the format described by [this ERD](https://fivetran.com/docs/applications/stripe/#schemainformation). These staging tables clean, test, and prepare your Stripe data from [Fivetran's connector](https://fivetran.com/docs/applications/stripe) for analysis by doing the following:
@@ -45,7 +46,7 @@ If you  are **not** using the [Stripe transformation package](https://github.com
 ```yaml
 packages:
   - package: fivetran/stripe_source
-    version: [">=0.12.0", "<0.13.0"]
+    version: [">=0.13.0", "<0.14.0"]
 ```
 ### Step 3: Define database and schema variables
 By default, this package runs using your destination and the `stripe` schema. If this is not where your stripe data is (for example, if your stripe schema is named `stripe_fivetran`), add the following configuration to your root `dbt_project.yml` file:
@@ -178,6 +179,17 @@ vars:
     stripe__subscription_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
 ```
 
+#### Enabling Cent to Dollar Conversion
+
+Amount-based fields, such as `amount` and `net`, are typically displayed in the smallest denomination (e.g., cents for USD). By default, amount-based fields will be in this raw form. However, some currencies use major and minor units (for example, cents and dollars when using USD). In these cases, it may be useful to divide the amounts by 100, converting amounts to major units (dollars for USD). To enable the division, configure the `stripe__convert_values` to `true` in your project.yml: 
+
+```yml
+vars:
+    stripe__convert_values: true  # default is false 
+```
+
+If you are working in a currency that does not differentiate between minor and major units, such as JPY or KRW, it may make more sense to keep the amount-based fields in raw form and therefore the package can be ran without configuration. As `stripe__convert_values` is disabled by default, these fields will not be impacted.
+
 #### Passing Through Additional Fields
 This package includes all source columns defined in the macros folder. You can add more columns using our pass-through column variables. These variables allow for the pass-through fields to be aliased (`alias`) and casted (`transform_sql`) if desired, but not required. Datatype casting is configured via a sql snippet within the `transform_sql` key. You may add the desired sql while omitting the `as field_name` at the end and your custom pass-though fields will be casted accordingly. Use the below format for declaring the respective pass-through variables:
 

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'stripe_source'
-version: '0.12.1'
+version: '0.12.2'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:

integration_tests/ci/sample.profiles.yml

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: stripe_source_integrations_tests_6
+      schema: stripe_source_integrations_tests_8
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: stripe_source_integrations_tests_6
+      schema: stripe_source_integrations_tests_8
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: stripe_source_integrations_tests_6
+      schema: stripe_source_integrations_tests_8
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: stripe_source_integrations_tests_6
+      schema: stripe_source_integrations_tests_8
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: stripe_source_integrations_tests_6
+      schema: stripe_source_integrations_tests_8
       threads: 2
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,14 +1,14 @@
 config-version: 2
 
 name: 'stripe_source_integration_tests'
-version: '0.12.1'
+version: '0.12.2'
 
 
 profile: 'integration_tests'
 
 vars:
-  stripe_schema: stripe_source_integrations_tests_6
-  
+  stripe_schema: stripe_source_integrations_tests_8
+
   stripe_source:
     card_pass_through_columns:
       - name: "description"

macros/convert_values.sql

@@ -0,0 +1,7 @@
+{% macro convert_values(field_name, divide_by=100.0, divide_var=var('stripe__convert_values',false), alias=None) %}
+    {% if divide_var == true %}
+        {{ field_name }} / {{ divide_by }} as {{ alias if alias else field_name }}
+    {% else %}
+        {{ field_name }} as {{ alias if alias else field_name }}
+    {% endif %}
+{% endmacro %}

models/docs.md

@@ -37,3 +37,7 @@ ZIP or postal code.
 {% docs state -%}
 State, county, province, or region.
 {%- enddocs %}
+
+{% docs convert_values -%}
+By default, this value kept in its raw form. However, in some currencies, it may be useful to divide the value by 100 to get the major currency unit. This works for scenarios in which the currency has a minor unit, which is displayed in the raw data, and major unit. For example, if in USD, if the raw value is 100, this represents 100 cents which then gets converted to $1 dollar, the major currency unit. To invoke the division, set the variable stripe__convert_values to True (False by default). If your currency does not contain minor units (for example the Japanese Yen ¥ which does not use decimals), you can run the package as-is. Note that in a future release, the division by 100 will be enabled by default instead.
+{%- enddocs %}