Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v5.0.0] Add query acceleration costs, incrementalize cost per query, account for new service type #141

Merged
merged 5 commits into from
Jan 14, 2024

Conversation

ian-whitestone
Copy link
Contributor

@ian-whitestone ian-whitestone commented Jan 14, 2024

This PR does the following:

  • cost_per_query model changes
    • Convert to incremental model
    • Use metering_history instead of warehouse_metering_history for consistency
    • Add in query acceleration costs
  • Update stg_metering_history and stg_warehouse_metering_history to re-process the last 7 days
    • In order to do this for stg_metering_history, we need to add a new field that is part of the unique key, entity_id. will require a full refresh.
  • Account for new WAREHOUSE_METERING service_type we are seeing in rate_sheet_daily

cost per query reconciliations:

check that everything lines up, ✅

with
existing as (
    select 
        query_id,
        start_time,
        query_cost,
        query_credits
    from cost_per_query -- current model in prod
    where start_time between '2023-11-01' and '2023-12-01'
),
new as (
    select 
        query_id,
        start_time,
        query_cost,
        query_credits
    from dev.ian_dbt_snowflake_monitoring.cost_per_query
    where start_time between '2023-11-01' and '2023-12-01'
)
select
    *
from existing
full outer join new
    on existing.query_id=new.query_id
where
    existing.query_cost <> new.query_cost
    or existing.query_credits <> new.query_credits
    or existing.query_id is null
    or new.query_id is null
;

check account with QA enabled, costs line up if you remove QA ✅

with
existing as (
    select 
        query_id,
        start_time,
        query_cost,
        query_credits
    from cost_per_query -- prod model
    where start_time between '2024-01-01' and '2024-01-05' -- spot checked multiple sets of dates
),
new as (
    select 
        query_id,
        start_time,
        query_cost - query_acceleration_cost as query_cost,
        query_credits - query_acceleration_credits as query_credits
    from dev.ian_dbt_snowflake_monitoring.cost_per_query
    where start_time between '2024-01-01' and '2024-01-05'
)
select
    *
from existing
full outer join new
    on existing.query_id=new.query_id
where
    round(existing.query_cost, 4) <> round(new.query_cost, 4)
    or round(existing.query_credits, 4) <> round(new.query_credits, 4)
    or existing.query_id is null
    or new.query_id is null
;

@ian-whitestone ian-whitestone temporarily deployed to Approve Integration Tests January 14, 2024 19:34 — with GitHub Actions Inactive
@ian-whitestone ian-whitestone temporarily deployed to Approve Integration Tests January 14, 2024 19:34 — with GitHub Actions Inactive
@ian-whitestone ian-whitestone marked this pull request as ready for review January 14, 2024 19:34
@ian-whitestone ian-whitestone temporarily deployed to Approve Integration Tests January 14, 2024 19:40 — with GitHub Actions Inactive
@ian-whitestone ian-whitestone temporarily deployed to Approve Integration Tests January 14, 2024 19:40 — with GitHub Actions Inactive
@ian-whitestone ian-whitestone changed the title Add query acceleration costs, incrementalize cost per query, account for new service type [v5.0.0] Add query acceleration costs, incrementalize cost per query, account for new service type Jan 14, 2024
@ian-whitestone ian-whitestone merged commit b69f913 into main Jan 14, 2024
3 checks passed
@ian-whitestone ian-whitestone deleted the buncha_stuff branch January 14, 2024 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants