Create klar_android.active users aggregates (#5624)

* Changes to active_users_aggregates.

* Remove days_since_seen and calculate activity segment.

* Remove is_core_active not required.

* Remove days_seen_bits from mobile queries not required.

* Adjust mobile query to use the baseline_clients_last_seen views.

* Update Focus Android query to use the baseline_clients_last_seen view.

* Get activity segments from the views instead of hard-coding it. Query attribution only when neccesary to improve performance. Delay update by 1 day to get metrics ping's data.

* Get app_name from clients_last_seen_v2 view.

* Set activity_segment as NULL for legacy Focus Android.

* Formatting

* Remove first_seen_date and correct column name in desktop_query.sql

* Update active_users_aggregates query for Desktop to get app_name and growth metrics from clients_last_seen_v2, replace language_name with locale and remove search metrics based on sprint decision (see DENG-1989).

* Update active_users_aggregates query for Mobile to remove search metrics based on sprint decision (see DENG-1989) and calculate the min metrics ping received between the current and next date, given that these pings can arrive in the same or next date as the baseline ping.

* Revert changes to Desktop queries in this PR, as this implementation is separated to [PR-5607](#5607).

* Space

* Formatting.

* Change name of activity_segment to segment.

* Add . Klar Android query and join in the Mobile view.

* Fix reference to app name.

* Update app_anme in checks for Focus

* Add klar android to the checks and update app_name in checks for Focus.

* Formatting.

* Use active_users instead of baseline_clients_last_seen and improve retrieving metrics ping's data.

* Formatting.

* --- (#5623)

updated-dependencies:
- dependency-name: requests
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Anna Scholtz <[email protected]>

* Update backfill.yaml (#5629)

* DENG-3719: Allow setting billing project for managed backfills (#5605)

* Added default billing project and param

* DENG-3851 Updated managed backfill docs with slack channel for notifcations (#5630)

* Updated docs

* Update docs/cookbooks/creating_a_derived_dataset.md

Co-authored-by: Alexander <[email protected]>

---------

Co-authored-by: Alexander <[email protected]>

* Inject some errors to fenix metrics_clients_last_seen_v1 to test observability tooling (#5631)

* feat: add kpi_support_metrics SQL generator (#5594)

* feat: add kpi_support_metrics SQL generator

* feat: replace baseline_clients_last_seen_activity_extended with active_users

* feat: add template to union all views inside telemetry namespace

* feat: remove retention queries for firefox_ios and fenix as now they are generated via kpi_support_metrics generator

* bug: correct app_name based logic used by templating

* bug: fix formatting inside kpi_support_metrics templates

* feat: implement feedback from PR by removing mozilla online logic as no longer needed and correcting app_name setting

* feat: fix active_users union mismatch

* feat: remove scheduling for now, will be added in the next PR.

* feat: make sure we user product_specific_attribution_fields additional field list instead of app_name logic in templates (#5633)

* delete new pageload experiments table (#5634)

* Use information_schema to find experiment tables for shredder (#5635)

* Update retention of search terms sanitized datasets (#5577)

* Be explicit about columns in clients_yearly (#5636)

* Add distribution_id and geo_subdivision to clients_yearly schema (#5637)

* Added two new fields to col list (#5639)

* feat: add backfill.yaml to org_mozilla_fennec_aurora_derived.baseline_clients_daily_v1 (#5632)

* Add flipkart and home_depot views (#5638)

* Add flipkart and home_depot views

* remove manually created views. use the user_facing view syndication instead

* fix yaml lint

* DENG-3288 add Focus iOS and Focus Android and switch to use new views instead of extended views (#5641)

* DENG-3288 Add "is_mobile" to mobile_engagement_clients_v1 and mobile_engagement_v1 (#5648)

* Update `bqetl_subplat` DAG description with new expected Stripe report error message. (#5642)

* feat: add schema files to baseline_clients_daily_v1 and baseline_clients_last_seen_v1 (#5646)

* fix: add fields: map to baseline_daily and baseline_last_seen schemas (#5651)

* feat: update kpi_support_metrics generation to append platform to the unioned view name (#5644)

Co-authored-by: Katie Windau <[email protected]>

* fix(backfills): switch to concurrent.futures to improve debuggability (#5653)

* fix(backfills): followup to concurrent.futures, raise Error if any failed dates (#5654)

* Reprocess past two days in jobs_by_organization_v1 (#5652)

* feat: rename baseline_last_seen references to active_users and add "platform" to only include platform specific fields when it matches (#5645)

* fix(backfills): switch from process to thread to avoid pickling bigquery object (#5656)

* feat: rename kpi_support_metrics to mobile_kpi_support_metrics (#5657)

* [RS-1246] Fix DDG DAU in search_revenue_levers_daily (#5655)

* Fix search_revenue_levers_daily

* Query formatting

---------

Co-authored-by: m-d-bowerman <[email protected]>

* DENG-2975 active_users (#5478)

* Add is_desktop to engagement and retention tables (#5658)

* add is_desktop to engagement and retention tables

* replace clients_last_seen by active_users

* format

* remove unused normalized_channel

* [DENG-3889] Rewrite events_stream_v1 for better performance (#5659)

* Add more null channels for metrics_clients_last_seen test (#5662)

* chore(glam): accommodate for dag reschedule (#5664)

* [DENG-3889] Replace events_stream temp udfs with inline usages (#5666)

* Add telemetry_derived.latest_version to data-observability-dev (#5663)

* [RS-834] Add country to search_revenue_levers_daily (#4739)

* Add country

* change channel spellings so match revenue table

* update country list

* Reformat

* Update desktop DAU definition

Can't update mobile DAU by engine breakouts w/o changes to mobile_search_clients_daily. But we also don't need that option since default engine is currently buggy and searchers presumably have > 0 active seconds

* Fix sql format

---------

Co-authored-by: Alekhya <[email protected]>
Co-authored-by: Alekhya Kommasani <[email protected]>

* Fix PyPI package publishing in CI (#5669)

* Fix PyPI package publishing in CI

* Update .circleci/workflows.yml

Co-authored-by: Mike Williams <[email protected]>

---------

Co-authored-by: Mike Williams <[email protected]>

* Use ad group ID to get campaign info (#5670)

* feat: move mobile engagement logic into mobile kpi support metrics generator (#5661)

* feat: move mobile_engagement SQL logic into mobile_kpi_support_metrics generator

* feat: remove mobile_engagement queries from telemetry and telemetry_erived

* feat: apply formatting to engagement_clients view and add checks template

* feat: add engagement view template and add engagement templates to the generation logic

* bug: fix engagement query errors

* feat: sneak in removing of retention views inside telemetry (mobile kpi generator will overwrite these anyways)

* feat: remove bqetl_mobile_engagement_model DAG from DAG list as no longer needed.

* feat: fix an engagement_v1 query template and removed unecessary comment from the metadata template

* fix: update engagement templates to ensure valid query is rendered

* [DENG-3905] Support queries with temp udfs when billing project is set (#5668)

* DENG-3237: Added deletion dates to deprecated tables (#5660)

* Added deletion dates

* fix: metadata.yaml generation change to allow backfill command execution (#5671)

* RS-788 Add support for organic searches with ads to the mobile search counts tables (#5598)

* adding organic searches with ads to this table

* updating mobile_search_aggregates table with search_with_ads_organic column

* updating the search revenue lever table

- include search_with_ads_organic columns for Bing, Google and DDG

* Fix CI issues

* Fix tests CI failure

* fix tests

* Fix test sql failure

* Update query.sql

reverting back to original code for search_revenue_levers table

---------

Co-authored-by: Alekhya Kommasani <[email protected]>
Co-authored-by: Alekhya <[email protected]>

* Update desktop active_users view to make dimensions available for KPIs. (#5675)

* Use desktop active_users view to make dimensions available for KPIs.

* Naming adjustment

* Fix reference to app name.

* Use active_users instead of baseline_clients_last_seen and improve retrieving metrics ping's data.

* Formatting.

* Get distribution_id for Fenix.

* Get distribution_id for Fenix.

* Format correction

* Skip unavailable data, becuase it's not KPI relevant.

* Formatting

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Anna Scholtz <[email protected]>
Co-authored-by: Teon L Brooks <[email protected]>
Co-authored-by: Winnie Chan <[email protected]>
Co-authored-by: Alexander <[email protected]>
Co-authored-by: kik-kik <[email protected]>
Co-authored-by: Ben Wu <[email protected]>
Co-authored-by: whd <[email protected]>
Co-authored-by: Frank Bertsch <[email protected]>
Co-authored-by: wil stuckey <[email protected]>
Co-authored-by: Katie Windau <[email protected]>
Co-authored-by: Sean Rose <[email protected]>
Co-authored-by: skahmann3 <[email protected]>
Co-authored-by: m-d-bowerman <[email protected]>
Co-authored-by: Vanessa Sabino <[email protected]>
Co-authored-by: Eduardo Filho <[email protected]>
Co-authored-by: Alekhya <[email protected]>
Co-authored-by: Alekhya Kommasani <[email protected]>
Co-authored-by: Mike Williams <[email protected]>
Co-authored-by: Preethi Issac <[email protected]>

Author: 20 people

sql_generators/active_users_aggregates_v3/__init__.py

@@ -50,9 +50,11 @@
     "klar_ios": [
         {"table": "`moz-fx-data-shared-prod.org_mozilla_ios_klar_live.baseline_v1`"}
     ],
+    "klar_android" : [
+        {"table" : "`moz-fx-data-shared-prod.org_mozilla_klar_live.baseline_v1`"}
+    ],
 }
 
-
 class Browsers(Enum):
     """Enumeration with browser names and equivalent dataset names."""
 
@@ -62,6 +64,7 @@ class Browsers(Enum):
     focus_android = "Focus Android"
     firefox_ios = "Firefox iOS"
     klar_ios = "Klar iOS"
+    klar_android = "Klar Android"
 
 
 @click.command()
@@ -129,6 +132,7 @@ def generate(target_project, output_dir, use_cloud_function):
                 )
             )
             schema_template = mobile_schema_template
+
         # create checks_sql
         if browser.name == "firefox_desktop":
             checks_sql = desktop_checks_template.render(
@@ -248,6 +252,7 @@ def generate(target_project, output_dir, use_cloud_function):
                 focus_android_dataset=Browsers("Focus Android").name,
                 firefox_ios_dataset=Browsers("Firefox iOS").name,
                 klar_ios_dataset=Browsers("Klar iOS").name,
+                klar_android_dataset=Browsers("Klar Android").name,
             )
         ),
         skip_existing=False,

sql_generators/active_users_aggregates_v3/templates/mobile_query.sql

@@ -62,10 +62,15 @@ metrics AS (
     ARRAY_AGG(normalized_channel IGNORE NULLS ORDER BY submission_date ASC)[
       SAFE_OFFSET(0)
     ] AS normalized_channel,
-    ARRAY_AGG(uri_count IGNORE NULLS ORDER BY submission_date ASC)[SAFE_OFFSET(0)] AS uri_count,
-    ARRAY_AGG(is_default_browser IGNORE NULLS ORDER BY submission_date ASC)[
-      SAFE_OFFSET(0)
-    ] AS is_default_browser
+    {% if app_name == "klar_android"%}
+      CAST(NULL AS INTEGER) AS uri_count,
+      CAST(NULL AS INTEGER) AS is_default_browser,
+    {% else %}
+      ARRAY_AGG(uri_count IGNORE NULLS ORDER BY submission_date ASC)[SAFE_OFFSET(0)] AS uri_count,
+      ARRAY_AGG(is_default_browser IGNORE NULLS ORDER BY submission_date ASC)[
+        SAFE_OFFSET(0)
+      ] AS is_default_browser
+    {% endif %}
   FROM
     `{{ project_id }}.{{ app_name }}.metrics_clients_last_seen`
   WHERE

sql_generators/active_users_aggregates_v3/templates/mobile_view.sql

@@ -26,3 +26,8 @@ SELECT
   *
 FROM
   `{{ project_id }}.{{ focus_android_dataset }}.active_users_aggregates`
+UNION ALL
+SELECT
+  *
+FROM
+  `{{ project_id }}.{{ klar_android_dataset }}.active_users_aggregates`