Add telemetry black box tests (#821)

This change required new testing command line flags to shorten the
upload interval of metrics. This included the startup delay and initial
upload interval of the metrics libraries.

I also used this as an opportunity to make some improvements to the
black box README.

b/287248166

Change-Id: I469c6b3c4d4a34fb7d5770c5030e5cecb077b396

cobalt/black_box_tests/README.md

@@ -15,37 +15,57 @@ interaction with different web server behavior.
   3. Re-fetch network requests after network shortage.
 
 
+## Adding a New Test
+
+  1. Add a python test script in tests/.
+     1.  The python script, at a minimum, needs to load your HTML (see step #2)
+         and call both `runner.WaitForJSTestsSetup()` and
+         `self.assertTrue(runner.JSTestsSucceeded())`. See one of the many python
+         scripts in this directory as an example.
+  2. Add target web page(s) and associated resources(if any) to testdata/.
+     1.  At a minimum, call `setupFinished()` and `onEndTest()` when the test is
+         ready to run and when it's completed running respectively. You'll need
+         to have loaded `<script src='black_box_js_test_utils.js'></script>` for
+         these utility methods to be available.
+  3. Add the test name (name of the python test script) to black_box_tests.py
+     to automate new test. Add the name to the list of tests requiring
+     app launcher support for system signals (e.g. suspend/resume), or the list
+     of tests requiring deep link support, or the list of tests that don't.
+
+
 ## Running Tests:
 
   1. To run all tests:
 
-     $ python path/to/black_box_tests.py --platform PLATFORM --config CONFIG
+     `$ python path/to/black_box_tests.py --platform PLATFORM --config CONFIG`
 
      e.g.
-     $ python path/to/black_box_tests.py --platform linux-x64x11 --config
-       devel
+     `$ python path/to/black_box_tests.py --platform linux-x64x11 --config
+       devel`
 
 
   2. To run an individual test:
 
-     $ python path/to/black_box_tests.py --platform PLATFORM --config CONFIG
-       --test_name TEST_NAME
+     `$ python path/to/black_box_tests.py --platform PLATFORM --config CONFIG
+       --test_name TEST_NAME`
 
      e.g.
-     $ python path/to/black_box_tests.py --platform linux-x64x11 --config devel
-       --test_name preload_font
+     `$ python path/to/black_box_tests.py --platform linux-x64x11 --config devel
+       --test_name preload_font`
 
-     You can apply --gtest_filter through --target_params, e.g.
-     $ python path/to/black_box_tests.py --platform linux-x64x11 --config devel
-       --test_name web_platform_tests
-       --target_params="--gtest_filter=fetch/*"
+     You can apply `--gtest_filter` through `--target_params`, e.g.
+     ```
+       $ python path/to/black_box_tests.py --platform linux-x64x11 --config devel
+         --test_name web_platform_tests
+         --target_params="--gtest_filter=fetch/*"
+     ```
 
      Note: When declaring a gtest_filter, it will override the filters defined
      in GetWebPlatformTestFilters in configuration.py.
 
   If you see a HTTPConnection error which eventually
   causes a TimeoutException for the test, you might want
-  to add "--server_binding_address localhost" in your
+  to add `--server_binding_address localhost` in your
   python command.
 
 ## Tests
@@ -59,36 +79,29 @@ BlackBoxCobaltRunner class.
 
 A wrapper around the app launcher. BlackBoxCobaltRunner includes a webdriver
 module attached to the app launcher's Cobalt instance after it starts running.
-Includes a method(JSTestsSucceeded()) to check test result on the JavaScript
+Includes a method(`JSTestsSucceeded()`) to check test result on the JavaScript
 side. Call this method to wait for JavaScript test result.
 black_box_js_test_utils.js provides some utility functions that are meant to
-work with runner.JSTestsSucceeded() in the python test scripts. Together,
+work with `runner.JSTestsSucceeded()` in the python test scripts. Together,
 they allow for test logic to exist in either the python test scripts or
 JavaScript test data.
-e.g. Call OnEndTest() to signal test completion on the JavaScript side,
-JSTestsSucceeded() will react to the signal and return the test status of
+
+Call `OnEndTest()` to signal test completion on the JavaScript side,
+`JSTestsSucceeded()` will react to the signal and return the test status of
 JavaScript test logic; another example is that when the python script wants to
-wait for some setup steps on JavaScript, call runner.WaitForJSTestsSetup().
-Calling setupFinished() in JavaScript whenever ready will unblock the wait.
+wait for some setup steps on JavaScript, call `runner.WaitForJSTestsSetup()`.
+Calling `setupFinished()` in JavaScript whenever ready will unblock the wait.
 
 
 ## Test Data
 
 A default local test server will be launched before any unit test starts to
 serve the test data in black_box_tests/testdata/. The server's port will be
 passed to the app launcher to fetch test data from.
+
 Test data can include the target web page for Cobalt to open and any additional
 resources (font file, JavaScripts...).
+
 Tests are free to start their own HTTP servers if the default test server is
 inadequate (e.g. The test is testing that Cobalt handles receipt of a specific
 HTTP server generated error code properly).
-
-
-## Adding a New Test
-
-  1. Add a python test script in tests/.
-  2. Add target web page(s) and associated resources(if any) to testdata/.
-  3. Add the test name(name of the python test script) to black_box_tests.py
-     to automate new test. Add the name to the list of tests requiring
-     app launcher support for system signals(e.g. suspend/resume), or the list
-     of tests requiring deep link support, or the list of tests that don't.

cobalt/black_box_tests/black_box_tests.py

@@ -89,6 +89,7 @@
     'service_worker_test',
     'service_worker_persist_test',
     'soft_mic_platform_service_test',
+    'telemetry_test',
     'text_encoding_test',
     'wasm_basic_test',
     'web_debugger',

cobalt/black_box_tests/testdata/telemetry_test.html

@@ -0,0 +1,71 @@
+<!DOCTYPE html>
+<!--
+  Copyright 2023 The Cobalt Authors. All Rights Reserved.
+
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+<html>
+
+<head>
+    <title>Cobalt Telemetry Test</title>
+    <script src='black_box_js_test_utils.js'></script>
+</head>
+
+<body>
+    <script>
+        const metrics = window.h5vcc.metrics;
+        const EVENT_INTERVAL_SECS = 5;
+        let lastMetricType = '';
+        let lastPayload = '';
+        let payloadCount = 0;
+
+        function metricEventHandler(metricType, payload) {
+            lastMetricType = metricType;
+            lastPayload = payload;
+            payloadCount++;
+        }
+
+        function initTelemetry() {
+            metrics.enable();
+            metrics.setMetricEventInterval(EVENT_INTERVAL_SECS);
+            metrics.onMetricEvent(metricEventHandler);
+        }
+
+        initTelemetry();
+
+        setupFinished();
+
+        setTimeout(() => {
+            assertTrue(metrics.isEnabled(), 'metrics should be enabled');
+            assertEqual('COBALT_UMA', lastMetricType, 'metricType should be uma');
+            assertTrue(lastPayload.length > 0, 'payload should be non-empty');
+            assertEqual(1, payloadCount, 'only one payload sent');
+        }, 4000);
+
+        setTimeout(() => {
+            assertEqual('COBALT_UMA', lastMetricType, 'metricType should be uma');
+            assertTrue(lastPayload.length > 0, 'payload should be non-empty');
+            assertEqual(2, payloadCount, 'two payloads sent');
+
+            metrics.disable();
+
+            setTimeout(() => {
+                assertFalse(metrics.isEnabled(), 'should disable metrics');
+                onEndTest();
+            }, 1000);
+        }, 11000);
+    </script>
+</body>
+
+</html>

cobalt/black_box_tests/tests/telemetry_test.py

@@ -0,0 +1,34 @@
+# Copyright 2023 The Cobalt Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Tests Cobalt Telemetry functionality."""
+
+from cobalt.black_box_tests import black_box_tests
+from cobalt.black_box_tests.threaded_web_server import ThreadedWebServer
+
+
+class TelemetryTest(black_box_tests.BlackBoxTestCase):
+  """Test Cobalt Telemetry functionality."""
+
+  def test_telemetry(self):
+
+    with ThreadedWebServer(binding_address=self.GetBindingAddress()) as server:
+      url = server.GetURL(file_name='testdata/telemetry_test.html')
+
+      # target_params to pass args to cobalt
+      with self.CreateCobaltRunner(
+          url=url,
+          poll_until_wait_seconds=70,
+          target_params=['--initial-metrics-upload-interval=0']) as runner:
+        runner.WaitForJSTestsSetup()
+        self.assertTrue(runner.JSTestsSucceeded())

cobalt/browser/metrics/cobalt_metrics_log_uploader.cc

@@ -57,6 +57,8 @@ void CobaltMetricsLogUploader::UploadLog(
       uma_event.ParseFromString(uncompressed_serialized_proto);
       CobaltUMAEvent cobalt_uma_event;
       PopulateCobaltUmaEvent(uma_event, reporting_info, cobalt_uma_event);
+      LOG(INFO) << "Publishing Cobalt metrics upload event. Type: "
+                << h5vcc::H5vccMetricType::kH5vccMetricTypeCobaltUma;
       // Publish the trimmed Cobalt UMA proto.
       upload_handler_->Run(h5vcc::H5vccMetricType::kH5vccMetricTypeCobaltUma,
                            cobalt_uma_event.SerializeAsString());

components/metrics/metrics_scheduler.cc

@@ -5,6 +5,12 @@
 #include "components/metrics/metrics_scheduler.h"
 
 #include "build/build_config.h"
+#if defined(STARBOARD)
+#include "base/command_line.h"
+#include "base/logging.h"
+#include "base/strings/string_number_conversions.h"
+#include "components/metrics/metrics_switches.h"
+#endif
 
 namespace metrics {
 
@@ -21,11 +27,39 @@ const int kInitialIntervalSeconds = 60;
 
 }  // namespace
 
+// In Cobalt, we need a command line argument to override the initial upload
+// interval for testing.
+#if defined(STARBOARD)
+MetricsScheduler::MetricsScheduler(const base::Closure& task_callback)
+    : task_callback_(task_callback), running_(false), callback_pending_(false) {
+  const base::CommandLine* command_line =
+      base::CommandLine::ForCurrentProcess();
+  if (command_line->HasSwitch(switches::kInitialMetricsUploadIntervalSec)) {
+    int custom_initial_upload_interval = -1;
+    if (base::StringToInt(command_line->GetSwitchValueASCII(
+                              switches::kInitialMetricsUploadIntervalSec),
+                          &custom_initial_upload_interval)) {
+      interval_ = base::TimeDelta::FromSeconds(custom_initial_upload_interval);
+      LOG(INFO) << "Initial upload interval overriden on the command line to: "
+                << custom_initial_upload_interval;
+    } else {
+      interval_ = base::TimeDelta::FromSeconds(kInitialIntervalSeconds);
+      LOG(WARNING)
+          << "Initial upload interval was set on the command line, but "
+             "converting it to an int failed. Falling back to the default: "
+          << kInitialIntervalSeconds;
+    }
+  } else {
+    interval_ = base::TimeDelta::FromSeconds(kInitialIntervalSeconds);
+  }
+}
+#else
 MetricsScheduler::MetricsScheduler(const base::Closure& task_callback)
     : task_callback_(task_callback),
       interval_(base::TimeDelta::FromSeconds(kInitialIntervalSeconds)),
       running_(false),
       callback_pending_(false) {}
+#endif
 
 MetricsScheduler::~MetricsScheduler() {}
 

components/metrics/metrics_service.cc

@@ -158,6 +158,11 @@
 #include "components/prefs/pref_registry_simple.h"
 #include "components/prefs/pref_service.h"
 #include "components/variations/entropy_provider.h"
+#if defined(STARBOARD)
+#include "base/logging.h"
+#include "base/strings/string_number_conversions.h"
+#include "components/metrics/metrics_switches.h"
+#endif
 
 namespace metrics {
 
@@ -605,6 +610,30 @@ void MetricsService::OpenNewLog() {
     // We only need to schedule that run once.
     state_ = INIT_TASK_SCHEDULED;
 
+#if defined(STARBOARD)
+    const base::CommandLine* command_line =
+        base::CommandLine::ForCurrentProcess();
+    int initialization_delay_secs = kInitializationDelaySeconds;
+    if (command_line->HasSwitch(switches::kInitialMetricsUploadIntervalSec)) {
+      if (base::StringToInt(command_line->GetSwitchValueASCII(
+                                switches::kInitialMetricsUploadIntervalSec),
+                            &initialization_delay_secs)) {
+        LOG(INFO) << "Metrics initial delay overriden to: "
+                  << initialization_delay_secs;
+      }
+    }
+    base::SequencedTaskRunnerHandle::Get()->PostDelayedTask(
+        FROM_HERE,
+        base::BindOnce(&MetricsService::StartInitTask,
+                       self_ptr_factory_.GetWeakPtr()),
+        base::TimeDelta::FromSeconds(initialization_delay_secs));
+
+    base::SequencedTaskRunnerHandle::Get()->PostDelayedTask(
+        FROM_HERE,
+        base::BindOnce(&MetricsService::PrepareProviderMetricsTask,
+                       self_ptr_factory_.GetWeakPtr()),
+        base::TimeDelta::FromSeconds(2 * initialization_delay_secs));
+#else
     base::SequencedTaskRunnerHandle::Get()->PostDelayedTask(
         FROM_HERE,
         base::BindOnce(&MetricsService::StartInitTask,
@@ -616,6 +645,7 @@ void MetricsService::OpenNewLog() {
         base::BindOnce(&MetricsService::PrepareProviderMetricsTask,
                        self_ptr_factory_.GetWeakPtr()),
         base::TimeDelta::FromSeconds(2 * kInitializationDelaySeconds));
+#endif
   }
 }
 

components/metrics/metrics_switches.cc

@@ -21,5 +21,14 @@ const char kResetVariationState[] = "reset-variation-state";
 // Forces metrics reporting to be enabled.
 const char kForceEnableMetricsReporting[] = "force-enable-metrics-reporting";
 
+#if defined(STARBOARD)
+// Override the initial time interval for sending the first startup metrics log.
+// After the initial startup, it starts to use the "standard" upload interval.
+// In Cobalt, this is configured via SetUploadInterval() calls in C++ or
+// "SetMetricEventInterval()" via an H5vccMetrics JavaScript API.
+const char kInitialMetricsUploadIntervalSec[] =
+    "initial-metrics-upload-interval";
+#endif
+
 }  // namespace switches
 }  // namespace metrics

components/metrics/metrics_switches.h

@@ -11,6 +11,9 @@ namespace switches {
 // Alphabetical list of switches specific to the metrics component. Document
 // each in the .cc file.
 
+#if defined(STARBOARD)
+extern const char kInitialMetricsUploadIntervalSec[];
+#endif
 extern const char kMetricsRecordingOnly[];
 extern const char kResetVariationState[];
 extern const char kForceEnableMetricsReporting[];