remove csv file write (GoogleCloudPlatform#484)

remove csv file write. custom_messages has bug that causes locust worker restarts

benchmarks/benchmark/tools/locust-load-inference/README.md

@@ -10,14 +10,14 @@
     - [Step 4: create and give service account access to write to output gcs bucket](#step-4-create-and-give-service-account-access-to-write-to-output-gcs-bucket)
     - [Step 5: create artifact repository for automated Locust docker build](#step-5-create-artifact-repository-for-automated-locust-docker-build)
     - [Step 6: create and configure terraform.tfvars](#step-6-create-and-configure-terraformtfvars)
-      - [optional: set-up credentials config with kubeconfig](#optional-set-up-credentials-config-with-kubeconfig)
-      - [optional: set up secret token in secret manager](#optional-set-up-secret-token-in-secret-manager)
+      - [\[optional\] set-up credentials config with kubeconfig](#optional-set-up-credentials-config-with-kubeconfig)
+      - [\[optional\] set up secret token in Secret Manager](#optional-set-up-secret-token-in-secret-manager)
     - [Step 7: login to gcloud](#step-7-login-to-gcloud)
     - [Step 8: terraform initialize, plan and apply](#step-8-terraform-initialize-plan-and-apply)
     - [Step 9: start an end to end benchmark](#step-9-start-an-end-to-end-benchmark)
       - [option 1: initiate a single end to end Locust benchmark run via curl command](#option-1-initiate-a-single-end-to-end-locust-benchmark-run-via-curl-command)
       - [option 2: interactive benchmark with locust web ui](#option-2-interactive-benchmark-with-locust-web-ui)
-      - [writing custom metrics](#writing-custom-metrics)
+    - [Step 10: viewing metrics](#step-10-viewing-metrics)
     - [Additional Tips](#additional-tips)
   - [Variables](#variables)
 <!-- END TOC -->
@@ -222,17 +222,15 @@ In a web browser, visit the following website:
 ```
 http://$LOCUST_SERVICE_IP:8089
 ```
-#### writing custom metrics
 
-If the variable `enable_custom_metrics` is set to `true` then  custom metrics collected by the locust master is available at the following endpoints:
-* While the test is running
-```
-http://$LOCUST_SERVICE_IP:8089/custom_metrics/custom_metrics.csv
-```
-* After a test ends:
-```
-http://$LOCUST_SERVICE_IP:8089/custom_metrics/custom_metrics_final.csv
-```
+### Step 10: viewing metrics
+
+Locust and custom inferencing metrics calculated by locust are exported to cloud monitoring. Metrics are stored in the prometheus namespace, eg:
+
+resource.type="prometheus_target"
+metric.type="prometheus.googleapis.com/locust_*"
+
+You can use the cloud monitoring dashboard to view charts of these metrics.
 
 ### Additional Tips
 
@@ -262,6 +260,5 @@ To change the benchmark configuration, you will have to rerun terraform destroy
 | <a name="input_sax_model"></a> [sax\_model](#input\_sax\_model)                                                      | Benchmark server configuration for sax model. Only required if framework is sax.                                            | `string`                                                                                                                                                                                                    | `""`                 |    no    |
 | <a name="input_tokenizer"></a> [tokenizer](#input\_tokenizer)                                                        | Benchmark server configuration for tokenizer.                                                                               | `string`                                                                                                                                                                                                    | `"tiiuae/falcon-7b"` |   yes    |
 | <a name="input_use_beam_search"></a> [use\_beam\_search](#input\_use\_beam\_search)                                  | Benchmark server configuration for use beam search.                                                                         | `bool`                                                                                                                                                                                                      | `false`              |    no    |
- <a name="enable_custom_metrics"></a> [enable\_custom\_metric](#input\_enable\_custom\_metrics)                                  | To collect custom metrics like number of tokens sent and received                                                                          | `bool`                                                                                                                                                                                                      | `false`              |    no    |
   <a name="huggingface_secret"></a> [huggingface_secret](#input\_huggingface_secret)                                  | Name of the kubectl huggingface secret token                                                                          | `string`                                                                                                                                                                                                      | `huggingface-secret`              |    no   |
 <!-- END_TF_DOCS -->

benchmarks/benchmark/tools/locust-load-inference/locust-docker/locust-tasks/custom_metric_aggregator.py

@@ -56,17 +56,3 @@ def json_dump_report(self):
         }
         return json.dumps(stats)
 
-    def write_to_csv(self, file_path='custom_metrics.csv'):
-        import csv
-        avg_sent, avg_received, avg_test_time, avg_output_token_latency = self.calculate_average_tokens()
-        with open(file_path, mode='w', newline='') as file:
-            writer = csv.writer(file)
-            writer.writerow(['Metric', 'Average Value'])
-            writer.writerow(['# of Successful Req', self.success_count])
-            writer.writerow(['# of Failed Req', self.failure_count])
-            writer.writerow(['Avg Tokens Sent Per Req', avg_sent])
-            writer.writerow(['Avg Tokens Received Per Req', avg_received])
-            writer.writerow(['Avg Test Time', avg_test_time])
-            writer.writerow(['Avg Output Tokens Latency',
-                            avg_output_token_latency])
-            writer.writerow(['Timestamp', datetime.datetime.now()])

benchmarks/benchmark/tools/locust-load-inference/locust-docker/locust-tasks/tasks.py

@@ -21,15 +21,13 @@
 import threading
 import time
 from locust import web  # Import the web module from Locust
-from flask import send_from_directory
 from typing import Callable, List
 from locust import FastHttpUser, task, events
 from locust.runners import MasterRunner
 from transformers import AutoTokenizer, PreTrainedTokenizerBase
 
 
 from custom_metric_aggregator import TokenMetricCollector
-metric_collector = TokenMetricCollector()
 local_metric_collector = TokenMetricCollector()
 
 logging.basicConfig(level=logging.INFO)
@@ -164,90 +162,42 @@ def lm_generate(self):
 
     def handle_successful_response(self, prompt, reponse, start_time):
         global model_params
-        if model_params['enable_custom_metrics'] == 'true':
-            test_time = time.time() - start_time
-            request_successful_bool = 1
-            tokens_sent, tokens_received = get_token_count(prompt, reponse)
-
-            local_metric_collector.add_metric(
-                tokens_sent, tokens_received, test_time, request_successful_bool)
-            logging.info(
-                f'sending to master: metric_update: {[tokens_sent, tokens_received, test_time, request_successful_bool]}')
-            self.environment.runner.send_message("metric_update", [
-                                                 tokens_sent, tokens_received, test_time, request_successful_bool])
+        test_time = time.time() - start_time
+        request_successful_bool = 1
+        tokens_sent, tokens_received = get_token_count(prompt, reponse)
+
+        local_metric_collector.add_metric(
+            tokens_sent, tokens_received, test_time, request_successful_bool)
+        logging.info(
+            f'sending to master: metric_update: {[tokens_sent, tokens_received, test_time, request_successful_bool]}')
 
     def handle_failed_response(self, request, response):
         global model_params
         response.failure("Got unexpected response")
         logging.error(f"request {request} failed with: {response.status_code}")
-        if model_params['enable_custom_metrics'] == 'true':
-            tokens_sent = -1
-            tokens_received = -1
-            test_time = -1
-            request_successful_bool = 0
-
-            local_metric_collector.add_metric(
-                tokens_sent, tokens_received, test_time, request_successful_bool)
-            logging.info(
-                f'sending to master: metric_update: {[tokens_sent, tokens_received, test_time, request_successful_bool]}')
-            self.environment.runner.send_message("metric_update", [
-                                                 tokens_sent, tokens_received, test_time, request_successful_bool])
-
-
-"""
-methods for the locust master to write custom metrics
-"""
-
-
-def collect_metrics(msg, **_kwargs):
-    """locust master collects the metrics emitted by the locust workers and updates the metric_collector object"""
-    sent = msg.data[0]
-    received = msg.data[1]
-    test_time = msg.data[2]
-    request_successful_bool = msg.data[3]
-    logging.info(f'received from worker {msg.data}')
-    metric_collector.add_metric(
-        sent, received, test_time, request_successful_bool)
-
+        tokens_sent = -1
+        tokens_received = -1
+        test_time = -1
+        request_successful_bool = 0
 
-def periodically_write_metrics(environment):
-    metric_collector.write_to_csv()
-    threading.Timer(environment.parsed_options.csv_upload_frequency,
-                    periodically_write_metrics, args=(environment,)).start()
-
-
-def setup_periodic_metrics_writer(environment,  **_kwargs):
-    """locust master periodically writes the collected metrics to csv"""
-    periodically_write_metrics(environment)
-
-
-def setup_custom_route(environment, **_kwargs):
-    """Sets up custom routes in the locust master for serving CSV files."""
-    directory = os.path.dirname('/')  # Directory where the file is located
-
-    @environment.web_ui.app.route("/custom_metrics/<filename>")
-    def custom_metrics(filename):
-        if filename not in ['custom_metrics.csv', 'custom_metrics_final.csv']:
-            return "File not found.", 404  # Basic validation to prevent unauthorized file access
-        return send_from_directory(directory, filename, as_attachment=True)
+        local_metric_collector.add_metric(
+            tokens_sent, tokens_received, test_time, request_successful_bool)
+        logging.info(
+            f'sending to master: metric_update: {[tokens_sent, tokens_received, test_time, request_successful_bool]}')
 
 
 @events.test_stop.add_listener
 def on_test_stop(environment, **kwargs):
-    """on test stop the locust master writes the output to custom_metrics_final and resets the metric_collector for next tests"""
-    if isinstance(environment.runner, MasterRunner) and environment.parsed_options.enable_custom_metrics == 'true':
+    """on test stop the locust master resets metric collector"""
+    if isinstance(environment.runner, MasterRunner):
         logging.info(f'init metric_collector')
-        metric_collector.write_to_csv('custom_metrics_final.csv')
-        metric_collector.__init__()
-        metric_collector.write_to_csv()
         local_metric_collector.__init__()
 
 
 """
-Methods for collecting request latencies to share to master webui
+Methods for collecting custom metrics to share to master webui
 """
 
-
 @events.report_to_master.add_listener
 def on_report_to_master(client_id, data):
     """
@@ -290,18 +240,12 @@ def _(parser):
                         include_in_web_ui=True, help="Whether to use beam search instead of sampling.")
     parser.add_argument("--tokenizer", type=str, env_var="TOKENIZER",
                         include_in_web_ui=False, default="", help="Tokenizer to use for token calculations")
-    parser.add_argument("--enable_custom_metrics", type=str, env_var="ENABLE_CUSTOM_METRICS",
-                        include_in_web_ui=True, default="false", help="enable custom metric")
-    parser.add_argument("--csv_upload_frequency", type=int, env_var="CSV_UPLOAD_FREQUENCY",
-                        include_in_web_ui=True, default=10, help="upload custom metrics every X seconds")
-
 
 @events.init.add_listener
 def _(environment, **kwargs):
     if not isinstance(environment.runner, MasterRunner):
         global model_params
         global test_data
-        global metric_collector
         global local_metric_collector
         global tokenizer
 
@@ -324,18 +268,10 @@ def _(environment, **kwargs):
             "sax_model": environment.parsed_options.sax_model,
             "use_beam_search": environment.parsed_options.use_beam_search,
             "tokenizer": environment.parsed_options.tokenizer,
-            "enable_custom_metrics": environment.parsed_options.enable_custom_metrics,
-            "csv_upload_frequency": environment.parsed_options.csv_upload_frequency,
         }
         logging.info(
             f"Using the following benchmark parameters:\n {model_params}")
 
-    elif environment.parsed_options.enable_custom_metrics == 'true':
-        # code to setup the locust master to write custom metrics
-        setup_periodic_metrics_writer(environment)
-        setup_custom_route(environment)
-        environment.runner.register_message("metric_update", collect_metrics)
-
 
 @events.init.add_listener
 def locust_init(environment, **kwargs):

benchmarks/benchmark/tools/locust-load-inference/main.tf

@@ -41,9 +41,7 @@ locals {
       sax_model                  = var.sax_model
       tokenizer                  = var.tokenizer
       use_beam_search            = var.use_beam_search
-      enable_custom_metrics      = var.enable_custom_metrics
       huggingface_secret         = var.huggingface_secret
-      csv_upload_frequency       = var.csv_upload_frequency
     })) : data]
   ])
 }

benchmarks/benchmark/tools/locust-load-inference/manifest-templates/locust-master-controller.yaml.tpl

@@ -26,8 +26,6 @@ spec:
               value: http://${inference_server_service}
             - name: BACKEND
               value: ${inference_server_framework}
-            - name: ENABLE_CUSTOM_METRICS
-              value: ${enable_custom_metrics}
           ports:
             - name: loc-master-web
               containerPort: 8089

benchmarks/benchmark/tools/locust-load-inference/manifest-templates/locust-worker-controller.yaml.tpl

@@ -44,10 +44,6 @@ spec:
               value: ${tokenizer}
             - name: USE_BEAM_SEARCH
               value: ${use_beam_search}
-            - name: ENABLE_CUSTOM_METRICS
-              value: ${enable_custom_metrics}
-            - name: CSV_UPLOAD_FREQUENCY
-              value: ${csv_upload_frequency}
             - name: HUGGINGFACE_TOKEN
               valueFrom:
                 secretKeyRef:

benchmarks/benchmark/tools/locust-load-inference/variables.tf

@@ -191,22 +191,9 @@ variable "run_test_automatically" {
   default     = false
 }
 
-variable "enable_custom_metrics" {
-  description = "enable custom metric output in Locust"
-  type        = bool
-  default     = false
-}
-
 variable "huggingface_secret" {
   description = "name of the kubectl huggingface secret token"
   type        = string
   nullable    = true
   default     = "huggingface-secret"
 }
-
-variable "csv_upload_frequency" {
-  description = "how frequently, in seconds, to upload csv if custom metrics is turned on"
-  type        = number
-  nullable    = true
-  default     = 10
-}