Skip to content

Latest commit

 

History

History
115 lines (88 loc) · 4.73 KB

README.md

File metadata and controls

115 lines (88 loc) · 4.73 KB
Raw

OPEA Benchmark Tool

This Tool provides a microservice benchmarking framework that uses YAML configurations to define test cases for different services. It executes these tests using stresscli, built on top of locust, a performance/load testing tool for HTTP and other protocols and logs the results for performance analysis and data visualization.

Features

  • Services load testing: Simulates high concurrency levels to test services like LLM, reranking, ASR, E2E and more.
  • YAML-based configuration: Define test cases, service endpoints, and testing parameters in YAML.
  • Service metrics collection: Optionally collect service metrics for detailed performance analysis.
  • Flexible testing: Supports various test cases like chatqna, codegen, codetrans, faqgen, audioqna, and visualqna.
  • Data analysis and visualization: After tests are executed, results can be analyzed and visualized to gain insights into the performance and behavior of each service. Performance trends, bottlenecks, and other key metrics are highlighted for decision-making.

Table of Contents

Installation

Prerequisites

  • Python 3.x
  • Install the required Python packages:
pip install -r ../../requirements.txt

Usage

1 Define the test cases and configurations in the benchmark.yaml file.

2 Temporarily increase the file descriptor limit before run test:

ulimit -n 100000

This command increases the maximum number of file descriptors (which represent open files, network connections, etc.) that a single process can use. By default, many systems set a conservative limit, such as 1024, which may not be sufficient for high-concurrency applications or large-scale load testing. Raising this limit ensures that the process can handle a larger number of open connections or files without running into errors caused by insufficient file descriptors.

3 Run the benchmark script:

python benchmark.py

The results will be stored in the directory specified by test_output_dir in the configuration.

Configuration

The benchmark.yaml file defines the test suite and individual test cases. Below are the primary sections:

Test Suite Configuration

test_suite_config: 
  examples: ["chatqna"]  # Test cases to be run (e.g., chatqna, codegen)
  deployment_type: "k8s"  # Default is "k8s", can also be "docker"
  service_ip: None  # Leave as None for k8s, specify for Docker
  service_port: None  # Leave as None for k8s, specify for Docker
  load_shape:              # Tenant concurrency pattern
    name: constant           # poisson or constant(locust default load shape)
    params:                  # Loadshape-specific parameters
      constant:                # Poisson load shape specific parameters, activate only if load_shape is poisson
        concurrent_level: 4      # If user_queries is specified, concurrent_level is target number of requests per user. If not, it is the number of simulated users
      poisson:                 # Poisson load shape specific parameters, activate only if load_shape is poisson
        arrival-rate: 1.0        # Request arrival rate
  warm_ups: 0  # Number of test requests for warm-ups
  run_time: 60m  # Total runtime for the test suite
  user_queries: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048]  # Number of test requests
  query_timeout: 120  # Number of seconds to wait for a simulated user to complete any executing task before exiting. 120 sec by defeult.
  random_prompt: false  # Use random prompts if true, fixed prompts if false
  collect_service_metric: false  # Enable service metrics collection
  data_visualization: false # Enable data visualization
  test_output_dir: "/home/sdp/benchmark_output"  # Directory for test outputs

Test Cases

Each test case includes multiple services, each of which can be toggled on/off using the run_test flag. You can also change specific parameters for each service for performance tuning.

Example test case configuration for chatqna:

test_cases:
  chatqna:
    embedding:
      run_test: false
      service_name: "embedding-svc"
    retriever:
      run_test: false
      service_name: "retriever-svc"
      parameters:
        search_type: "similarity"
        k: 4
        fetch_k: 20
        lambda_mult: 0.5
        score_threshold: 0.2
    llm:
      run_test: false
      service_name: "llm-svc"
      parameters:
        model_name: "Intel/neural-chat-7b-v3-3"
        max_new_tokens: 128
        temperature: 0.01
        streaming: true
    e2e:
      run_test: true
      service_name: "chatqna-backend-server-svc"