LocalStack benchmarking with locust

locust is a "modern load testing framework". We can use this to benchmark LocalStack performance by load testing.

Getting started

Start LocalStack with your performance improvements
Clone this repository and install the development dependencies python -m pip install .[dev]

Start the benchmarking web interface: locust [--tags <tag>...] and visit its URL (default: 127.0.0.1:8089)
Choose the load testing parameters (e.g. 100 concurrent users at 1 additional user per second)
Click "Start" and your load testing session will start

When you are happy with your benchmark, visit the reports page and download the report you are interested in.

Run the benchmark in headless mode with locust --headless --users <concurrent users> --spawn-rate <spawn rate> -H http://127.0.0.1:4566 [--run-time <run time>] [--tags <tag>...] [--html <report-file>] [--csv <report-file-prefix>]
Either add --run-time <run time> to your command line, or Ctrl-C the benchmark to write the report files

This repository contains a script that shows a comparison chart for the average

for each csv file passed in on the command line.

For example:

python ./compare_runs.py baseline_stats.csv new_feature.csv

This opens an interactive plot in your web browser to understand the results.