Running benchmarks

[juanpicado]

Since few months go I've been running benchmarks to compare performance improvements over the time and compare with previous releases.

Where it runs

It runs benchmarks for specific common steps like: npm info, fetch tarball for all major versions. The test runs every day.

https://github.com/verdaccio/verdaccio/blob/master/.github/workflows/benckmark.yml

How it works

We run hyperfine and autocannon on every test and version, we collect that information and feed an API that at the same time feed a database.

hyperfine

The script run several runs by using npm running info and install.

{
  "results": [
    {
      "command": "npm info jquery",
      "mean": 0.9477428863450001,
      "stddev": 0.049996844308338936,
      "median": 0.9397961800450001,
      "user": 0.4920053449999999,
      "system": 0.051991659999999995,
      "min": 0.8972533635450001,
      "max": 1.0718330145450001,
      "times": [
        1.0718330145450001,
         .....
        0.9606509365450001
      ]
    }
  ]
}

autocannon

The script hits an specific endpoint instead running directly a package manager, the outcome is more complete, here an example:

{
  "results": [
    {
      "url": "http://localhost:4873/jquery/-/jquery-3.6.0.tgz",
      "connections": 10,
      "pipelining": 1,
      "duration": 10.53,
      "start": "2021-09-08T01:31:24.827Z",
      ....
      "statusCodeStats": {
        "200": {
          "count": 838
        }
      },
      "latency": {
      },
      "requests": {
      },
      "throughput": {
      }
    }
  ]
}

Data

The current data is going to a DynamoDB (free tier 😊, we have a tight budget) where is stored until could be processed, in the next steps.

Next steps

What's missing by far is process that data and great a nice dashboard that represent the changes over the time.

If you want to contribute on bench-marking verdaccio or have any idea that could improve this process, feel free to drop your thoughts here.

The raw data for each run can be find here