README.md

Benchmarking with Optimum-Benchmark

Pre-requites

Enable Docker as a non-root user (mandatory for Lambda Labs instances).

sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

Installation

Install optimum-benchmark

Set up HF_TOKEN environement variable.

export HF_TOKEN=...

Usage

Benchmarking with PyTorch backend

optimum-benchmark --config-dir configs --config-name pytorch

Benchmarking with TGI backend

optimum-benchmark --config-dir configs --config-name tgi

Then manually stop TGI.