Initialization benchmarks are introduced in #542

We should extend these benchmarks to measure the performance of inference.

Goal

Outlines shouldn't be a bottleneck for most inference. A reasonable goal can be set based on

• 1,200 output tok/s on H100 w/ Llama 2 70B on Optimum-Nvidia

Benchmarks will help us achieve and maintain that goal.

What must be benchmarked

• End-to-end CFG and Regex guided generation
• Guided generation with Ray IPC logits processing with vLLM (related VLLM tensor-parallel and RegexLogitsProcessor #524)

Proposed method

It's annoying to need a GPU to run tests. We shouldn't do actual inference in performance benchmarks.

• 1. Create a mock inference engine
• 2. Simple benchmark to ensure unguided mock inference engine takes infinitesimally small time
• 3. Guided benchmarks that show true throughput of outlines