[Feature] metrics support

[CUHKSZzxy]

Objective

Align with vLLM v1 metrics system and beyond. Here are several key alignments

• Monotonic Timestamps:
  -- Uses time.perf_counter() for interval calculations (avoids clock drift issues).
• Metric Types:
  -- Gauges: Active requests, cache usage, etc
  -- Counters: Token totals, request success / failure counts, etc
  -- Histograms: TTFT (Time-To-First-Token), TPOT (Inter-Token Latency), end-to-end latency, etc
• Metrics Publishing:
  -- CLI logging
  -- Prometheus

We only record critical timestamps and events during the main loop and scheduling without further processing. Heavy-weight metrics calculations or metrics publishing are put inside separate coroutines to reduce the main engine loop overhead.

TODO

• Use time.perf_counter()
• Refactor to minimize the overhead of async engine generate() or engine _async_loop_main()
• Expert information collections (maybe deferred in another PR)
• Grafana visualization (WIP, maybe deferred in another PR)

Usage

Start the server with --enable-metrics

lmdeploy serve api_server models--Qwen--Qwen2.5-7B-Instruct --enable-metrics

• Metrics Publishing - Logging
  With --enable-metrics, key metrics (e.g., running / waiting requests, cache usage, token throughput) are printed to the terminal every 5 seconds.
  

• Metrics Publishing - Prometheus & Grafana (WIP)
  Access metrics via http://localhost:23333/metrics/ , or curl the metrics endpoint as follows:

curl http:///localhost:23333/metrics/

Performance Impacts

Conclusion:

• No obvious throughput degradation for Qwen-2.5-32B, minor degradation (1~2%) for Qwen-2.5-7B, and notable degradation (15% ~ 20%) for small models like Qwen-2.5-0.5B.

For details, you may check the following figures. Benchmark settings: 1000 prompts, input len 1000, output len 1000.

• Qwen-2.5-7B (TP1), without the metrics.
  

• Qwen-2.5-7B (TP1), with the metrics.
  

• Qwen-2.5-0.5B (TP1), without the metrics.
  

• Qwen-2.5-0.5B (TP1), with the metrics.
  

Related Issues & PR

Issue 2638, Issue 2673, PR1423