Compatibility with vLLM with tensor_parallel_size argument

[gabrielmbmb]

Description

This PR adds a few changes that enables using vLLM with the argument tensor_parallel_size. This argument enables the use of multiple GPUs with vLLM, and to do so, vLLM uses multiprocessing or ray:

1. When using the multiprocessing approach, it was not working as vLLM was trying to create new processes but it was not able to because the process that distilabel creates was a daemon process, which is not allowed to create child processes. To by pass these issue, a _NoDaemonPool class has been created that creates non-daemon processes and it's used in Pipeline.
2. With ray it works installing the version in the main branch which includes the changes of [Core] Introduce SPMD worker execution using Ray accelerated DAG vllm-project/vllm#6032. It's needed to set VLLM_USE_RAY_COMPILED_DAG=1 and VLLM_USE_RAY_SPMD_WORKER=1 environment variables. Update: vllm==0.5.3 has been released which includes the changes needed.

[github-actions]

Documentation for this PR has been built. You can view it at: https://distilabel.argilla.io/pr-805/

[codspeed-hq]

CodSpeed Performance Report

Merging #805 will not alter performance

_{Comparing compatibility-vllm-tensor-parallel-size (c0ece53) with develop (ea1c44b)}

Summary

✅ 1 untouched benchmarks