I want to show a demo about using ray in preprocess for asynchronous processing

[mc112611]

colab_demo

I found ray really useful!

Just a @ ray. remote() modifier class is needed to enable asynchronous processing.

So, I added ray to PreProcess.

You need to pay attention to the last two code blocks, one is the original execution time, and the other is the execution time using ray.

Using the official tutorial：Tutorial: Preprocessing Your Documents

Can ray join Haystack in the future?

Because ray is also compatible with the Transformer library and Pytorch. I will try adding ray to Haystack. I think that should greatly increase the execution speed of Haystack

Ray is really useful, I want to say it again

[anakin87]

Hey @mc112611...

thanks for your example!

Undoubtedly, Ray is a nice family of projects and can be integrated into Haystack in several ways.
Some examples:

• Distributed Pipelines with Ray
• Run Haystack Pipelines in production with Ray Serve