Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create Ray Serve integration #3231

Open
szokeasaurusrex opened this issue Jul 1, 2024 · 0 comments
Open

Create Ray Serve integration #3231

szokeasaurusrex opened this issue Jul 1, 2024 · 0 comments
Labels
New Integration Integrating with a new framework or library Triaged Has been looked at recently during old issue triage

Comments

@szokeasaurusrex
Copy link
Member

szokeasaurusrex commented Jul 1, 2024

Problem Statement

From Ray Serve's docs:

Ray Serve is a scalable model serving library for building online inference APIs. Serve is framework-agnostic, so you can use a single toolkit to serve everything from deep learning models built with frameworks like PyTorch, TensorFlow, and Keras, to Scikit-Learn models, to arbitrary Python business logic. It has several features and performance optimizations for serving Large Language Models such as response streaming, dynamic request batching, multi-node/multi-GPU serving, etc.

We should create a Ray Serve integration (ideally both tracing and errors). This integration was requested during a customer call with @leokster and @smeubank (internal call notes), and it would help enhance our LLM Monitoring offering.

Solution Brainstorm

Ray integrates with OTel, so we might want to wait until #2251 is implemented so we can use OTel for this integration. However, the Ray's OTel integration is no longer actively maintained, so we might need to fork it and maintain it ourselves.

Related: #2400 (Ray Remote integration)

@szokeasaurusrex szokeasaurusrex added Triaged Has been looked at recently during old issue triage Enhancement labels Nov 4, 2024
@sentrivana sentrivana added the New Integration Integrating with a new framework or library label Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
New Integration Integrating with a new framework or library Triaged Has been looked at recently during old issue triage
Projects
None yet
Development

No branches or pull requests

3 participants