Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit maximum available number of sessions per model service #1947

Closed
kyujin-cho opened this issue Mar 4, 2024 · 0 comments · Fixed by #1948
Closed

Limit maximum available number of sessions per model service #1947

kyujin-cho opened this issue Mar 4, 2024 · 0 comments · Fixed by #1948
Milestone

Comments

@kyujin-cho
Copy link
Member

Main idea

We discovered that increasing desired_session_count to an abundant amount can cause tons of stress to session scheduler and eventually make system down. Let's update current API handler implementation to limit those factors. We should make this value customizable per each project or user.

Alternative ideas

No response

Anything else?

No response

@kyujin-cho kyujin-cho added the type:feature Add new features label Mar 4, 2024
@kyujin-cho kyujin-cho added this to the 23.09 milestone Mar 4, 2024
@achimnol achimnol removed the type:feature Add new features label Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants