[Feature]: Allow running services and models without a gateway

[peterschmidt85]

Use case:

• As a user, I want to run a service/model and make it accessible to my project team.
• I don't want to set up a domain.
• I don't want to make the endpoint public.

Suggestion:

• In addition to the existing ability to create "managed" gateways (public IP address, custom domain), allow the dstack server to handle service/model traffic directly.

Motivation:

• So far, services have primarily been used for development, but the need to set up a domain and create a public endpoint poses a significant barrier. If the dstack server could run services/models out of the box, it would greatly simplify usage for both individuals and companies.

[peterschmidt85]

Another reason why this issue is important is that in our documentation, we often use tasks for deploying models because it doesn’t require creating a gateway, which is also not supported for all backends. However, when a model is running as a task, the user cannot use the UI to interact with the model.

[jvstme]

Implementation progress

• Python-based reverse proxy for services
  • HTTP proxying
  • Request/response streaming
  • Websocket proxying
  • Proxying to multiple service replicas
  • Auth
  • Configuring whether path prefix is stripped
• Running services without a gateway
• OpenAI endpoint
• [UI]: Support in-server model proxy #1954
• Configuring which gateway to use for the run
• Collecting stats for replicas autoscaling
• Using the new proxy implementation on gateways
  • Nginx- and certbot- related logic
  • Installation and updates

Usage instructions for the current version

1. Make sure your project doesn't have a default gateway or set gateway: false in your service configuration.
2. Run the service.
3. Your service is now available at <dstack-server-base-url>/proxy/services/<project-name>/<run-name>/. If the service defines a model mapping, the OpenAI-compatible API is available at <dstack-server-base-url>/proxy/models/<project-name>/

[peterschmidt85]

Plus, ensure the UI can leverage the built-in proxy

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[jvstme]

Gateways have become optional in dstack 0.18.23, so we will close this issue. The remaining technical debt has been moved to separate issues to allow for more granular prioritization:

• Update the gateway contributing guide #2252
• Drop the legacy dstack-gateway Python package #2251
• [Feature]: Configuring whether path prefix is stripped for services w/o a gateway #2250
• [Bug]: Poor connection management in in-server proxy #2238
• Send common proxy headers to services #2237
• [Feature]: RPS autoscaling for services without a gateway #2232
• [Feature]: Support websockets for services without a gateway #2231