Preliminary Vast AI support #4365

kristopolous · 2024-11-15T00:15:01Z

This is preliminary support for Vast. It currently works on an unreleased version of the SDK which we will soon get up to PyPy

The document https://docs.google.com/document/d/1oWox3qb3Kz3wXXSGg9ZJWwijoa99a3PIQUHBR8UgEGs/edit?pli=1&tab=t.0 was followed and all the testing passed

Tested (run the relevant ones):

Code formatting: bash format.sh
Any manual or new tests for this PR (please specify below)
All smoke tests: pytest tests/test_smoke.py
Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

I'm pretty sure there will need to be edits, I'm fine with that. This is attempt 1. The outstanding work:

We need to

tidy up our dockerhub and will get a better image to launch.
release the updates to the sdk and come up with a pip name for it.
get our catalog to update in the git hook flow as described (my goal is every 6 hours)

cblmemo

Thanks for contributing to this @kristopolous ! This is really exciting. Left some discussions. One main confusion I have is that, does vast ai like runpod, a cloud providing pods to users as their "VM"s? Asking because I'm seeing a lot of docker related code, and just want to confirm :)

sky/adaptors/vast.py

sky/clouds/vast.py

cblmemo · 2024-11-18T00:29:39Z

sky/clouds/vast.py

+        clouds.CloudImplementationFeatures.STORAGE_MOUNTING:
+            ('Mounting object stores is not supported on Vast. To read data '
+             'from object stores on Vast, use `mode: COPY` to copy the data '
+             'to local disk.'),


same, is this not supported as well? This seems only not enabled in runpod for now, which I suppose is due to its pod nature for provisioned "VM"s. Is vast ai also provide pods to the user?

That was a copy/paste. We're actually working on releasing storage by the end of the calendar year. Right now, purchasing additional storage through us is not supported. I will change this to be more reflective of the current state of things.

see 98e3db5

Just want to make sure, it sounds like you are referring to cloud storage from VastAI? IIUC this feature actually means can you mount storages from any cloud (e.g. S3, GCS) in the VastAI clusters, and it should be supported as long as some mounting utils is supported. Though if VastAI using the same container fashion provisioning, it might not work as well. cc @romilbhardwaj - what is the reason runpod not support this feature?

users are succeeding in this through our container model but we don't have any "official" support. We just announced/released VM support today and with it users will be able to do more conventional mounting of remote stores. If this is really a deal breaker I can work on making it officially supported everywhere somehow but that's a larger project.

I see. If that is the case then LGTM!

cblmemo · 2024-11-18T00:33:11Z

sky/clouds/vast.py

+             'from object stores on Vast, use `mode: COPY` to copy the data '
+             'to local disk.'),
+    }
+    _MAX_CLUSTER_NAME_LEN_LIMIT = 120


I tried to search this but does not seems yielding an answer for me. Is it possible to add some doc reference for this?

right, that was just sloppiness on my part. I'll update it. Thanks.

there was some additional cleanup required: kristopolous@3eb8823

bump for this - not seeing the reference now?

sky/clouds/vast.py

sky/provision/vast/utils.py

sky/provision/vast/instance.py

kristopolous · 2024-11-19T19:21:57Z

Thanks for contributing to this @kristopolous ! This is really exciting. Left some discussions. One main confusion I have is that, does vast ai like runpod, a cloud providing pods to users as their "VM"s? Asking because I'm seeing a lot of docker related code, and just want to confirm :)

historically, runpod was a clone of vast. We currently offer docker-style containers and will be providing vms soonish (probably before end of year)

kristopolous · 2024-11-22T00:35:23Z

these test passing is blocked by https://github.com/skypilot-org/skypilot-catalog/pull/100/commits

Co-authored-by: Tian Xia <[email protected]>

cblmemo

Thanks @kristopolous for the amazing speed! Just finished a pass and left some discussion :))

cblmemo · 2024-12-21T16:26:28Z

sky/clouds/service_catalog/data_fetchers/fetch_vast.py

@@ -0,0 +1,67 @@
+"""A script that generates the Vast Cloud catalog. """
+
+# pylint: disable=assignment-from-no-return


Why do we need this?

cblmemo · 2024-12-21T16:28:37Z

sky/clouds/service_catalog/data_fetchers/fetch_vast.py

+import re
+import sys
+
+from vastai_sdk import VastAI


Should we use adaptors? Reference

skypilot/sky/clouds/service_catalog/data_fetchers/fetch_aws.py

Line 21 in ee3cabd

from sky.adaptors import aws

cblmemo · 2024-12-21T16:29:33Z

sky/provision/vast/utils.py

+    gpu_name = instance_type.split('-')[1].replace('_', ' ')
+    num_gpus = int(instance_type.split('-')[0].replace('x', ''))


I mean this part

cblmemo · 2024-12-21T16:29:54Z

sky/clouds/service_catalog/data_fetchers/fetch_vast.py

+from vastai_sdk import VastAI
+
+
+def create_instance_type(obj):


Suggested change

def create_instance_type(obj):

def create_instance_type(obj: Dict[str, Any]) -> str:

cblmemo · 2024-12-21T16:32:11Z

sky/clouds/service_catalog/data_fetchers/fetch_vast.py

+                              obj['cpu_cores'])
+
+
+def dot_get(d, key):


Suggested change

def dot_get(d, key):

def dot_get(d: dict, key: str) -> Any:

cblmemo · 2024-12-21T16:57:01Z

sky/provision/vast/instance.py

+    head_instance_id = _get_head_instance_id(running_instances)
+    stopped_instances = status_filter(instances, ['EXITED', 'STOPPED'])
+
+    if config.resume_stopped_nodes and len(stopped_instances):


Suggested change

if config.resume_stopped_nodes and len(stopped_instances):

if config.resume_stopped_nodes and stopped_instances:

nit

cblmemo · 2024-12-21T17:03:22Z

sky/provision/vast/instance.py

+    del cluster_name_on_cloud, ports, provider_config  # Unused.
+
+
+def query_ports(


Correct me if I'm wrong, but it seems like the open_ports function is missing..? while the OPEN_PORTS is not in the cloud unimplemented features? Should we implement this?

cblmemo · 2024-12-21T17:07:00Z

sky/provision/vast/instance.py

+                instance_id=instance_id,
+                internal_ip=instance_info['local_ipaddrs'].strip(),
+                external_ip=instance_info['public_ipaddr'],
+                ssh_port=instance_info['ports']['22/tcp'][0]['HostPort'],


Should we use the insatnce['ssh_ports'] field instead of hard code a 22?

cblmemo · 2024-12-21T17:08:02Z

sky/setup_files/setup.py

pls double check this file - correct me if im wrong, but seems like some merge conflict is not handled..? why are we adding so much code

cblmemo · 2024-12-21T17:08:28Z

sky/utils/controller_utils.py

same, pls check the merge conflict for this.

Michaelvll requested a review from cblmemo November 16, 2024 02:46

cblmemo reviewed Nov 18, 2024

View reviewed changes

kristopolous force-pushed the vast.ai-support branch 3 times, most recently from e9e922a to 4c9aff9 Compare November 21, 2024 22:28

kristopolous force-pushed the vast.ai-support branch from 2b3e658 to 25b99f9 Compare November 22, 2024 21:34

kristopolous and others added 18 commits December 11, 2024 16:38

Preliminary Vast AI support

a41d633

Update sky/provision/vast/instance.py

a3e770f

Co-authored-by: Tian Xia <[email protected]>

Update sky/clouds/vast.py

d2c4dca

Co-authored-by: Tian Xia <[email protected]>

Update sky/clouds/vast.py

5deac26

Co-authored-by: Tian Xia <[email protected]>

Update sky/clouds/vast.py

372b860

Co-authored-by: Tian Xia <[email protected]>

Updating the vast dependencies in the setup.py

e5f5f3f

Vast: Copy update on object stores

15b5f5b

Vast: update base image dockerhub link

fbeaa14

Vast: removing errant comment

60abaec

Vast: provision/utils cleanup of a shallow copy

0f8a035

Vast: Simplifying the credential files mount

bb8a6eb

Vast: Linter cleanup

6107b82

Vast: Internal api cleanup

0cba961

Vast: Adding the catalog_fetcher

9287709

Vast: Linting fixes

b299ea4

Vast: Linting fixes

b8e3752

Vast: ordering the ports

1bbbbcc

Vast: Updating a function signature

7face5e

kristopolous force-pushed the vast.ai-support branch from 3010706 to 7face5e Compare December 12, 2024 00:55

kristopolous added 3 commits December 11, 2024 16:58

Vast: comment cleanup

3eb8823

Vast: Adding a comment for disk_size calculation

6a6c215

Vast: Comment on the geolocation string processing

a9d2ff4

kristopolous added 2 commits December 12, 2024 16:21

Vast: Comment on the rammifications of searching for instances

fc69f73

Vast: Leaving a comment for a pylint exception

5f9cc3b

cblmemo reviewed Dec 21, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preliminary Vast AI support #4365

Preliminary Vast AI support #4365

kristopolous commented Nov 15, 2024 •

edited

Loading

cblmemo left a comment

cblmemo Nov 18, 2024

kristopolous Nov 19, 2024

kristopolous Nov 20, 2024

cblmemo Dec 2, 2024

kristopolous Dec 11, 2024

cblmemo Dec 21, 2024

cblmemo Nov 18, 2024

kristopolous Nov 19, 2024

kristopolous Dec 12, 2024

cblmemo Dec 21, 2024

kristopolous commented Nov 19, 2024

kristopolous commented Nov 22, 2024

cblmemo left a comment

cblmemo Dec 21, 2024

cblmemo Dec 21, 2024

cblmemo Dec 21, 2024

cblmemo Dec 21, 2024

cblmemo Dec 21, 2024

cblmemo Dec 21, 2024

cblmemo Dec 21, 2024

cblmemo Dec 21, 2024

cblmemo Dec 21, 2024

cblmemo Dec 21, 2024

		@@ -0,0 +1,67 @@
		"""A script that generates the Vast Cloud catalog. """

		# pylint: disable=assignment-from-no-return

		gpu_name = instance_type.split('-')[1].replace('_', ' ')
		num_gpus = int(instance_type.split('-')[0].replace('x', ''))

	def create_instance_type(obj):
	def create_instance_type(obj: Dict[str, Any]) -> str:

	if config.resume_stopped_nodes and len(stopped_instances):
	if config.resume_stopped_nodes and stopped_instances:

		del cluster_name_on_cloud, ports, provider_config # Unused.


		def query_ports(

Preliminary Vast AI support #4365

Are you sure you want to change the base?

Preliminary Vast AI support #4365

Conversation

kristopolous commented Nov 15, 2024 • edited Loading

cblmemo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kristopolous commented Nov 19, 2024

kristopolous commented Nov 22, 2024

cblmemo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kristopolous commented Nov 15, 2024 •

edited

Loading