[perf] use uv for venv creation and pip install #4414

cg505 · 2024-11-25T23:23:02Z

This was previously merged in as #4394, which was reverted in #4401 due to a bug.
This PR:

readds the old uv changes
fixes the bug on Azure (see below)
uses the --seed flag with uv venv to make sure pip is installed in the venv

Performance impact:

Cluster provisioning:
- Using skypilot custom images:
  - AWS: saves about 0.5s (8.5s -> 8s)
  - Azure: saves about 5s (26s -> 21s)
- Using stock ubuntu images ~3x speedup:
  - AWS: saves about 30s (45s -> 15s)
  - Azure: saves about 100s (130s -> 30s)
  - These results likely also apply to other user-provided images, including custom kubernetes images.
Job controller setup/cloud dependency installation saves about 45s (85s -> 25s) on first run
- with GCP, AWS, and Azure enabled
- on run 2+ with the controller already setup, saves about 3s (6.5s -> 3.5s)

Azure bug:
uv pip install skypilot[azure] would fail to install azure-cli and complain as follows:

  × No solution found when resolving dependencies:
  ╰─▶ Because there is no version of azure-keyvault-administration==4.4.0b2
      and azure-cli>=2.65.0 depends on azure-keyvault-administration==4.4.0b2,
      we can conclude that azure-cli>=2.65.0 cannot be used.
      And because only the following versions of azure-cli are available:
          azure-cli<=2.65.0
          azure-cli==2.66.0
          azure-cli==2.67.0
      and skypilot[azure]==1.0.0.dev0 depends on azure-cli>=2.65.0, we can
      conclude that skypilot[azure]==1.0.0.dev0 cannot be used.
      And because only skypilot[azure]==1.0.0.dev0 is available and you
      require skypilot[azure], we can conclude that your requirements are
      unsatisfiable.

      hint: azure-keyvault-administration was requested with a pre-release
      marker (e.g., azure-keyvault-administration==4.4.0b2), but pre-releases
      weren't enabled (try: `--prerelease=allow`)

This is because azure-cli directly depends on a pre-release version of azure-keyvault-administration (and several other packages). Normal pip handles this by allowing the pre-release version since it's specified as an explicit version requirement, however uv by default does not consider pre-releases at all. See https://docs.astral.sh/uv/pip/compatibility/#pre-release-compatibility.
We don't want to use uv pip install --prerelease=allow skypilot[azure] because this will allow pre-release versions to satisfy any dependency, not just these explicitly specified ones. I tested this and the resolved version was different for a few packages. This seems like an unacceptable difference, especially since we will be unnecessarily relying on pre-release packages.
Once azure-cli is installed, uv pip install skypilot[azure] will work just fine, since it doesn't need to worry about resolving azure-cli's dependencies. So, we just need to find a way to install azure-cli. pip install azure-cli works, but it's VERY slow (takes >60s on my machine). uv pip install --prerelease=allow azure-cli is much faster, so I went with this.
Technically, uv pip install --prerelease=allow azure-cli uses different resolution than pip install azure-cli, for the same reason described above for installing skypilot[azure]. However, I tested this, and in practice only one package has a different version. (azure-mgmt-search 9.1.0 -> 9.2.0b2)

Tested (run the relevant ones):

Code formatting: bash format.sh
Any manual or new tests for this PR (please specify below)
All smoke tests: pytest tests/test_smoke.py
Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh
- k8s
- aws
- azure
  - technically I can't get this to pass - everything passes except step 7 which fails because it takes 50 minutes to schedule the job lol

skypilot-org#4401)" This reverts commit 0b20d56.

cg505 · 2024-11-25T23:59:06Z

@romilbhardwaj Rather than conditionally using pip/uv based on what's installed, I just changed the RAY_INSTALLATION_COMMANDS to also install uv if it's missing. I think this is better because we need different behavior between pip and uv, so trying to make things compatible with both is going to get increasingly difficult. As a brief example, pip uninstall needs the -y flag to work non-interactively, but this flag does not exist in uv pip uninstall.

Michaelvll

Thanks @cg505! It looks mostly good to me. We should run the backward compatibility tests to make sure it works, especially on AWS, GCP, Azure and Kubernetes

Michaelvll · 2024-11-26T00:51:03Z

sky/skylet/constants.py

+    'if [ "{cloud}" = "azure" ]; then '
+    f'{SKY_UV_PIP_CMD} install --prerelease=allow "azure-cli>=2.65.0"; fi;'


We should search for all occurence of azure-cli across the project and pip installs. Are we planning to install the dependencies with uv for controller dependencies as well?

skypilot/sky/utils/controller_utils.py

Line 232 in 1b3c277

'pip install "azure-cli>=2.31.0" azure-core '

Oh wow, I didn't know about this code. We should definitely fix this up. To be clear, everything should work as-is but ideally we would move this all to uv as well.

There are no other relevant mentions of azure-cli in the repo.

Michaelvll

Thanks @cg505!

Michaelvll · 2024-11-27T00:52:35Z

sky/skylet/constants.py

@@ -39,6 +39,7 @@
                           'which python3')
 # Python executable, e.g., /opt/conda/bin/python3
 SKY_PYTHON_CMD = f'$({SKY_GET_PYTHON_PATH_CMD})'
+# Prefer SKY_UV_PIP_CMD, which is faster. TODO(cooper): remove all usages.


Mention the reason in the comment for why we keep this for future reference.

Michaelvll · 2024-11-27T17:44:28Z

sky/setup_files/setup.py

+sys.path.append(SETUP_FILE_DIR)
+import dependencies


dependencies seems not included in this PR?

Michaelvll · 2024-11-27T17:47:02Z

sky/utils/controller_utils.py

@@ -25,6 +25,7 @@
 from sky.jobs import utils as managed_job_utils


This is awesome!

cg505 · 2024-11-27T19:11:05Z

Ran backwards compatibility tests as shown in PR description.

Michaelvll

Thanks for adding this @cg505! We may want to run some tests in a clean env: tests for normal launches and jobs/serve with new controller set up by this PR.

Michaelvll · 2024-11-27T19:26:57Z

sky/setup_files/setup.py

+# setuptools does not include the script dir on the search path, so manually add
+# it so that we can import the dependencies file.
+sys.path.append(SETUP_FILE_DIR)
+import dependencies  # pylint: disable=wrong-import-position


Is this a common practice?

Would it cause any issue with our wheel building during sky launch? Would be great if we can test it in a clean env, e.g. start a new container for running some tests.

I tested the wheel build command manually as well as sky launch to make sure this would work. I looked for information on whether this is common practice but couldn't find any useful info.

I found this: pypa/setuptools#3134
Obviously it's not ideal to hack sys.path. We could use importlib as suggested there. What do you think?

python build systems are so complicated...

Updated this to use runpy.run_path, which seems a lot cleaner.

Michaelvll · 2024-11-27T19:29:09Z

sky/utils/controller_utils.py

+                    f'uv -V > /dev/null 2>&1 ||'
+                    'curl -LsSf https://astral.sh/uv/install.sh 2>/dev/null |'
+                    'UV_INSTALL_DIR="$HOME/.local/bin" sh >/dev/null 2>&1')


Just curious, should we just use uv setup in our provision setup, i.e. use {constants.UV_PIP_CMD} directly?

Also, we use {{ sky_activate_python_env }} in jobs controller yml to make sure pip install goes to the separate env we created. Will the new changes cause any issue?

We probably could use SKY_UV_PIP_CMD - I didn't because the existing code wasn't using SKY_PIP_CMD.
Activating the python env normally should work correctly with uv.

Updated to use the commands from skylet.constants.

Michaelvll

Thanks @cg505! It looks mostly good to me with a small comment below

sky/utils/controller_utils.py

Michaelvll

Thanks @cg505! LGTM.

Can we have a quick number for the performance improvement in the PR description, for future reference?

sky/utils/controller_utils.py

Co-authored-by: Zhanghao Wu <[email protected]>

cg505 · 2024-12-03T23:46:21Z

Can we have a quick number for the performance improvement in the PR description, for future reference?

@Michaelvll updated PR description!

* [perf] use uv for venv creation and pip install (#4414) * Revert "remove `uv` from runtime setup due to azure installation issue (#4401)" This reverts commit 0b20d56. * on azure, use --prerelease=allow to install azure-cli * use uv venv --seed * fix backwards compatibility * really fix backwards compatibility * use uv to set up controller dependencies * fix python 3.8 * lint * add missing file * update comment * split out azure-cli dep * fix lint for dependencies * use runpy.run_path rather than modifying sys.path * fix cloud dependency installation commands * lint * Update sky/utils/controller_utils.py Co-authored-by: Zhanghao Wu <[email protected]> --------- Co-authored-by: Zhanghao Wu <[email protected]> * [Minor] README updates. (#4436) * [Minor] README touches. * update * update * make --fast robust against credential or wheel updates (#4289) * add config_dict['config_hash'] output to write_cluster_config * fix docstring for write_cluster_config This used to be true, but since #2943, 'ray' is the only provisioner. Add other keys that are now present instead. * when using --fast, check if config_hash matches, and if not, provision * mock hashing method in unit test This is needed since some files in the fake file mounts don't actually exist, like the wheel path. * check config hash within provision with lock held * address other PR review comments * rename to skip_if_no_cluster_updates Co-authored-by: Zhanghao Wu <[email protected]> * add assert details Co-authored-by: Zhanghao Wu <[email protected]> * address PR comments and update docstrings * fix test * update docstrings Co-authored-by: Zhanghao Wu <[email protected]> * address PR comments * fix lint and tests * Update sky/backends/cloud_vm_ray_backend.py Co-authored-by: Zhanghao Wu <[email protected]> * refactor skip_if_no_cluster_update var * clarify comment * format exception --------- Co-authored-by: Zhanghao Wu <[email protected]> * format * format * format * fix --------- Co-authored-by: Christopher Cooper <[email protected]> Co-authored-by: Zongheng Yang <[email protected]>

* [perf] use uv for venv creation and pip install (#4414) * Revert "remove `uv` from runtime setup due to azure installation issue (#4401)" This reverts commit 0b20d56. * on azure, use --prerelease=allow to install azure-cli * use uv venv --seed * fix backwards compatibility * really fix backwards compatibility * use uv to set up controller dependencies * fix python 3.8 * lint * add missing file * update comment * split out azure-cli dep * fix lint for dependencies * use runpy.run_path rather than modifying sys.path * fix cloud dependency installation commands * lint * Update sky/utils/controller_utils.py Co-authored-by: Zhanghao Wu <[email protected]> --------- Co-authored-by: Zhanghao Wu <[email protected]> * [Minor] README updates. (#4436) * [Minor] README touches. * update * update * basic impl * fixes * start dashboard when restarting the controller * Fix dashboard support * Fix * format * format * format * format * format * address comments * format * fix --------- Co-authored-by: Christopher Cooper <[email protected]> Co-authored-by: Zongheng Yang <[email protected]>

#62) * [perf] use uv for venv creation and pip install (#4414) * Revert "remove `uv` from runtime setup due to azure installation issue (#4401)" This reverts commit 0b20d56. * on azure, use --prerelease=allow to install azure-cli * use uv venv --seed * fix backwards compatibility * really fix backwards compatibility * use uv to set up controller dependencies * fix python 3.8 * lint * add missing file * update comment * split out azure-cli dep * fix lint for dependencies * use runpy.run_path rather than modifying sys.path * fix cloud dependency installation commands * lint * Update sky/utils/controller_utils.py Co-authored-by: Zhanghao Wu <[email protected]> --------- Co-authored-by: Zhanghao Wu <[email protected]> * [Minor] README updates. (#4436) * [Minor] README touches. * update * update * make --fast robust against credential or wheel updates (#4289) * add config_dict['config_hash'] output to write_cluster_config * fix docstring for write_cluster_config This used to be true, but since #2943, 'ray' is the only provisioner. Add other keys that are now present instead. * when using --fast, check if config_hash matches, and if not, provision * mock hashing method in unit test This is needed since some files in the fake file mounts don't actually exist, like the wheel path. * check config hash within provision with lock held * address other PR review comments * rename to skip_if_no_cluster_updates Co-authored-by: Zhanghao Wu <[email protected]> * add assert details Co-authored-by: Zhanghao Wu <[email protected]> * address PR comments and update docstrings * fix test * update docstrings Co-authored-by: Zhanghao Wu <[email protected]> * address PR comments * fix lint and tests * Update sky/backends/cloud_vm_ray_backend.py Co-authored-by: Zhanghao Wu <[email protected]> * refactor skip_if_no_cluster_update var * clarify comment * format exception --------- Co-authored-by: Zhanghao Wu <[email protected]> * [k8s] Add resource limits only if they exist (#4440) Add limits only if they exist * [robustness] cover some potential resource leakage cases (#4443) * if a newly-created cluster is missing from the cloud, wait before deleting Addresses #4431. * confirm cluster actually terminates before deleting from the db * avoid deleting cluster data outside the primary provision loop * tweaks * Apply suggestions from code review Co-authored-by: Zhanghao Wu <[email protected]> * use usage_intervals for new cluster detection get_cluster_duration will include the total duration of the cluster since its initial launch, while launched_at may be reset by sky launch on an existing cluster. So this is a more accurate method to check. * fix terminating/stopping state for Lambda and Paperspace * Revert "use usage_intervals for new cluster detection" This reverts commit aa6d2e9. * check cloud.STATUS_VERSION before calling query_instances * avoid try/catch when querying instances * update comments --------- Co-authored-by: Zhanghao Wu <[email protected]> * smoke tests support storage mount only (#4446) * smoke tests support storage mount only * fix verify command * rename to only_mount * [Feature] support spot pod on RunPod (#4447) * wip * wip * wip * wip * wip * wip * resolve comments * wip * wip * wip * wip * wip * wip --------- Co-authored-by: hwei <[email protected]> * use lazy import for runpod (#4451) Fixes runpod import issues introduced in #4447. * [k8s] Fix show-gpus when running with incluster auth (#4452) * Add limits only if they exist * Fix incluster auth handling * Merge branch 'master' of github.com:skypilot-org/skypilot into restapi * Add comment * Not mutate azure dep list at runtime (#4457) * format --------- Co-authored-by: Christopher Cooper <[email protected]> Co-authored-by: Zongheng Yang <[email protected]> Co-authored-by: Romil Bhardwaj <[email protected]> Co-authored-by: zpoint <[email protected]> Co-authored-by: Hong <[email protected]> Co-authored-by: hwei <[email protected]> Co-authored-by: Yika <[email protected]>

* [perf] use uv for venv creation and pip install (#4414) * Revert "remove `uv` from runtime setup due to azure installation issue (#4401)" This reverts commit 0b20d56. * on azure, use --prerelease=allow to install azure-cli * use uv venv --seed * fix backwards compatibility * really fix backwards compatibility * use uv to set up controller dependencies * fix python 3.8 * lint * add missing file * update comment * split out azure-cli dep * fix lint for dependencies * use runpy.run_path rather than modifying sys.path * fix cloud dependency installation commands * lint * Update sky/utils/controller_utils.py Co-authored-by: Zhanghao Wu <[email protected]> --------- Co-authored-by: Zhanghao Wu <[email protected]> * [Minor] README updates. (#4436) * [Minor] README touches. * update * update * make --fast robust against credential or wheel updates (#4289) * add config_dict['config_hash'] output to write_cluster_config * fix docstring for write_cluster_config This used to be true, but since #2943, 'ray' is the only provisioner. Add other keys that are now present instead. * when using --fast, check if config_hash matches, and if not, provision * mock hashing method in unit test This is needed since some files in the fake file mounts don't actually exist, like the wheel path. * check config hash within provision with lock held * address other PR review comments * rename to skip_if_no_cluster_updates Co-authored-by: Zhanghao Wu <[email protected]> * add assert details Co-authored-by: Zhanghao Wu <[email protected]> * address PR comments and update docstrings * fix test * update docstrings Co-authored-by: Zhanghao Wu <[email protected]> * address PR comments * fix lint and tests * Update sky/backends/cloud_vm_ray_backend.py Co-authored-by: Zhanghao Wu <[email protected]> * refactor skip_if_no_cluster_update var * clarify comment * format exception --------- Co-authored-by: Zhanghao Wu <[email protected]> * [k8s] Add resource limits only if they exist (#4440) Add limits only if they exist * [robustness] cover some potential resource leakage cases (#4443) * if a newly-created cluster is missing from the cloud, wait before deleting Addresses #4431. * confirm cluster actually terminates before deleting from the db * avoid deleting cluster data outside the primary provision loop * tweaks * Apply suggestions from code review Co-authored-by: Zhanghao Wu <[email protected]> * use usage_intervals for new cluster detection get_cluster_duration will include the total duration of the cluster since its initial launch, while launched_at may be reset by sky launch on an existing cluster. So this is a more accurate method to check. * fix terminating/stopping state for Lambda and Paperspace * Revert "use usage_intervals for new cluster detection" This reverts commit aa6d2e9. * check cloud.STATUS_VERSION before calling query_instances * avoid try/catch when querying instances * update comments --------- Co-authored-by: Zhanghao Wu <[email protected]> * smoke tests support storage mount only (#4446) * smoke tests support storage mount only * fix verify command * rename to only_mount * [Feature] support spot pod on RunPod (#4447) * wip * wip * wip * wip * wip * wip * resolve comments * wip * wip * wip * wip * wip * wip --------- Co-authored-by: hwei <[email protected]> * use lazy import for runpod (#4451) Fixes runpod import issues introduced in #4447. * [k8s] Fix show-gpus when running with incluster auth (#4452) * Add limits only if they exist * Fix incluster auth handling * Not mutate azure dep list at runtime (#4457) * add 1, 2, 4 size H100's to GCP (#4456) * add 1, 2, 4 size H100's to GCP * update * Support buildkite CICD and restructure smoke tests (#4396) * event based smoke test * more event based smoke test * more test cases * more test cases with managed jobs * bug fix * bump up seconds * merge master and resolve conflict * more test case * support test_managed_jobs_pipeline_failed_setup * support test_managed_jobs_recovery_aws * manged job status * bug fix * test managed job cancel * test_managed_jobs_storage * more test cases * resolve pr comment * private member function * bug fix * restructure * fix import * buildkite config * fix stdout problem * update pipeline test * test again * smoke test for buildkite * remove unsupport cloud for now * merge branch 'reliable_smoke_test_more' * bug fix * bug fix * bug fix * test pipeline pre merge * build test * test again * trigger test * bug fix * generate pipeline * robust generate pipeline * refactor pipeline * remove runpod * hot fix to pass smoke test * random order * allow parameter * bug fix * bug fix * exclude lambda cloud * dynamic generate pipeline * fix pre-commit * format * support SUPPRESS_SENSITIVE_LOG * support env SKYPILOT_SUPPRESS_SENSITIVE_LOG to suppress debug log * support env SKYPILOT_SUPPRESS_SENSITIVE_LOG to suppress debug log * add backward_compatibility_tests to pipeline * pip install uv for backward compatibility test * import style * generate all cloud * resolve PR comment * update comment * naming fix * grammar correction * resolve PR comment * fix import * fix import * support gcp on pre merge test * no gcp test case for pre merge * [k8s] Make node termination robust (#4469) * Add limits only if they exist * retry deletion * lint * lint * comments * lint * [Catalog] Bump catalog schema version (#4470) * Bump catalog schema version * trigger CI * [core] skip provider.availability_zone in the cluster config hash (#4463) skip provider.availability_zone in the cluster config hash * remove sky jobs launch --fast (#4467) * remove sky jobs launch --fast The --fast behavior is now always enabled. This was unsafe before but since \#4289 it should be safe. We will remove the flag before 0.8.0 so that it never touches a stable version. sky launch still has the --fast flag. This flag is unsafe because it could cause setup to be skipped even though it should be re-run. In the managed jobs case, this is not an issue because we fully control the setup and know it will not change. * fix lint * [docs] Change urls to docs.skypilot.co, add 404 page (#4413) * Add 404 page, change to docs.skypilot.co * lint * [UX] Fix unnecessary OCI logging (#4476) Sync PR: fix-oci-logging-master Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * [Example] PyTorch distributed training with minGPT (#4464) * Add example for distributed pytorch * update * Update examples/distributed-pytorch/README.md Co-authored-by: Romil Bhardwaj <[email protected]> * Update examples/distributed-pytorch/README.md Co-authored-by: Romil Bhardwaj <[email protected]> * Update examples/distributed-pytorch/README.md Co-authored-by: Romil Bhardwaj <[email protected]> * Update examples/distributed-pytorch/README.md Co-authored-by: Romil Bhardwaj <[email protected]> * Update examples/distributed-pytorch/README.md Co-authored-by: Romil Bhardwaj <[email protected]> * Update examples/distributed-pytorch/README.md Co-authored-by: Romil Bhardwaj <[email protected]> * Update examples/distributed-pytorch/README.md Co-authored-by: Romil Bhardwaj <[email protected]> * Fix --------- Co-authored-by: Romil Bhardwaj <[email protected]> * Add tests for Azure spot instance (#4475) * verify azure spot instance * string style * echo * echo vm detail * bug fix * remove comment * rename pre-merge test to quicktest-core (#4486) * rename to test core * rename file * [k8s] support to use custom gpu resource name if it's not nvidia.com/gpu (#4337) * [k8s] support to use custom gpu resource name if it's not nvidia.com/gpu Signed-off-by: nkwangleiGIT <[email protected]> * fix format issue Signed-off-by: nkwangleiGIT <[email protected]> --------- Signed-off-by: nkwangleiGIT <[email protected]> * [k8s] Fix IPv6 ssh support (#4497) * Add limits only if they exist * Fix ipv6 support * Fix ipv6 support * [Serve] Add and adopt least load policy as default poicy. (#4439) * [Serve] Add and adopt least load policy as default poicy. * Docs & smoke tests * error message for different lb policy * add minimal example * fix * [Docs] Update logo in docs (#4500) * WIP updating Elisa logo; issues with light/dark modes * Fix SVG in navbar rendering by hardcoding SVG + defining text color in css * Update readme images * newline --------- Co-authored-by: Zongheng Yang <[email protected]> * Replace `len()` Zero Checks with Pythonic Empty Sequence Checks (#4298) * style: mainly replace len() comparisons with 0/1 with pythonic empty sequence checks * chore: more typings * use `df.empty` for dataframe * fix: more `df.empty` * format * revert partially * style: add back comments * style: format * refactor: `dict[str, str]` Co-authored-by: Tian Xia <[email protected]> --------- Co-authored-by: Tian Xia <[email protected]> * [Docs] Fix logo file path (#4504) * Add limits only if they exist * rename * [Storage] Show logs for storage mount (#4387) * commit for logging change * logger for storage * grammar * fix format * better comment * resolve copilot review * resolve PR comment * remove unuse var * Update sky/data/data_utils.py Co-authored-by: Romil Bhardwaj <[email protected]> * resolve PR comment * update comment for get_run_timestamp * rename backend_util.get_run_timestamp to sky_logging.get_run_timestamp --------- Co-authored-by: Romil Bhardwaj <[email protected]> * [Examples] Update Ollama setup commands (#4510) wip * [OCI] Support OCI Object Storage (#4501) * OCI Object Storage Support * example yaml update * example update * add more example yaml * Support RClone-RPM pkg * Add smoke test * ver * smoke test * Resolve dependancy conflict between oci-cli and runpod * Use latest RClone version (v1.68.2) * minor optimize * Address review comments * typo * test * sync code with repo * Address review comments & more testing. * address one more comment * [Jobs] Allowing to specify intermediate bucket for file upload (#4257) * debug * support workdir_bucket_name config on yaml file * change the match statement to if else due to mypy limit * pass mypy * yapf format fix * reformat * remove debug line * all dir to same bucket * private member function * fix mypy * support sub dir config to separate to different directory * rename and add smoke test * bucketname * support sub dir mount * private member for _bucket_sub_path and smoke test fix * support copy mount for sub dir * support gcs, s3 delete folder * doc * r2 remove_objects_from_sub_path * support azure remove directory and cos remove * doc string for remove_objects_from_sub_path * fix sky jobs subdir issue * test case update * rename to _bucket_sub_path * change the config schema * setter * bug fix and test update * delete bucket depends on user config or sky generated * add test case * smoke test bug fix * robust smoke test * fix comment * bug fix * set the storage manually * better structure * fix mypy * Update docs/source/reference/config.rst Co-authored-by: Romil Bhardwaj <[email protected]> * Update docs/source/reference/config.rst Co-authored-by: Romil Bhardwaj <[email protected]> * limit creation for bucket and delete sub dir only * resolve comment * Update docs/source/reference/config.rst Co-authored-by: Romil Bhardwaj <[email protected]> * Update sky/utils/controller_utils.py Co-authored-by: Romil Bhardwaj <[email protected]> * resolve PR comment * bug fix * bug fix * fix test case * bug fix * fix * fix test case * bug fix * support is_sky_managed param in config * pass param intermediate_bucket_is_sky_managed * resolve PR comment * Update sky/utils/controller_utils.py Co-authored-by: Romil Bhardwaj <[email protected]> * hide bucket creation log * reset green color * rename is_sky_managed to _is_sky_managed * bug fix * retrieve _is_sky_managed from stores * propogate the log --------- Co-authored-by: Romil Bhardwaj <[email protected]> * [Core] Deprecate LocalDockerBackend (#4516) Deprecate local docker backend * [docs] Add newer examples for AI tutorial and distributed training (#4509) * Update tutorial and distributed training examples. * Add examples link * add rdvz * [k8s] Fix L40 detection for nvidia GFD labels (#4511) Fix L40 detection * [docs] Support OCI Object Storage (#4513) * Support OCI Object Storage * Add oci bucket for file_mount * [Docs] Disable Kapa AI (#4518) Disable kapa * [DigitalOcean] droplet integration (#3832) * init digital ocean droplet integration * abbreviate cloud name * switch to pydo * adjust polling logic and mount block storage to instance * filter by paginated * lint * sky launch, start, stop functional * fix credential file mounts, autodown works now * set gpu droplet image * cleanup * remove more tests * atomically destroy instance and block storage simulatenously * install docker * disable spot test * fix ip address bug for multinode * lint * patch ssh from job/serve controller * switch to EA slugs * do adaptor * lint * Update sky/clouds/do.py Co-authored-by: Tian Xia <[email protected]> * Update sky/clouds/do.py Co-authored-by: Tian Xia <[email protected]> * comment template * comment patch * add h100 test case * comment on instance name length * Update sky/clouds/do.py Co-authored-by: Tian Xia <[email protected]> * Update sky/clouds/service_catalog/do_catalog.py Co-authored-by: Tian Xia <[email protected]> * comment on max node char len * comment on weird azure import * comment acc price is included in instance price * fix return type * switch with do_utils * remove broad except * Update sky/provision/do/instance.py Co-authored-by: Tian Xia <[email protected]> * Update sky/provision/do/instance.py Co-authored-by: Tian Xia <[email protected]> * remove azure * comment on non_terminated_only * add open port debug message * wrap start instance api * use f-string * wrap stop * wrap instance down * assert credentials and check against all contexts * assert client is None * remove pending instances during instance restart * wrap rename * rename ssh key var * fix tags * add tags for block device * f strings for errors * support image ids * update do tests * only store head instance id * rename image slugs * add digital ocean alias * wait for docker to be available * update requirements and tests * increase docker timeout * lint * move tests * lint * patch test * lint * typo fix * fix typo * patch tests * fix tests * no_mark spot test * handle 2cpu serve tests * lint * lint * use logger.debug * fix none cred path * lint * handle get_cred path * pylint * patch for DO test_optimizer_dryruns.py * revert optimizer dryrun --------- Co-authored-by: Tian Xia <[email protected]> Co-authored-by: Ubuntu <[email protected]> * [Docs] Refactor pod_config docs (#4427) * refactor pod_config docs * Update docs/source/reference/kubernetes/kubernetes-getting-started.rst Co-authored-by: Zongheng Yang <[email protected]> * Update docs/source/reference/kubernetes/kubernetes-getting-started.rst Co-authored-by: Zongheng Yang <[email protected]> --------- Co-authored-by: Zongheng Yang <[email protected]> * [OCI] Set default image to ubuntu LTS 22.04 (#4517) * set default gpu image to skypilot:gpu-ubuntu-2204 * add example * remove comment line * set cpu default image to 2204 * update change history * [OCI] 1. Support specify OS with custom image id. 2. Corner case fix (#4524) * Support specify os type with custom image id. * trim space * nit * comment * Update intermediate bucket related doc (#4521) * doc * Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj <[email protected]> * Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj <[email protected]> * Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj <[email protected]> * Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj <[email protected]> * Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj <[email protected]> * Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj <[email protected]> * add tip * minor changes --------- Co-authored-by: Romil Bhardwaj <[email protected]> * [aws] cache user identity by 'aws configure list' (#4507) * [aws] cache user identity by 'aws configure list' Signed-off-by: Aylei <[email protected]> * refine get_user_identities docstring Signed-off-by: Aylei <[email protected]> * address review comments Signed-off-by: Aylei <[email protected]> --------- Signed-off-by: Aylei <[email protected]> * [k8s] Add validation for pod_config #4206 (#4466) * [k8s] Add validation for pod_config #4206 Check pod_config when run 'sky check k8s' by using k8s api * update: check pod_config when launch check merged pod_config during launch using k8s api * fix test * ignore check failed when test with dryrun if there is no kube config in env, ignore ValueError when launch with dryrun. For now, we don't support check schema offline. * use deserialize api to check pod_config schema * test * create another api_client with no kubeconfig * test * update error message * update test * test * test * Update sky/backends/backend_utils.py --------- Co-authored-by: Romil Bhardwaj <[email protected]> * [core] fix wheel timestamp check (#4488) Previously, we were only taking the max timestamp of all the subdirectories of the given directory. So the timestamp could be incorrect if only a file changed, and no directory changed. This fixes the issue by looking at all directories and files given by os.walk(). * [docs] Add image_id doc in task YAML for OCI (#4526) * Add image_id doc for OCI * nit * Update docs/source/reference/yaml-spec.rst Co-authored-by: Tian Xia <[email protected]> --------- Co-authored-by: Tian Xia <[email protected]> * [UX] warning before launching jobs/serve when using a reauth required credentials (#4479) * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * Update sky/backends/cloud_vm_ray_backend.py Minor fix * Update sky/clouds/aws.py Co-authored-by: Romil Bhardwaj <[email protected]> * wip * minor changes * wip --------- Co-authored-by: hong <[email protected]> Co-authored-by: Romil Bhardwaj <[email protected]> * [GCP] Activate service account for storage and controller (#4529) * Activate service account for storage * disable logging if not using service account * Activate for controller as well. * revert controller activate * Add comments * format * fix smoke * [OCI] Support reuse existing VCN for SkyServe (#4530) * Support reuse existing VCN for SkyServe * fix * remove unused import * format * [docs] OCI: advanced configuration & add vcn_ocid (#4531) * Add vcn_ocid configuration * Update config.rst * fix merge issues WIP * fix merging issues * fix imports * fix stores --------- Signed-off-by: nkwangleiGIT <[email protected]> Signed-off-by: Aylei <[email protected]> Co-authored-by: Christopher Cooper <[email protected]> Co-authored-by: Zongheng Yang <[email protected]> Co-authored-by: Romil Bhardwaj <[email protected]> Co-authored-by: zpoint <[email protected]> Co-authored-by: Hong <[email protected]> Co-authored-by: hwei <[email protected]> Co-authored-by: Yika <[email protected]> Co-authored-by: Seth Kimmel <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Lei <[email protected]> Co-authored-by: Tian Xia <[email protected]> Co-authored-by: Andy Lee <[email protected]> Co-authored-by: Romil Bhardwaj <[email protected]> Co-authored-by: Hysun He <[email protected]> Co-authored-by: Andrew Aikawa <[email protected]> Co-authored-by: Ubuntu <[email protected]> Co-authored-by: Aylei <[email protected]> Co-authored-by: Chester Li <[email protected]> Co-authored-by: hong <[email protected]>

cg505 added 3 commits November 25, 2024 15:00

Revert "remove uv from runtime setup due to azure installation issue (

9ccb69e

skypilot-org#4401)" This reverts commit 0b20d56.

on azure, use --prerelease=allow to install azure-cli

a31e4c1

use uv venv --seed

7574b94

cg505 requested review from romilbhardwaj and Michaelvll November 25, 2024 23:23

fix backwards compatibility

11040e3

Michaelvll reviewed Nov 26, 2024

View reviewed changes

cg505 added 2 commits November 26, 2024 11:54

really fix backwards compatibility

c8c910c

use uv to set up controller dependencies

64a0981

Michaelvll reviewed Nov 27, 2024

View reviewed changes

cg505 added 4 commits November 27, 2024 09:56

fix python 3.8

46b7e15

lint

b06b77c

add missing file

62a780a

update comment

2a1b494

cg505 requested a review from Michaelvll November 27, 2024 19:11

Michaelvll reviewed Nov 27, 2024

View reviewed changes

cg505 added 2 commits November 27, 2024 11:39

split out azure-cli dep

9ef8d22

fix lint for dependencies

1c0ee29

Michaelvll mentioned this pull request Dec 2, 2024

[Tests] Move tests to uv to speed up the dependency installation by >10x #4424

Merged

5 tasks

Michaelvll reviewed Dec 2, 2024

View reviewed changes

sky/utils/controller_utils.py Show resolved Hide resolved

cg505 added 3 commits December 3, 2024 11:32

use runpy.run_path rather than modifying sys.path

a392afd

fix cloud dependency installation commands

4fda5fe

lint

9e188a6

cg505 requested a review from Michaelvll December 3, 2024 20:05

Michaelvll approved these changes Dec 3, 2024

View reviewed changes

sky/utils/controller_utils.py Outdated Show resolved Hide resolved

Update sky/utils/controller_utils.py

478c56e

Co-authored-by: Zhanghao Wu <[email protected]>

cg505 merged commit de561c2 into skypilot-org:master Dec 3, 2024
19 checks passed

cg505 mentioned this pull request Dec 11, 2024

Support buildkite CICD and restructure smoke tests #4396

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[perf] use uv for venv creation and pip install #4414

[perf] use uv for venv creation and pip install #4414

cg505 commented Nov 25, 2024 •

edited

Loading

cg505 commented Nov 25, 2024

Michaelvll left a comment

Michaelvll Nov 26, 2024

cg505 Nov 26, 2024

Michaelvll left a comment

Michaelvll Nov 27, 2024

Michaelvll Nov 27, 2024

Michaelvll Nov 27, 2024

cg505 commented Nov 27, 2024

Michaelvll left a comment

Michaelvll Nov 27, 2024

cg505 Nov 27, 2024

cg505 Nov 27, 2024

cg505 Nov 27, 2024

cg505 Dec 3, 2024

Michaelvll Nov 27, 2024

Michaelvll Nov 27, 2024

cg505 Nov 27, 2024

cg505 Dec 3, 2024

Michaelvll left a comment

Michaelvll left a comment •

edited

Loading

cg505 commented Dec 3, 2024

		'if [ "{cloud}" = "azure" ]; then '
		f'{SKY_UV_PIP_CMD} install --prerelease=allow "azure-cli>=2.65.0"; fi;'

		@@ -25,6 +25,7 @@
		from sky.jobs import utils as managed_job_utils

[perf] use uv for venv creation and pip install #4414

[perf] use uv for venv creation and pip install #4414

Conversation

cg505 commented Nov 25, 2024 • edited Loading

cg505 commented Nov 25, 2024

Michaelvll left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Michaelvll left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cg505 commented Nov 27, 2024

Michaelvll left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Michaelvll left a comment

Choose a reason for hiding this comment

Michaelvll left a comment • edited Loading

Choose a reason for hiding this comment

cg505 commented Dec 3, 2024

cg505 commented Nov 25, 2024 •

edited

Loading

Michaelvll left a comment •

edited

Loading