[Jobs] Allowing to specify intermediate bucket for file upload #4257

zpoint · 2024-11-04T05:26:55Z

Feature for #3978

Add a config jobs: bucket in ~/.sky/config.yaml.
- Support s3, gcs, azure, r2, ibm
- If this bucket is specified, the bucket name and storage type will be used. Otherwise, one will be generated named: skypilot-filemounts-{username}-{run_id}.
- Add an attribute for storage YAML config: _bucket_sub_path(Support both COPY and MOUNT based on the _bucket_sub_path, The user will only see the subpath instead of the whole bucket if a subpath is specified)
  - _ means internal use only, remove _ to expose in the future
- smoke test
  - Add a smoke test case: pytest -s tests/test_smoke.py::test_managed_jobs_intermediate_storage
    - Verify that the new config file is loaded and intermediate bucket names in the config are used.
  - Add a test case: pytest -s tests/test_smoke.py::TestStorageWithCredentials::test_bucket_sub_path --gcp
    - Verify that the bucket subdirectory file sync and deletion have the expected behavior. Mainly, ensure the _bucket_sub_path works as expected.

For managed jobs:

All these file sync operations will now use the intermediate buckets. Previously, each file sync would create a new temporary bucket:
- Previous:
  - workdir specified in the task.yaml workdir -> {tmp-bucket-A}/workdir
  - Local file mount specified in the task.yaml file_mount -> {tmp-bucket-B}{i}
  - Temporary files tmp_files -> {tmp-bucket-C}/tmp-files
- Now:
  - workdir specified in the task.yaml workdir -> {intermediate-bucket}/job-{run_id}/workdir
  - Local file mount specified in the task.yaml file_mount -> {intermediate-bucket}/job-{run_id}/local-file-mounts/{i}
  - Temporary files tmp_files -> {intermediate-bucket}/job-{run_id}/tmp-files
Clean up
- After the managed job finishes, when cleanup is triggered
  - If user doesn't specific bucket config, sky generated {intermediate-bucket}/job-{run_id} bucket will be deleted automatically as before.
  - If user specific bucket name in config, the directory created above will be deleted, the bucket still exists after that, and the user needs to manually delete the bucket as usual.(Due to potential race problem)
- Currently aws, gcs, r2 use the command line support to remove the directory, while cos and https(azure) use the SDK to list all files under directory and then delete

Even using the same bucket, each job creates its own subdirectory based on the '{run-id}', and subdirectories of auto-created files will be cleaned up after the job is finished.

This allows users who don't have permission to create a bucket to manually specify a bucket name under ~/.sky/config.yaml.

Test plan

smoke test

pytest -s tests/test_smoke.py::test_managed_jobs_storage
pytest -s tests/test_smoke.py::TestStorageWithCredentials::test_bucket_sub_path --aws

custom test

(sky) ➜  cat ~/.sky/config.yaml
jobs:
  bucket: s3://zpoint-bucket-s3/

(sky) ➜ cat ~/Desktop/hello-sky/work_dir_1.yaml
name: test_workdir_bucket_name_1

workdir: .

resources:
  cloud: aws
  instance_type: t3.small

file_mounts:
  # this will use the user config
  /checkpoint:
    name: zpoint-filemounts-bucket
    source: ~/Desktop/dir1
    mode: MOUNT
    store: azure

  # these will all use same bucket configured in ~/.sky/config.yaml jobs->bucket now for bucket storage
  /dir1: ~/Desktop/dir1
  /dir2: ~/Desktop/dir2
  /dir3/dir3.py: ~/Desktop/dir1/dir1.py

run: |
 sleep 10
  ls /checkpoint
  ls .

Launch

(sky) ➜ sky jobs launch ~/Desktop/hello-sky/work_dir_1.yaml
⠋ Syncing ~/Desktop/dir1 -> https://xxx/zpoint-filemounts-bucket/
⠹ Syncing . -> s3://zpoint-bucket-s3/
⠧ Syncing ~/Desktop/dir1 -> s3://zpoint-bucket-s3/
⠦ Syncing ~/Desktop/dir2 -> s3://zpoint-bucket-s3/
⠴ Syncing /var/folders/83/zxqx914s57x310rfnhq8kk9r0000gn/T/skypilot-filemounts-files-aca97801 -> s3://zpoint-bucket-s3/

Looks good

Tested (run the relevant ones):

Code formatting: bash format.sh
Any manual or new tests for this PR (please specify below)
All smoke tests: pytest tests/test_smoke.py
Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

romilbhardwaj

Thanks @zpoint! Left some comments about the design, correctness and lifecycle management for the bucket contents.

sky/utils/controller_utils.py

sky/data/storage.py

tests/test_smoke.py

sky/utils/controller_utils.py

sky/data/storage.py

sky/utils/controller_utils.py

romilbhardwaj · 2024-11-11T20:55:44Z

sky/utils/schemas.py

+    jobs_configs['properties']['bucket'] = {
+        'type': 'string',
+        'pattern': '^(https|s3|gs|r2|cos)://.+',
+        'required': []
+    }


Maybe we should include it in controller_resources_schema and enable this feature for SkyServe as well?

Should we add this to our docs https://github.com/skypilot-org/skypilot/blob/master/docs/source/reference/config.rst?

Adde to controller_resources_schema

Could u point me out the code to change if I need to enable for SkyServe?

sky/data/storage.py

sky/utils/controller_utils.py

tests/test_yamls/intermediate_bucket.yaml

romilbhardwaj · 2024-11-11T21:29:20Z

Thanks @zpoint! I'm trying to run this yaml:

resources:
  cloud: aws

file_mounts:
  ~/aws: ~/aws

workdir: ~/tmp-workdir

num_nodes: 1

run: |
  echo "Hello, world!"
  ls ~/aws
  ls .

The task output should show me contents of my workdir and file_mount. Instead, I get:

├── Waiting for task resources on 1 node.
└── Job started. Streaming logs... (Ctrl-C to exit log streaming; job will not be killed)
(sky-3ba1-romilb, pid=2100) Hello, world!
(sky-3ba1-romilb, pid=2100) job-89ab894f
(sky-3ba1-romilb, pid=2100) job-89ab894f
✓ Managed job finished: 1 (status: SUCCEEDED).

Related to this: #4257 (comment)?

sky/data/storage.py

romilbhardwaj · 2024-12-19T07:00:00Z

sky/data/storage.py

+        is_sky_managed: Optional[bool] = None,
+        sync_on_reconstruction: Optional[bool] = True,
+        _bucket_sub_path: Optional[str] = None,  # pylint: disable=invalid-name
+        _allow_bucket_creation: bool = True):  # pylint: disable=invalid-name


Why is this _allow_bucket_creation arg needed?

Can we instead check store.is_sky_managed field after the store's initialization to check if bucket was created, and if it was created then raise an error?

Eg. near L727 in controller_utils.py, we can initialize the store object and check is_sky_managed, and raise error appropriately.

store = store_cls(name=bucket_name, source=workdir, _bucket_sub_path=bucket_sub_path, **store_kwargs) if store.is_sky_managed: # Bucket was created, this should not happen since the bucket already exists # cleanup and raise error store.delete() raise ...

Let me know if I'm missing something. It would be great if we could avoid adding new arguments to the Store/Storage interface.

We can do that in controller_utils.py. This avoids adding a parameter but keeps the logic outside of the store. If we're not going to reuse the limitation of creating a bucket, it will be fine to handle it in controller_utils.py

sky/task.py

tests/test_yamls/intermediate_bucket.yaml

zpoint · 2024-12-22T03:51:16Z

@romilbhardwaj Review reminder 😊

romilbhardwaj

Thanks @zpoint! Tested functionality - works nicely. Left some comments on the code.

sky/data/storage.py

romilbhardwaj · 2024-12-23T22:16:58Z

sky/utils/controller_utils.py

+                    f'Configured to use a non-existent bucket: {store.name}, '
+                    'please check the jobs->bucket in your sky config.')


Suggested change

f'Configured to use a non-existent bucket: {store.name}, '

'please check the jobs->bucket in your sky config.')

f'Jobs bucket {store.name!r} does not exist. '

'Please check jobs.bucket configuration in your SkyPilot config.')

BTW, the logging shown above is quite confusing ("Created S3 bucket", "Deleted S3 bucket"). Is there a way to hide it / different way to check if the bucket already exists?

I 12-23 14:28:46 controller_utils.py:691] ⚙︎ Translating workdir to SkyPilot Storage... I 12-23 14:28:47 storage.py:1590] Created S3 bucket 'romil-jobs-bucket2' in us-east-1 I 12-23 14:28:54 storage.py:1386] Deleted S3 bucket romil-jobs-bucket2. sky.exceptions.StorageBucketCreateError: Configured to use a non-existent bucket: romil-jobs-bucket2, please check the jobs->bucket in your sky config.

Find a way to hide the above logs by using sky_logging.silent()

⚙︎ Translating workdir and file_mounts with local source paths to SkyPilot Storage... sky.exceptions.StorageBucketCreateError: Jobs bucket 'zpointggg-bucket-s3-intermediate' does not exist. Please check jobs.bucket configuration in your SkyPilot config.

sky/data/storage.py

Co-authored-by: Romil Bhardwaj <[email protected]>

sky/utils/controller_utils.py

romilbhardwaj

Thanks @zpoint! lgtm, we can merge this after storage related smoke tests pass.

romilbhardwaj · 2024-12-30T06:55:25Z

Should we merge this @zpoint?

zpoint · 2024-12-30T07:03:14Z

Should we merge this @zpoint?

I am setting up buildkite tests and will merge after the tests pass. It should be done today.
The problem is that I can only test aws, azure, and gcp, as I don't have IBM or Cloudflare accounts, but the changes also affect their storage, Does it sound good if I merge after aws, azuer, and gcp tests pass?
code. @romilbhardwaj

zpoint · 2024-12-30T08:20:17Z

Another OCI stoarge has been merged by @HysunHe. Comment out the _bucket_sub_path for OCI as a TODO and merge this first.

zpoint · 2024-12-30T08:54:52Z

Related smoke tests pass
Proceed merge

Michaelvll · 2025-01-07T00:40:32Z

sky/data/storage.py

+          _is_sky_managed: Optional[bool]; Indicates if the storage is managed
+            by Sky. Without this argument, the controller's behavior differs
+            from the local machine. For example, if a bucket does not exist:
+            Local Machine (is_sky_managed=True) →
+            Controller (is_sky_managed=False).
+            With this argument, the controller aligns with the local machine,
+            ensuring it retains the is_sky_managed information from the YAML.
+            During teardown, if is_sky_managed is True, the controller should
+            delete the bucket. Otherwise, it might mistakenly delete only the
+            sub-path, assuming is_sky_managed is False.


I am super confused by this additional _is_sky_managed field, as the underlying store can have a different is_sky_managed value, which one we should trust then? It seems when we are calling to_yaml_config we base on the is_sky_managed of the store, which makes it more confusing...

Also, can we consolidate the two fields: _sky_sky_managed and force_delete?

Also, can we just use persistent=True for the case that we only want to delete the sub_path.

cc'ing @romilbhardwaj @zpoint

persistent=True means the controller will trigger deletion behavior of the bucket. However the deletion behavior depends on whether the bucket is _is_sky_managed (Call API to delete or just delete local cache), whether it has sub_path configured(Call API to delete path or delete entire bucket).

@zpoint can't we use storage.force_delete coupled with store.is_sky_managed to build the deletion logic on the controller? I also found it confusing previously, and it seems like the use of force_delete should allow us to get rid of storage._is_sky_managed arg.

persistent=True means the controller will trigger deletion behavior of the bucket. However the deletion behavior depends on whether the bucket is _is_sky_managed (Call API to delete or just delete local cache), whether it has sub_path configured(Call API to delete path or delete entire bucket).

Note: persistent=True means controller will NOT trigger deletion behavior, and it seems the behavior we want when it is a sub path.

We need to

Delete entire bucket if its system generated bucket

Delete only subpath if its user configed bucket

Its hard to do that without messing up their original force_delete/persistent meaning. I would suggest deduced is_sky_managed from bucket name.

* [perf] use uv for venv creation and pip install (#4414) * Revert "remove `uv` from runtime setup due to azure installation issue (#4401)" This reverts commit 0b20d56. * on azure, use --prerelease=allow to install azure-cli * use uv venv --seed * fix backwards compatibility * really fix backwards compatibility * use uv to set up controller dependencies * fix python 3.8 * lint * add missing file * update comment * split out azure-cli dep * fix lint for dependencies * use runpy.run_path rather than modifying sys.path * fix cloud dependency installation commands * lint * Update sky/utils/controller_utils.py Co-authored-by: Zhanghao Wu <[email protected]> --------- Co-authored-by: Zhanghao Wu <[email protected]> * [Minor] README updates. (#4436) * [Minor] README touches. * update * update * make --fast robust against credential or wheel updates (#4289) * add config_dict['config_hash'] output to write_cluster_config * fix docstring for write_cluster_config This used to be true, but since #2943, 'ray' is the only provisioner. Add other keys that are now present instead. * when using --fast, check if config_hash matches, and if not, provision * mock hashing method in unit test This is needed since some files in the fake file mounts don't actually exist, like the wheel path. * check config hash within provision with lock held * address other PR review comments * rename to skip_if_no_cluster_updates Co-authored-by: Zhanghao Wu <[email protected]> * add assert details Co-authored-by: Zhanghao Wu <[email protected]> * address PR comments and update docstrings * fix test * update docstrings Co-authored-by: Zhanghao Wu <[email protected]> * address PR comments * fix lint and tests * Update sky/backends/cloud_vm_ray_backend.py Co-authored-by: Zhanghao Wu <[email protected]> * refactor skip_if_no_cluster_update var * clarify comment * format exception --------- Co-authored-by: Zhanghao Wu <[email protected]> * [k8s] Add resource limits only if they exist (#4440) Add limits only if they exist * [robustness] cover some potential resource leakage cases (#4443) * if a newly-created cluster is missing from the cloud, wait before deleting Addresses #4431. * confirm cluster actually terminates before deleting from the db * avoid deleting cluster data outside the primary provision loop * tweaks * Apply suggestions from code review Co-authored-by: Zhanghao Wu <[email protected]> * use usage_intervals for new cluster detection get_cluster_duration will include the total duration of the cluster since its initial launch, while launched_at may be reset by sky launch on an existing cluster. So this is a more accurate method to check. * fix terminating/stopping state for Lambda and Paperspace * Revert "use usage_intervals for new cluster detection" This reverts commit aa6d2e9. * check cloud.STATUS_VERSION before calling query_instances * avoid try/catch when querying instances * update comments --------- Co-authored-by: Zhanghao Wu <[email protected]> * smoke tests support storage mount only (#4446) * smoke tests support storage mount only * fix verify command * rename to only_mount * [Feature] support spot pod on RunPod (#4447) * wip * wip * wip * wip * wip * wip * resolve comments * wip * wip * wip * wip * wip * wip --------- Co-authored-by: hwei <[email protected]> * use lazy import for runpod (#4451) Fixes runpod import issues introduced in #4447. * [k8s] Fix show-gpus when running with incluster auth (#4452) * Add limits only if they exist * Fix incluster auth handling * Not mutate azure dep list at runtime (#4457) * add 1, 2, 4 size H100's to GCP (#4456) * add 1, 2, 4 size H100's to GCP * update * Support buildkite CICD and restructure smoke tests (#4396) * event based smoke test * more event based smoke test * more test cases * more test cases with managed jobs * bug fix * bump up seconds * merge master and resolve conflict * more test case * support test_managed_jobs_pipeline_failed_setup * support test_managed_jobs_recovery_aws * manged job status * bug fix * test managed job cancel * test_managed_jobs_storage * more test cases * resolve pr comment * private member function * bug fix * restructure * fix import * buildkite config * fix stdout problem * update pipeline test * test again * smoke test for buildkite * remove unsupport cloud for now * merge branch 'reliable_smoke_test_more' * bug fix * bug fix * bug fix * test pipeline pre merge * build test * test again * trigger test * bug fix * generate pipeline * robust generate pipeline * refactor pipeline * remove runpod * hot fix to pass smoke test * random order * allow parameter * bug fix * bug fix * exclude lambda cloud * dynamic generate pipeline * fix pre-commit * format * support SUPPRESS_SENSITIVE_LOG * support env SKYPILOT_SUPPRESS_SENSITIVE_LOG to suppress debug log * support env SKYPILOT_SUPPRESS_SENSITIVE_LOG to suppress debug log * add backward_compatibility_tests to pipeline * pip install uv for backward compatibility test * import style * generate all cloud * resolve PR comment * update comment * naming fix * grammar correction * resolve PR comment * fix import * fix import * support gcp on pre merge test * no gcp test case for pre merge * [k8s] Make node termination robust (#4469) * Add limits only if they exist * retry deletion * lint * lint * comments * lint * [Catalog] Bump catalog schema version (#4470) * Bump catalog schema version * trigger CI * [core] skip provider.availability_zone in the cluster config hash (#4463) skip provider.availability_zone in the cluster config hash * remove sky jobs launch --fast (#4467) * remove sky jobs launch --fast The --fast behavior is now always enabled. This was unsafe before but since \#4289 it should be safe. We will remove the flag before 0.8.0 so that it never touches a stable version. sky launch still has the --fast flag. This flag is unsafe because it could cause setup to be skipped even though it should be re-run. In the managed jobs case, this is not an issue because we fully control the setup and know it will not change. * fix lint * [docs] Change urls to docs.skypilot.co, add 404 page (#4413) * Add 404 page, change to docs.skypilot.co * lint * [UX] Fix unnecessary OCI logging (#4476) Sync PR: fix-oci-logging-master Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * [Example] PyTorch distributed training with minGPT (#4464) * Add example for distributed pytorch * update * Update examples/distributed-pytorch/README.md Co-authored-by: Romil Bhardwaj <[email protected]> * Update examples/distributed-pytorch/README.md Co-authored-by: Romil Bhardwaj <[email protected]> * Update examples/distributed-pytorch/README.md Co-authored-by: Romil Bhardwaj <[email protected]> * Update examples/distributed-pytorch/README.md Co-authored-by: Romil Bhardwaj <[email protected]> * Update examples/distributed-pytorch/README.md Co-authored-by: Romil Bhardwaj <[email protected]> * Update examples/distributed-pytorch/README.md Co-authored-by: Romil Bhardwaj <[email protected]> * Update examples/distributed-pytorch/README.md Co-authored-by: Romil Bhardwaj <[email protected]> * Fix --------- Co-authored-by: Romil Bhardwaj <[email protected]> * Add tests for Azure spot instance (#4475) * verify azure spot instance * string style * echo * echo vm detail * bug fix * remove comment * rename pre-merge test to quicktest-core (#4486) * rename to test core * rename file * [k8s] support to use custom gpu resource name if it's not nvidia.com/gpu (#4337) * [k8s] support to use custom gpu resource name if it's not nvidia.com/gpu Signed-off-by: nkwangleiGIT <[email protected]> * fix format issue Signed-off-by: nkwangleiGIT <[email protected]> --------- Signed-off-by: nkwangleiGIT <[email protected]> * [k8s] Fix IPv6 ssh support (#4497) * Add limits only if they exist * Fix ipv6 support * Fix ipv6 support * [Serve] Add and adopt least load policy as default poicy. (#4439) * [Serve] Add and adopt least load policy as default poicy. * Docs & smoke tests * error message for different lb policy * add minimal example * fix * [Docs] Update logo in docs (#4500) * WIP updating Elisa logo; issues with light/dark modes * Fix SVG in navbar rendering by hardcoding SVG + defining text color in css * Update readme images * newline --------- Co-authored-by: Zongheng Yang <[email protected]> * Replace `len()` Zero Checks with Pythonic Empty Sequence Checks (#4298) * style: mainly replace len() comparisons with 0/1 with pythonic empty sequence checks * chore: more typings * use `df.empty` for dataframe * fix: more `df.empty` * format * revert partially * style: add back comments * style: format * refactor: `dict[str, str]` Co-authored-by: Tian Xia <[email protected]> --------- Co-authored-by: Tian Xia <[email protected]> * [Docs] Fix logo file path (#4504) * Add limits only if they exist * rename * [Storage] Show logs for storage mount (#4387) * commit for logging change * logger for storage * grammar * fix format * better comment * resolve copilot review * resolve PR comment * remove unuse var * Update sky/data/data_utils.py Co-authored-by: Romil Bhardwaj <[email protected]> * resolve PR comment * update comment for get_run_timestamp * rename backend_util.get_run_timestamp to sky_logging.get_run_timestamp --------- Co-authored-by: Romil Bhardwaj <[email protected]> * [Examples] Update Ollama setup commands (#4510) wip * [OCI] Support OCI Object Storage (#4501) * OCI Object Storage Support * example yaml update * example update * add more example yaml * Support RClone-RPM pkg * Add smoke test * ver * smoke test * Resolve dependancy conflict between oci-cli and runpod * Use latest RClone version (v1.68.2) * minor optimize * Address review comments * typo * test * sync code with repo * Address review comments & more testing. * address one more comment * [Jobs] Allowing to specify intermediate bucket for file upload (#4257) * debug * support workdir_bucket_name config on yaml file * change the match statement to if else due to mypy limit * pass mypy * yapf format fix * reformat * remove debug line * all dir to same bucket * private member function * fix mypy * support sub dir config to separate to different directory * rename and add smoke test * bucketname * support sub dir mount * private member for _bucket_sub_path and smoke test fix * support copy mount for sub dir * support gcs, s3 delete folder * doc * r2 remove_objects_from_sub_path * support azure remove directory and cos remove * doc string for remove_objects_from_sub_path * fix sky jobs subdir issue * test case update * rename to _bucket_sub_path * change the config schema * setter * bug fix and test update * delete bucket depends on user config or sky generated * add test case * smoke test bug fix * robust smoke test * fix comment * bug fix * set the storage manually * better structure * fix mypy * Update docs/source/reference/config.rst Co-authored-by: Romil Bhardwaj <[email protected]> * Update docs/source/reference/config.rst Co-authored-by: Romil Bhardwaj <[email protected]> * limit creation for bucket and delete sub dir only * resolve comment * Update docs/source/reference/config.rst Co-authored-by: Romil Bhardwaj <[email protected]> * Update sky/utils/controller_utils.py Co-authored-by: Romil Bhardwaj <[email protected]> * resolve PR comment * bug fix * bug fix * fix test case * bug fix * fix * fix test case * bug fix * support is_sky_managed param in config * pass param intermediate_bucket_is_sky_managed * resolve PR comment * Update sky/utils/controller_utils.py Co-authored-by: Romil Bhardwaj <[email protected]> * hide bucket creation log * reset green color * rename is_sky_managed to _is_sky_managed * bug fix * retrieve _is_sky_managed from stores * propogate the log --------- Co-authored-by: Romil Bhardwaj <[email protected]> * [Core] Deprecate LocalDockerBackend (#4516) Deprecate local docker backend * [docs] Add newer examples for AI tutorial and distributed training (#4509) * Update tutorial and distributed training examples. * Add examples link * add rdvz * [k8s] Fix L40 detection for nvidia GFD labels (#4511) Fix L40 detection * [docs] Support OCI Object Storage (#4513) * Support OCI Object Storage * Add oci bucket for file_mount * [Docs] Disable Kapa AI (#4518) Disable kapa * [DigitalOcean] droplet integration (#3832) * init digital ocean droplet integration * abbreviate cloud name * switch to pydo * adjust polling logic and mount block storage to instance * filter by paginated * lint * sky launch, start, stop functional * fix credential file mounts, autodown works now * set gpu droplet image * cleanup * remove more tests * atomically destroy instance and block storage simulatenously * install docker * disable spot test * fix ip address bug for multinode * lint * patch ssh from job/serve controller * switch to EA slugs * do adaptor * lint * Update sky/clouds/do.py Co-authored-by: Tian Xia <[email protected]> * Update sky/clouds/do.py Co-authored-by: Tian Xia <[email protected]> * comment template * comment patch * add h100 test case * comment on instance name length * Update sky/clouds/do.py Co-authored-by: Tian Xia <[email protected]> * Update sky/clouds/service_catalog/do_catalog.py Co-authored-by: Tian Xia <[email protected]> * comment on max node char len * comment on weird azure import * comment acc price is included in instance price * fix return type * switch with do_utils * remove broad except * Update sky/provision/do/instance.py Co-authored-by: Tian Xia <[email protected]> * Update sky/provision/do/instance.py Co-authored-by: Tian Xia <[email protected]> * remove azure * comment on non_terminated_only * add open port debug message * wrap start instance api * use f-string * wrap stop * wrap instance down * assert credentials and check against all contexts * assert client is None * remove pending instances during instance restart * wrap rename * rename ssh key var * fix tags * add tags for block device * f strings for errors * support image ids * update do tests * only store head instance id * rename image slugs * add digital ocean alias * wait for docker to be available * update requirements and tests * increase docker timeout * lint * move tests * lint * patch test * lint * typo fix * fix typo * patch tests * fix tests * no_mark spot test * handle 2cpu serve tests * lint * lint * use logger.debug * fix none cred path * lint * handle get_cred path * pylint * patch for DO test_optimizer_dryruns.py * revert optimizer dryrun --------- Co-authored-by: Tian Xia <[email protected]> Co-authored-by: Ubuntu <[email protected]> * [Docs] Refactor pod_config docs (#4427) * refactor pod_config docs * Update docs/source/reference/kubernetes/kubernetes-getting-started.rst Co-authored-by: Zongheng Yang <[email protected]> * Update docs/source/reference/kubernetes/kubernetes-getting-started.rst Co-authored-by: Zongheng Yang <[email protected]> --------- Co-authored-by: Zongheng Yang <[email protected]> * [OCI] Set default image to ubuntu LTS 22.04 (#4517) * set default gpu image to skypilot:gpu-ubuntu-2204 * add example * remove comment line * set cpu default image to 2204 * update change history * [OCI] 1. Support specify OS with custom image id. 2. Corner case fix (#4524) * Support specify os type with custom image id. * trim space * nit * comment * Update intermediate bucket related doc (#4521) * doc * Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj <[email protected]> * Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj <[email protected]> * Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj <[email protected]> * Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj <[email protected]> * Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj <[email protected]> * Update docs/source/examples/managed-jobs.rst Co-authored-by: Romil Bhardwaj <[email protected]> * add tip * minor changes --------- Co-authored-by: Romil Bhardwaj <[email protected]> * [aws] cache user identity by 'aws configure list' (#4507) * [aws] cache user identity by 'aws configure list' Signed-off-by: Aylei <[email protected]> * refine get_user_identities docstring Signed-off-by: Aylei <[email protected]> * address review comments Signed-off-by: Aylei <[email protected]> --------- Signed-off-by: Aylei <[email protected]> * [k8s] Add validation for pod_config #4206 (#4466) * [k8s] Add validation for pod_config #4206 Check pod_config when run 'sky check k8s' by using k8s api * update: check pod_config when launch check merged pod_config during launch using k8s api * fix test * ignore check failed when test with dryrun if there is no kube config in env, ignore ValueError when launch with dryrun. For now, we don't support check schema offline. * use deserialize api to check pod_config schema * test * create another api_client with no kubeconfig * test * update error message * update test * test * test * Update sky/backends/backend_utils.py --------- Co-authored-by: Romil Bhardwaj <[email protected]> * [core] fix wheel timestamp check (#4488) Previously, we were only taking the max timestamp of all the subdirectories of the given directory. So the timestamp could be incorrect if only a file changed, and no directory changed. This fixes the issue by looking at all directories and files given by os.walk(). * [docs] Add image_id doc in task YAML for OCI (#4526) * Add image_id doc for OCI * nit * Update docs/source/reference/yaml-spec.rst Co-authored-by: Tian Xia <[email protected]> --------- Co-authored-by: Tian Xia <[email protected]> * [UX] warning before launching jobs/serve when using a reauth required credentials (#4479) * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * wip * Update sky/backends/cloud_vm_ray_backend.py Minor fix * Update sky/clouds/aws.py Co-authored-by: Romil Bhardwaj <[email protected]> * wip * minor changes * wip --------- Co-authored-by: hong <[email protected]> Co-authored-by: Romil Bhardwaj <[email protected]> * [GCP] Activate service account for storage and controller (#4529) * Activate service account for storage * disable logging if not using service account * Activate for controller as well. * revert controller activate * Add comments * format * fix smoke * [OCI] Support reuse existing VCN for SkyServe (#4530) * Support reuse existing VCN for SkyServe * fix * remove unused import * format * [docs] OCI: advanced configuration & add vcn_ocid (#4531) * Add vcn_ocid configuration * Update config.rst * fix merge issues WIP * fix merging issues * fix imports * fix stores --------- Signed-off-by: nkwangleiGIT <[email protected]> Signed-off-by: Aylei <[email protected]> Co-authored-by: Christopher Cooper <[email protected]> Co-authored-by: Zongheng Yang <[email protected]> Co-authored-by: Romil Bhardwaj <[email protected]> Co-authored-by: zpoint <[email protected]> Co-authored-by: Hong <[email protected]> Co-authored-by: hwei <[email protected]> Co-authored-by: Yika <[email protected]> Co-authored-by: Seth Kimmel <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Lei <[email protected]> Co-authored-by: Tian Xia <[email protected]> Co-authored-by: Andy Lee <[email protected]> Co-authored-by: Romil Bhardwaj <[email protected]> Co-authored-by: Hysun He <[email protected]> Co-authored-by: Andrew Aikawa <[email protected]> Co-authored-by: Ubuntu <[email protected]> Co-authored-by: Aylei <[email protected]> Co-authored-by: Chester Li <[email protected]> Co-authored-by: hong <[email protected]>

zpoint added 10 commits November 3, 2024 14:58

debug

bbfe91a

support workdir_bucket_name config on yaml file

043dd28

change the match statement to if else due to mypy limit

bc767c8

pass mypy

8200400

yapf format fix

0e395ca

reformat

2b1a063

remove debug line

fa9cc69

all dir to same bucket

4c98e11

private member function

144a4a3

fix mypy

42de23a

romilbhardwaj self-requested a review November 6, 2024 18:42

romilbhardwaj requested changes Nov 7, 2024

View reviewed changes

sky/utils/controller_utils.py Outdated Show resolved Hide resolved

sky/utils/controller_utils.py Outdated Show resolved Hide resolved

support sub dir config to separate to different directory

888c0fa

romilbhardwaj reviewed Nov 8, 2024

View reviewed changes

sky/data/storage.py Outdated Show resolved Hide resolved

rename and add smoke test

3b1adcc

zpoint requested a review from romilbhardwaj November 8, 2024 09:13

romilbhardwaj reviewed Nov 9, 2024

View reviewed changes

sky/data/storage.py Outdated Show resolved Hide resolved

tests/test_smoke.py Outdated Show resolved Hide resolved

sky/utils/controller_utils.py Outdated Show resolved Hide resolved

sky/utils/controller_utils.py Outdated Show resolved Hide resolved

bucketname

06c7891

zpoint requested a review from romilbhardwaj November 9, 2024 15:39

support sub dir mount

3aaf0f1

zpoint commented Nov 11, 2024

View reviewed changes

sky/data/storage.py Outdated Show resolved Hide resolved

sky/data/storage.py Outdated Show resolved Hide resolved

romilbhardwaj reviewed Nov 11, 2024

View reviewed changes

tests/test_yamls/intermediate_bucket.yaml Show resolved Hide resolved

zpoint added 4 commits November 12, 2024 12:12

private member for _bucket_sub_path and smoke test fix

0116fa0

support copy mount for sub dir

0750900

support gcs, s3 delete folder

51dfcd3

doc

6ba05cc

zpoint requested a review from romilbhardwaj November 12, 2024 10:01

r2 remove_objects_from_sub_path

1751fab

zpoint added 3 commits December 16, 2024 16:39

bug fix

d4e28dc

support is_sky_managed param in config

2525b30

pass param intermediate_bucket_is_sky_managed

da6f32d

romilbhardwaj reviewed Dec 19, 2024

View reviewed changes

resolve PR comment

4a078c9

zpoint requested a review from romilbhardwaj December 20, 2024 04:34

romilbhardwaj reviewed Dec 23, 2024

View reviewed changes

zpoint and others added 6 commits December 24, 2024 10:27

Update sky/utils/controller_utils.py

3b9f8bc

Co-authored-by: Romil Bhardwaj <[email protected]>

Merge branch 'master' into dev/zeping/specific_bucket_for_sky_jobs

2f1e2b7

hide bucket creation log

51530a0

reset green color

6232731

rename is_sky_managed to _is_sky_managed

ab84547

bug fix

bc55222

zpoint requested a review from romilbhardwaj December 24, 2024 04:15

retrieve _is_sky_managed from stores

80a010a

romilbhardwaj reviewed Dec 25, 2024

View reviewed changes

sky/utils/controller_utils.py Show resolved Hide resolved

propogate the log

8708a81

zpoint requested a review from romilbhardwaj December 27, 2024 04:38

romilbhardwaj approved these changes Dec 27, 2024

View reviewed changes

zpoint added 2 commits December 30, 2024 16:10

Merge branch 'master' into dev/zeping/specific_bucket_for_sky_jobs

c4bac50

merge master

c101e8b

zpoint merged commit 7e40bcd into skypilot-org:master Dec 30, 2024
19 checks passed

Michaelvll reviewed Jan 7, 2025

View reviewed changes

zpoint mentioned this pull request Jan 8, 2025

remove field _is_sky_managed for intermediate bucket #4545

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Jobs] Allowing to specify intermediate bucket for file upload #4257

[Jobs] Allowing to specify intermediate bucket for file upload #4257

zpoint commented Nov 4, 2024 •

edited

Loading

romilbhardwaj left a comment

romilbhardwaj Nov 11, 2024

romilbhardwaj Nov 11, 2024

zpoint Nov 12, 2024

romilbhardwaj commented Nov 11, 2024

romilbhardwaj Dec 19, 2024

zpoint Dec 20, 2024

zpoint commented Dec 22, 2024

romilbhardwaj left a comment

romilbhardwaj Dec 23, 2024

romilbhardwaj Dec 23, 2024

zpoint Dec 24, 2024

romilbhardwaj left a comment

romilbhardwaj commented Dec 30, 2024

zpoint commented Dec 30, 2024 •

edited

Loading

zpoint commented Dec 30, 2024

zpoint commented Dec 30, 2024

Michaelvll Jan 7, 2025 •

edited

Loading

Michaelvll Jan 7, 2025

Michaelvll Jan 7, 2025

zpoint Jan 7, 2025

romilbhardwaj Jan 7, 2025

Michaelvll Jan 7, 2025

zpoint Jan 8, 2025 •

edited

Loading

		f'Configured to use a non-existent bucket: {store.name}, '
		'please check the jobs->bucket in your sky config.')

[Jobs] Allowing to specify intermediate bucket for file upload #4257

[Jobs] Allowing to specify intermediate bucket for file upload #4257

Conversation

zpoint commented Nov 4, 2024 • edited Loading

Test plan

smoke test

custom test

romilbhardwaj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

romilbhardwaj commented Nov 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zpoint commented Dec 22, 2024

romilbhardwaj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

romilbhardwaj left a comment

Choose a reason for hiding this comment

romilbhardwaj commented Dec 30, 2024

zpoint commented Dec 30, 2024 • edited Loading

zpoint commented Dec 30, 2024

zpoint commented Dec 30, 2024

Michaelvll Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zpoint Jan 8, 2025 • edited Loading

Choose a reason for hiding this comment

zpoint commented Nov 4, 2024 •

edited

Loading

zpoint commented Dec 30, 2024 •

edited

Loading

Michaelvll Jan 7, 2025 •

edited

Loading

zpoint Jan 8, 2025 •

edited

Loading