Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Storage] Rclone to Cache Storage #3455

Closed
wants to merge 54 commits into from

Conversation

shethhriday29
Copy link
Contributor

@shethhriday29 shethhriday29 commented Apr 21, 2024

Tested (run the relevant ones):

  • Code formatting: bash format.sh
  • Check if mountpoints remain mounted after restarting by running sky stop and sky start:
    • pytest tests/test_smoke.py::test_aws_storage_mounts_with_stop --aws
    • pytest tests/test_smoke.py::test_gcp_storage_mounts_with_stop --gcp
    • Manually running sky stop and sky start with comp_test.yaml to see if all the MOUNT_CACHED remains storage remains mounted.
  • Set of comprehensive manual tests with comp_test.yaml:
    • launch tests:
      • sky launch comp_test.yaml --cloud aws -y
      • sky launch comp_test.yaml --cloud gcp -y
    • managed spot jobs tests:
      • sky jobs launch comp_test.yaml --use-spot --cloud aws -y
      • sky jobs launch comp_test.yaml --use-spot --cloud gcp -y
    • launch on docker containers:
      • sky launch comp_test.yaml --cloud aws --image-id docker:continuumio/miniconda3:latest -y
      • sky launch comp_test.yaml --cloud gcp --image-id docker:continuumio/miniconda3:latest -y

comp_test.yaml:

file_mounts:
  #### S3 tests ####
  /s3-sky-managed-copy:
    name: s3-sky-managed-copy
    source: ~/sky_logs
    store: s3
    mode: COPY

  /s3-sky-managed-mount:
    name: s3-sky-managed-mount
    source: ~/sky_logs
    store: s3
    mode: MOUNT

  /s3-sky-managed-mount-cached:
    name: s3-sky-managed-mount-cached
    source: ~/sky_logs
    store: s3
    mode: MOUNT_CACHED

  /s3-external-private-by-user:
    source: s3://s3-external-private-by-user-dy
    mode: MOUNT_CACHED

  /s3-external-public-by-user:
    source: s3://s3-external-public-by-user-dy
    mode: MOUNT_CACHED

  /s3-external-public-not-by-user:
    source: s3://digitalcorpora
    mode: MOUNT_CACHED

  #### GCS tests ####
  /gcs-sky-managed-copy:
    name: gcs-sky-managed-copy
    source: ~/sky_logs
    store: gcs
    mode: COPY

  /gcs-sky-managed-mount:
    name: gcs-sky-managed-mount
    source: ~/sky_logs
    store: gcs
    mode: MOUNT

  /gcs-sky-managed-mount-cached:
    name: gcs-sky-managed-mount-cached
    source: ~/sky_logs
    store: gcs
    mode: MOUNT_CACHED

  /gcs-external-private-by-user:
    source: gs://gcs-external-private-by-user-dy
    mode: MOUNT_CACHED

  /gcs-external-public-by-user:
    source: gs://gcs-external-public-by-user-dy
    mode: MOUNT_CACHED

  /gcs-external-public-not-by-user:
    source: gs://gcp-public-data-sentinel-2
    mode: MOUNT_CACHED


workdir: ~/yaml

setup: |
  echo hi

run: |
  # Show workdir
  ls -l .

  # Show private/public storage copy/mount
  ls -l /s3-sky-managed-copy
  ls -l /s3-sky-managed-mount
  ls -l /s3-sky-managed-mount-cached
  ls -l /s3-external-private-by-user
  ls -l /s3-external-public-by-user
  ls -l /s3-external-public-not-by-user
  ls -l /gcs-sky-managed-copy
  ls -l /gcs-sky-managed-mount
  ls -l /gcs-sky-managed-mount-cached
  ls -l /gcs-external-private-by-user
  ls -l /gcs-external-public-by-user
  ls -l /gcs-external-public-not-by-user

  # Write files on a mounted storage with access
  date > /s3-sky-managed-mount/hellotest.txt
  date > /s3-sky-managed-mount-cached/hellotest.txt
  date > /s3-external-private-by-user/hellotest.txt
  date > /s3-external-public-by-user/hellotest.txt
  date > /gcs-sky-managed-mount/hellotest.txt
  date > /gcs-sky-managed-mount-cached/hellotest.txt
  date > /gcs-external-private-by-user/hellotest.txt
  date > /gcs-external-public-by-user/hellotest.txt

  # Confirm file was written
  cat /s3-sky-managed-mount/hellotest.txt
  cat /s3-sky-managed-mount-cached/hellotest.txt
  cat /s3-external-private-by-user/hellotest.txt
  cat /s3-external-public-by-user/hellotest.txt
  cat /gcs-sky-managed-mount/hellotest.txt
  cat /gcs-sky-managed-mount-cached/hellotest.txt
  cat /gcs-external-private-by-user/hellotest.txt
  cat /gcs-external-public-by-user/hellotest.txt

Sheth and others added 30 commits February 16, 2024 23:04
* Add serve for gemma and fix mixtral dependency

* Add hf token

* fix model len

* Add comment

* Serve your private gemma

* fix serve yaml

* readme

* Remove chat completion due to the wrong template

* add readme

* Update llm/gemma/README.md

Co-authored-by: Zongheng Yang <[email protected]>

* address comments

* Update README.md

Co-authored-by: Zongheng Yang <[email protected]>

* Update llm/gemma/README.md

Co-authored-by: Zongheng Yang <[email protected]>

* Update llm/gemma/README.md

Co-authored-by: Zongheng Yang <[email protected]>

* Update llm/gemma/README.md

Co-authored-by: Zongheng Yang <[email protected]>

* Change to it

* Add chat API

* use HF_TOKEN env

* typo

---------

Co-authored-by: Zongheng Yang <[email protected]>
* when removing cudo credential, sky check fails

* remove tips

* minor hint fix

* fix cluster version for k8s

* fix typo
…org#3218)

* Add retry for docker pull due to daemon not ready

* longer wait time

* longer wait time

* retry earlier

* add retry for retries as well

* longer wait time

* change wait time

* format

* Add comment

* Fix

* Fix indent for azure docker config

* Fix docker login config

* Fix comments

* More robust docker login config

* Add retry for docker check

* minor fix

* Add additional test for stop and start with docker

* Fix cancelled
…o hriday/unsloth

# Conflicts:
#	README.md
#	docs/source/index.rst
@shethhriday29 shethhriday29 marked this pull request as ready for review May 28, 2024 07:10
name: {{storage_name}}
source: ~/tmp-workdir
store: {{store_type}}
mode: RCLONE
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this new mode, or should we directly change the behavior of MOUNT to rclone?

Copy link
Collaborator

@landscapepainter landscapepainter Jul 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed online, we are referring this mode separately as MOUNT_CACHED mode, and keeping our original MOUNT mode as it is.

Copy link
Contributor

github-actions bot commented Nov 6, 2024

This PR is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the Stale label Nov 6, 2024
Copy link
Contributor

This PR was closed because it has been stalled for 10 days with no activity.

@github-actions github-actions bot closed this Nov 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants