Add device memory #2565

jc9123 · 2023-09-17T04:59:03Z

Modified fetch_azure.py so that the catalog now contains device memory for GPU.

Tested (run the relevant ones):

Code formatting: bash format.sh
Any manual or new tests for this PR (please specify below)
All smoke tests: pytest tests/test_smoke.py
Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

update tf version

…g#2604) * [GCP] Avoid dumping cachetools for backward compatibilty * Add comment

update tpu runtime version for doc

…pilot-org#2606) * Add debug dockerfile * Add docker

…#2610) * Fix aws catalog fetcher for removed offerings * lint

This is a minor update to use `sky status --ip` to simplify the fetching of IP.

…lot-org#2577) fix

* News: Tweak width. * update * tweak * LRU, tweaks * tweaks * updates * updates

* Minor: Update CONTRIBUTING.md * Update CONTRIBUTING.md Co-authored-by: Romil Bhardwaj <[email protected]> --------- Co-authored-by: Romil Bhardwaj <[email protected]>

* add aws, gcp * fix * fix * add IBM message * Update sky/clouds/gcp.py Co-authored-by: Zongheng Yang <[email protected]> * Update sky/clouds/aws.py Co-authored-by: Zongheng Yang <[email protected]> * apply suggestion from code review * add newline * add blank line * add comments for constants --------- Co-authored-by: Zongheng Yang <[email protected]>

* new provisioner with AWS support fix sky status refresh update setup update metadata per-instance logs improve ux update comments update logging reduce retries of provisioning to match the original number fix Ray compat issue update the config with the latest changes sync events with the latest changes sync with upstream resolve conflict * cleanup unused functions * lint * update * fix * fix * Fix network configuration and authentication * Fix creation cluster name * fix logging * format * format * slight improvement for logging * minor fix logging * fix autostop * fix autostop * Fix ray ports used * fix ray start * Fix logging for ray start and skylet * fix ports for ray dashboard * add head / worker in name * More logging information for launching * fix cluster name in progress bar * Fix ssh * format * grammar * Fix message * fix ssh_proxy_commands * fix merge error * Get rid of ray logging * use info * change logger to use contextmanager * format * [major] remove dependency for ray on aws * minor fix * allow warning to be printed * adopt changes * rename * Better spinner * format * update logging * rich_utils in provision_utils * Fix instance log path * Log head node to provision.log * remove unused var * fix ssh method * Fix ray for worker node * Fix ray cluster on worker * fix ray cluster * Fix logging and ip fetching * show cluster name * fix wait after cluster launch * Add logging * Add continue * Change back to sleep before wait * use sleep 1 instead * clean * Avoid check skylet twice * skip docker * remove get_feasible_ips * fix comments * minor fix for logs * minor fix * Fix the ip fetching logic * Add quote for `cluster_name!r` * fix return * fix test * fix comments * fix comments * fix comments * Smoke tests changes to use a private VPC region. * refactor get_ips * minor debug info fix * fix * UX fix * Fix proxy command to be None * Address comments * format * Fix quote * update logging * compatibility for python 3.7 * format * format * change launched to provisioned * clean up ray yaml * format * [Provisioner] Add docker support back (skypilot-org#2507) * Add docker support back * Fix * Works! * format * Add docker back to the supported features * fix docker_cmd * Fix docker cmd * fix DockerLoginConfig * Move docker login config * Fix backward compatibility for docker * Fix docker login * Fix docker login config new line issue * Fix string process in ray yaml * Update sky/provision/docker_utils.py Co-authored-by: Tian Xia <[email protected]> * Update sky/provision/instance_setup.py Co-authored-by: Tian Xia <[email protected]> * Update sky/provision/docker_utils.py Co-authored-by: Tian Xia <[email protected]> * Update sky/provision/instance_setup.py Co-authored-by: Tian Xia <[email protected]> * address comments * format * format * Add comment * Address comments * format --------- Co-authored-by: Tian Xia <[email protected]> * Wording for SSH connection * Fix ray status check for backward compatibility * remnant * stronger backward compatibility tests * remove unused tag * Revert "remove unused tag" This reverts commit a12df8e. * Add default value for `docker_user` * fix botocore config * lint * Fix mypy * remove unused variable * minor fix for logging * expose bootstrap error * format * minor change for `wait_instances` API * lint * use `command_runner.ssh_options_list` * fix ssh command * reword * Dim error for bootstrapping * fix user known file * Update sky/setup_files/setup.py Co-authored-by: Zongheng Yang <[email protected]> * Add ray back, will fix the dependency issue skypilot-org#2625 * move dependency to local ray * Address renaming comments * renamed to provisioner * refactoring for logging to reduce confusion * format * rename back to meta data for metadata utils * format * move provisioner to `provision/` * Do not propagate to provisioner logger * Minor changes * Fix color for error of provisioner * Remove dimmer * Add back missing handler for `provision_logger` * add comments * Print error message for the failed ssh command * Make the error yellow --------- Co-authored-by: Zhanghao Wu <[email protected]> Co-authored-by: Zongheng Yang <[email protected]> Co-authored-by: Tian Xia <[email protected]>

* UX: Fix spot launch hint. * lint

* Add falcon to list of LLMs * Add falcon to list of LLMs

* fix nits * update

…#2625) * new provisioner with AWS support fix sky status refresh update setup update metadata per-instance logs improve ux update comments update logging reduce retries of provisioning to match the original number fix Ray compat issue update the config with the latest changes sync events with the latest changes sync with upstream resolve conflict * cleanup unused functions * lint * update * fix * fix * Fix network configuration and authentication * Fix creation cluster name * fix logging * format * format * slight improvement for logging * minor fix logging * fix autostop * fix autostop * Fix ray ports used * fix ray start * Fix logging for ray start and skylet * fix ports for ray dashboard * add head / worker in name * More logging information for launching * fix cluster name in progress bar * Fix ssh * format * grammar * Fix message * fix ssh_proxy_commands * fix merge error * Get rid of ray logging * use info * change logger to use contextmanager * format * [major] remove dependency for ray on aws * minor fix * allow warning to be printed * adopt changes * rename * Better spinner * format * update logging * rich_utils in provision_utils * Fix instance log path * Log head node to provision.log * remove unused var * fix ssh method * Fix ray for worker node * Fix ray cluster on worker * fix ray cluster * Fix logging and ip fetching * show cluster name * fix wait after cluster launch * Add logging * Add continue * Change back to sleep before wait * use sleep 1 instead * clean * Avoid check skylet twice * skip docker * remove get_feasible_ips * fix comments * minor fix for logs * minor fix * Fix the ip fetching logic * Add quote for `cluster_name!r` * fix return * fix test * fix comments * fix comments * fix comments * Smoke tests changes to use a private VPC region. * refactor get_ips * minor debug info fix * fix * UX fix * Fix proxy command to be None * Address comments * format * Fix quote * update logging * compatibility for python 3.7 * format * format * change launched to provisioned * clean up ray yaml * format * [Provisioner] Add docker support back (skypilot-org#2507) * Add docker support back * Fix * Works! * format * Add docker back to the supported features * fix docker_cmd * Fix docker cmd * fix DockerLoginConfig * Move docker login config * Fix backward compatibility for docker * Fix docker login * Fix docker login config new line issue * Fix string process in ray yaml * Update sky/provision/docker_utils.py Co-authored-by: Tian Xia <[email protected]> * Update sky/provision/instance_setup.py Co-authored-by: Tian Xia <[email protected]> * Update sky/provision/docker_utils.py Co-authored-by: Tian Xia <[email protected]> * Update sky/provision/instance_setup.py Co-authored-by: Tian Xia <[email protected]> * address comments * format * format * Add comment * Address comments * format --------- Co-authored-by: Tian Xia <[email protected]> * Wording for SSH connection * Fix ray status check for backward compatibility * remnant * stronger backward compatibility tests * remove unused tag * Revert "remove unused tag" This reverts commit a12df8e. * Add default value for `docker_user` * fix botocore config * lint * Fix mypy * remove unused variable * minor fix for logging * expose bootstrap error * format * minor change for `wait_instances` API * lint * use `command_runner.ssh_options_list` * fix ssh command * reword * Dim error for bootstrapping * fix user known file * avoid ray dependency for aws * fix setup.py * format * remove unused file * fix setup.py * __init__ for utils * format * format * Address comments * Elaborate readme --------- Co-authored-by: Siyuan <[email protected]> Co-authored-by: Zongheng Yang <[email protected]> Co-authored-by: Tian Xia <[email protected]>

…2642) install typin_extensions for all versions

* README: update news and LLM list. * updates

* fix * apply suggestions from code review * Apply suggestions from code review Co-authored-by: Zhanghao Wu <[email protected]> * lint --------- Co-authored-by: Zhanghao Wu <[email protected]>

…org#2650) * feat(update): Adding CoreWeave label to identify GPUs in K8s * feat(fix): Remove broken line * feat(fix): Fixing label_formatter detection * feat(code): Lint

…sible (skypilot-org#2657) * Fix optimizer for dag when some of the resources provided are invalid * format * format * address comments * better output * spacing * Fix inconsistent repr * use str instead of repr * more robust replace for region, zone

skip v5 catalog

* Add traceback for exceptions raised and detailed reasons for CommandError * format * Add exception column to make exception easier to see * remove color for exception * Fix docstr * revert cli and fix comment

* Initial multi-node support * Add pod anti-affinity * Fix concurrent SSH for Kubernetes * lint * comments * update readme * remove lsof dependency * newline * Update roadmap in readme

* fix * lint * Update sky/cli.py Co-authored-by: Zhanghao Wu <[email protected]> * fix * var name * var name --------- Co-authored-by: Zhanghao Wu <[email protected]>

* UX: Allow infering cloud from region or zone. * format * minor fix * Remove Local cloud from registry. * UX for 1-cloud cases * Format * Fix test fixtures. * isort

* Pin remote pydantic for ray * remote to be the second requirement * Add remote for k8s yml * format * format

fix

…ommands (skypilot-org#2667) * fix * change position

…ypilot-org#2669) * Add retry for flaky error during launching GCP clusters * handle error * format * Do not log out stderr * Add retry for gcloud crash * fix retry return code

* Add sky show-gpus support for Kubernetes * Update sky/clouds/service_catalog/kubernetes_catalog.py Co-authored-by: Romil Bhardwaj <[email protected]> * PR feedback * PR feedback part 2 * Format fix * PR feedback part 3 * Fix bug with checking enabled clouds in k8s list_accelerators * Pylint fixes * Pylint fixes part 2 * Pylint fixes part 3 * Pylint fixes part 4 --------- Co-authored-by: Romil Bhardwaj <[email protected]>

* Fix the NoOp for rich status * fix shell completion installation * pin pandas * revert dependency

* Fix caching error for aws resources * Add todo for local cache * backward compatibility * Use lru cache instead * format * add unit test for aws resources memory leakage * pytest larger memory limit * Fix unit test * fix comment * add requirement for memory profiler * install memory profiler in pytest * fix ci * Update sky/provision/aws/instance.py Co-authored-by: Zongheng Yang <[email protected]> * Address comments * Add new line * backward compatibility * Use thread_local LRU instead * init * Update sky/adaptors/aws.py Co-authored-by: Zongheng Yang <[email protected]> * make private * Add comments * fail early for memory exceeding * Less frequent memory test * shorter period * Update sky/provision/aws/instance.py * refactor * Address comments * fix test * Add comment to modules --------- Co-authored-by: Zongheng Yang <[email protected]>

…org#2594) * refactor: 💡 update faq and add more detailed error message * fix

…ypilot-org#2682) * import cloud provisioners in advance instead of importing it online * Fix lower * format * bump skylet version * Add comment * Add a comment for skylet version

* Add kapa * fix color and logo * height * z-index fix for rtd flyout * newline * newline

infwinston

@jc9123 thanks! the gpu memory are super useful info to the catalog. left a few comments.

infwinston · 2023-10-12T01:01:49Z

sky/clouds/service_catalog/data_fetchers/fetch_gcp.py

+        'V100': 16,
+        'P100': 16,
+        'K80': 12,
+        '': ''


sorry why this ''?

This was a mistake on my end, I will have this removed.

infwinston · 2023-10-12T01:02:12Z

sky/clouds/service_catalog/data_fetchers/fetch_gcp.py

@@ -513,6 +513,23 @@ def get_catalog_df(region_prefix: str) -> pd.DataFrame:
    # Round the prices.
    df['Price'] = df['Price'].round(PRICE_ROUNDING)
    df['SpotPrice'] = df['SpotPrice'].round(PRICE_ROUNDING)
+    gpu_map = {


Could you add a reference link with comment above?

infwinston · 2023-10-12T01:03:03Z

sky/clouds/service_catalog/data_fetchers/fetch_azure.py

+    def create_gpu_map(df):
+        # Map of Azure's machine with GPU to their corresponding memory
+        # Result is hard-coded since Azure's API to not return such info
+        # may be outdated so need to be maintained


Could you add a reference link on how these information are found? also, how did we make sure we cover all the instance types?

I looked through the azure documentation to ensure that all instance type have been included and also ran the script on --all-regions and the result looked fine to me.
However, I think the approach you mentioned below makes more sense in which we map instance type -> gpu name then from gpu name -> gpu memory. There is already a mapping from instance type -> gpu name in the script. Assuming this mapping is complete, we can easily map the gpu name to their corresponding memory.

infwinston · 2023-10-12T01:05:33Z

sky/clouds/service_catalog/data_fetchers/fetch_azure.py

+            'Standard_ND6s': 24,
+            'Standard_ND12s': 48,
+            'Standard_ND24s': 96,
+            'Standard_ND24rs*': 96,


why is there a *?

Its one of the instance type offered by azure.
https://learn.microsoft.com/en-us/azure/virtual-machines/nd-series

infwinston · 2023-10-12T01:08:06Z

sky/clouds/service_catalog/data_fetchers/fetch_azure.py

+        # Map of Azure's machine with GPU to their corresponding memory
+        # Result is hard-coded since Azure's API to not return such info
+        # may be outdated so need to be maintained
+        gpu_map = {


could we try to map instance type -> gpu name first and then calculate the resulting device memory later? this two-level approach might be cleaner.

I agree this approach is much cleaner as it uses much less hard-coding and utilize already fetched info. I will change the script to use this approach.

github-actions · 2024-05-23T01:46:50Z

This PR is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

github-actions · 2024-06-02T01:50:08Z

This PR was closed because it has been stalled for 10 days with no activity.

jc9123 added 10 commits September 16, 2023 21:44

Fixed fetch_azure.py to also fetch Device_Memory

8f08deb

Added device memory to fetch_azure

5acdcb1

fixed formatting

78324fd

fixed more formatting

3d0c20e

fixed more format error

d8c07c8

fixed format

e0e13bc

fixed format

8a21f28

Added device memory to fetch_gcp

f7d3403

fixed device memory conflict

99148c7

updated conflict

b1cb7ab

Michaelvll requested a review from infwinston October 3, 2023 18:31

Michaelvll and others added 19 commits October 11, 2023 00:31

[TPU] Update default runtime version for TPU node (skypilot-org#2601)

ccd750f

update tf version

[GCP] Avoid dumping cachetools for backward compatibilty (skypilot-or…

96036de

…g#2604) * [GCP] Avoid dumping cachetools for backward compatibilty * Add comment

[GCP] Update tpu runtime version for doc (skypilot-org#2602)

7b00f1c

update tpu runtime version for doc

[Docs] Add debugging container instructions to contributing.md (sky…

84609be

…pilot-org#2606) * Add debug dockerfile * Add docker

[Catalog] Fix aws catalog fetcher for removed offerings (skypilot-org…

d232a82

…#2610) * Fix aws catalog fetcher for removed offerings * lint

[LLM] Use --ip CLI to get cluster IPs (skypilot-org#2614)

a97cd62

This is a minor update to use `sky status --ip` to simplify the fetching of IP.

News: Mistral 7B (skypilot-org#2615)

0f29a5f

[Core] Fix version requirements for google-api-python-client (skypi…

f14245a

…lot-org#2577) fix

News: tweak width. (skypilot-org#2616)

077911c

* News: Tweak width. * update * tweak * LRU, tweaks * tweaks * updates * updates

Minor: Update CONTRIBUTING.md (skypilot-org#2624)

e814a30

* Minor: Update CONTRIBUTING.md * Update CONTRIBUTING.md Co-authored-by: Romil Bhardwaj <[email protected]> --------- Co-authored-by: Romil Bhardwaj <[email protected]>

UX: Fix spot launch hint. (skypilot-org#2630)

da33d4f

* UX: Fix spot launch hint. * lint

[Docs] Add falcon to list of LLMs (skypilot-org#2637)

a05cd33

* Add falcon to list of LLMs * Add falcon to list of LLMs

Fix nits (skypilot-org#2633)

202e663

* fix nits * update

[Dependency] install typin_extensions for all versions (skypilot-org#…

d27df87

…2642) install typin_extensions for all versions

README: update news and LLM list. (skypilot-org#2643)

13e1533

* README: update news and LLM list. * updates

[Core] Fix type of job_id when querying job status (skypilot-org#2541)

14fe045

* fix * apply suggestions from code review * Apply suggestions from code review Co-authored-by: Zhanghao Wu <[email protected]> * lint --------- Co-authored-by: Zhanghao Wu <[email protected]>

rtalaricw and others added 23 commits October 11, 2023 00:31

[CoreWeave] Adding CoreWeave label to identify GPUs in K8s (skypilot-…

eb6f490

…org#2650) * feat(update): Adding CoreWeave label to identify GPUs in K8s * feat(fix): Remove broken line * feat(fix): Fixing label_formatter detection * feat(code): Lint

skip v5 catalog fetching (skypilot-org#2656)

3ae9b72

skip v5 catalog

Add more details in exception (skypilot-org#2654)

55b4b2e

* Add traceback for exceptions raised and detailed reasons for CommandError * format * Add exception column to make exception easier to see * remove color for exception * Fix docstr * revert cli and fix comment

[k8s] Multi-node support for Kubernetes (skypilot-org#2609)

dbae488

* Initial multi-node support * Add pod anti-affinity * Fix concurrent SSH for Kubernetes * lint * comments * update readme * remove lsof dependency * newline * Update roadmap in readme

[CLI] Restore sky logs <cluster> * --sync-down (skypilot-org#2660)

f56c2b1

* fix * lint * Update sky/cli.py Co-authored-by: Zhanghao Wu <[email protected]> * fix * var name * var name --------- Co-authored-by: Zhanghao Wu <[email protected]>

UX: Allow inferring cloud from region or zone. (skypilot-org#2632)

65ad5db

* UX: Allow infering cloud from region or zone. * format * minor fix * Remove Local cloud from registry. * UX for 1-cloud cases * Format * Fix test fixtures. * isort

[Core] Pin remote dependency for ray job (skypilot-org#2659)

753585f

* Pin remote pydantic for ray * remote to be the second requirement * Add remote for k8s yml * format * format

[CLI] Fix bugs when specify job_id in sky logs (skypilot-org#2662)

6c9d215

fix

Fix creation of ~/sky_logging/{timestamp} under dir running sky c…

b15d29c

…ommands (skypilot-org#2667) * fix * change position

[GCP] Add retry for transient error during launching GCP clusters (sk…

163e4b5

…ypilot-org#2669) * Add retry for flaky error during launching GCP clusters * handle error * format * Do not log out stderr * Add retry for gcloud crash * fix retry return code

Fix usage by checking detailed_reason (skypilot-org#2672)

c6ee6bc

[Core] Fix the NoOp for rich status (skypilot-org#2678)

92bec55

* Fix the NoOp for rich status * fix shell completion installation * pin pandas * revert dependency

refactor: 💡 update faq and add more detailed error message (skypilot-…

e00a3f9

…org#2594) * refactor: 💡 update faq and add more detailed error message * fix

[Provisioner] Avoid backward compatibility issue with provisioner (sk…

6d3ada5

…ypilot-org#2682) * import cloud provisioners in advance instead of importing it online * Fix lower * format * bump skylet version * Add comment * Add a comment for skylet version

[Docs] Add AI assistant for docs (skypilot-org#2688)

2f43fd7

* Add kapa * fix color and logo * height * z-index fix for rtd flyout * newline * newline

fixed branch

940d864

add device memory

b2732ff

fixed branch

a814e8c

updated branch

8d4e9bd

Merge branch 'skypilot-org:master' into AddDeviceMemory

45f02f3

infwinston reviewed Oct 12, 2023

View reviewed changes

fixed gpu mapping

2d621e2

Michaelvll force-pushed the master branch from 71213e5 to 9743aa0 Compare January 13, 2024 05:30

github-actions bot added the Stale label May 23, 2024

github-actions bot closed this Jun 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add device memory #2565

Add device memory #2565

jc9123 commented Sep 17, 2023 •

edited

Loading

infwinston left a comment

infwinston Oct 12, 2023

jc9123 Oct 13, 2023

infwinston Oct 12, 2023

infwinston Oct 12, 2023

jc9123 Oct 13, 2023

infwinston Oct 12, 2023

jc9123 Oct 13, 2023 •

edited

Loading

infwinston Oct 12, 2023

jc9123 Oct 13, 2023

github-actions bot commented May 23, 2024

github-actions bot commented Jun 2, 2024

Add device memory #2565

Add device memory #2565

Conversation

jc9123 commented Sep 17, 2023 • edited Loading

infwinston left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jc9123 Oct 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented May 23, 2024

github-actions bot commented Jun 2, 2024

jc9123 commented Sep 17, 2023 •

edited

Loading

jc9123 Oct 13, 2023 •

edited

Loading