Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] Fix wheel creation when multiple versions are installed #3866

Merged
merged 4 commits into from
Aug 25, 2024
Merged

[Core] Fix wheel creation when multiple versions are installed #3866

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a2b8371
Fix wheel creation
romilbhardwaj Aug 23, 2024
0867903
Fix wheel creation
romilbhardwaj Aug 23, 2024
611d74a
Refactor _get_latest_wheel_and_remove_all_others
romilbhardwaj Aug 24, 2024
e6be5e6
lint
romilbhardwaj Aug 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions sky/backends/wheel_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,12 +161,19 @@ def _get_latest_modification_time(path: pathlib.Path) -> float:
last_modification_time = _get_latest_modification_time(SKY_PACKAGE_PATH)
last_wheel_modification_time = _get_latest_modification_time(WHEEL_DIR)

# only build wheels if the wheel is outdated
if last_wheel_modification_time < last_modification_time:
# Only build wheels if the wheel is outdated or wheel does not exist
# for the requested version.
if (last_wheel_modification_time < last_modification_time) or not any(
WHEEL_DIR.glob(_WHEEL_PATTERN)):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we instead check WHEEL_DIR.glob(f'**/{_WHEEL_PATTERN}') instead, similar to https://github.com/skypilot-org/skypilot/blob/master/sky/backends/wheel_utils.py#L43?

Also, since we are checking the whether the current wheel exists, it might be fine to just use the current wheel for the later latest_wheel, i.e. it can become _get_current_wheel_and_remove_older_others?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah good idea. Refactored _get_latest_wheel_and_remove_all_others into separate methods to get the latest wheel and remove others.

if not WHEEL_DIR.exists():
WHEEL_DIR.mkdir(parents=True, exist_ok=True)
_build_sky_wheel()

# We remove all wheels except the latest one for garbage collection.
# Otherwise stale wheels will accumulate over time.
# TODO(romilb): If the user switches versions every alternate launch,
# the wheel will be rebuilt every time. At the risk of adding
# complexity, we can consider TTL caching wheels by version here.
latest_wheel = _get_latest_wheel_and_remove_all_others()

wheel_hash = latest_wheel.parent.name
Expand Down
Loading