[Core] More robust conda setup when python 3.12 is in the image

[Michaelvll]

Not urgent, but this makes the python path for the case where python 3.12 is the default python in the custom image.

Performance tests

On default image

Ran for 5 times

Master

===> multitime results
1: sky launch -y --cpus 2 --cloud aws
            Mean        Std.Dev.    Min         Median      Max
real        79.540      29.046      59.366      63.620      136.207     
user        3.890       0.392       3.574       3.690       4.646       
sys         0.178       0.026       0.153       0.162       0.219 

Current PR

1: sky launch -y --cpus 2 --cloud aws
            Mean        Std.Dev.    Min         Median      Max
real        62.031      2.946       56.946      63.365      65.049      
user        3.968       0.155       3.774       3.964       4.235       
sys         0.313       0.073       0.236       0.290       0.442 

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  • Test on an image with python 3.12 as default python version and check performance difference
    sky launch --image-id docker:ubuntu --cpus 2 --cloud aws
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

[Michaelvll]

This is the main place that we get the correct path for python3

[cblmemo]

Thanks for fixing this @Michaelvll ! A quick question - do we need to change the python path file to $PYTHON_EXEC as well?

[cg505]

Cool, I didn't know uv venv had this feature! Maybe this is too ambitious, but do we need conda at all if we can do this?

[Michaelvll]

Cool, I didn't know uv venv had this feature! Maybe this is too ambitious, but do we need conda at all if we can do this?

Yep, we should gradually get rid of conda. As we have move off the dependency of conda for our internal dependencies, the main purpose of conda now is just to make sure that a user gets the same python env when they start machines on any cloud and can do conda commands.

The next step would be:
Completely move to uv, add uv as the recommended way to install packages in the machine in SkyPilot doc, and add a flag to install conda if people need it.

[cg505]

Sounds good, ty. We should do smoke tests and the testing plan you mention in the PR description.

[Michaelvll]

A user said this PR resolves their issue with a image with python 3.12 as the base python

[Michaelvll]

There seems no significant performance difference. Merging this PR after the smoke tests passed.

[Michaelvll]

/smoke-test --aws

[Michaelvll]

/quicktest-core

[Michaelvll]

/smoke-test --aws -k test_job_queue_with_docker

[Michaelvll]

/smoke-test --aws -k test_managed_jobs_pipeline_recovery_aws

[Michaelvll]

/smoke-test --aws -k test_managed_jobs_retry_logs