[BUG] Unable to select default Python env

[twsl]

Describe the bug
I run my project in a docker devcontainer. I have poetry setup already, yet I am not able to select any default Python environment, but need to create a .venv or conda env

To Reproduce
Steps to reproduce the behavior:

1. Set up Python env

Screenshots
If applicable, add screenshots to help explain your problem.

System information:

1. Paste the output ot the Help: About command (CMD-Shift-P).

Version: 1.95.3 (system setup)
Commit: f1a4fb101478ce6ec82fe9627c43efbf9e98c813
Date: 2024-11-13T14:50:04.152Z
Electron: 32.2.1
ElectronBuildId: 10427718
Chromium: 128.0.6613.186
Node.js: 20.18.0
V8: 12.8.374.38-electron.0
OS: Windows_NT x64 10.0.22621

3. Databricks Extension Version
   v2.4.8

Databricks Extension Logs
Please attach the databricks extension logs

Additional context
Add any other context about the problem here.

[ilia-db]

Hi, we utilise python extension API for env selection. Can you select your environment for a python file using the python extension?

Either click here when a python file is open (you might already have something selected there):

Or execute this command: Python: Select Interpreter

Which will lead to this UI:

Do you see your environment in that selection UI?
If not, you still have an option to enter the path to your environment's python executable manually.

Also keep in mind that even if you select the environment, we might still show a warning in our Databricks panel if the python version there doesn't match the DBR requirements of your cluster (usually it's either 3.10 or 3.11).

[twsl]

I'm talking about this env selection

only the newly created venv seems to work

[ilia-db]

Do I understand correctly, that the Poetry venv in the list on your screenshot doesn't work? Or that is a newly created venv? Does existing vent work when you manually enter a path to its python executable? (by clicking on Enter interpreter path)

[twsl]

the venv is the newly created one, when the setup asked me to create one. the rest of the python envs are shown when i clicked on Use Python Extension to setup environments. My issue is, that I cannot select my default Python env provided by the devcontainer/selected in the python.defaultInterpreterPath variable

[ilia-db]

My issue is, that I cannot select my default Python env provided by the devcontainer/selected in the python.defaultInterpreterPath variable

Does python extension sees your default venv? E.g can you open any py or ipynb files and run it using your default environment?

Also, on the screenshot I see that python.defaultInterpreterPath is located in the /usr/local/bin/python - is that the expected path? Does it mean that in the dev container you don't actually have a virtual environment, but use the "global" (to the container) interpreter? In that case we indeed won't "accept" it, as right now we require a virtual environment

[twsl]

Exactly, my devcontainer does not have a venv and uses the global interpreter as the environment separation is provided by the docker container already

[twsl]

Any chance, we can drop the venv requirement? It seems quite arbitrary

[ilia-db]

Sorry, this issue got a bit lost 😬

I can see how it can be limiting. On the other hand we don't want to pollute global python installation with our dependencies - for the majority of the use cases it's the wrong thing to do. databricks-connect versions can be different from project to project, and even within one project you might want to change them depending on what cluster you have connected (and what its DBR version).

In your container definition, can you create a project-scoped venv with access to system packages (python -m venv --system-site-packages path/to/venv)? This way you can setup your global python with anything you want, and the project venv should have access to it.

Or do you have some additional setup with which the venv won't work well (even with system-site-packages enabled)?

[twsl]

No worries.
I have not tested this, will give it a try. This might be a working workaround for now.
I don't use venv as I have no need to do so.
But why don't we keep the venv as default and allow the user to select their own python env based on their own risk assessment and setup. While I can understand you don't want to pollute the global install, in a devcontainer this would cause no harm and only limit a developer.