PR: User environment variables to take precedence for IPython Console kernel

[mrclary]

Description of Changes

Ensure that user environment variables supersede Spyder runtime environment variables when starting the IPython Console kernel.

Issue(s) Resolved

Fixes #23711

[mrclary]

@spyder-ide/core-developers,
So I've run into an issue on this PR for Windows. The IPython console tests fail because a user-defined PATH variable clobbers the system-defined PATH variable. I see the same symptoms as #23582, so it may be related.

I'd like your feedback because I think we should clarify what get_user_environment_variables should do on all platforms.

Currently, get_user_environment_variables retrieves both "system" and "user" variables on posix systems, but only "user" variables on Windows. "User" variables will clobber "system" variables on all platforms except for PATH and PYTHONPATH on Windows. These seem to be an exception in which "system" and "user" values are concatenated. This is a problem when these are defined in the "system" variables and when using winreg to get user variables from HKEY_CURRENT_USER since the "system" values will not be retrieved.

In the specific case of this PR and possibly #23582, if the PATH environment variable sent to kernelspec is missing critical system paths for whatever reason, then the 'chcp' is not recognized as an internal or external command error will certainly result.

So should we also retrieve the environment variables from HKEY_LOCAL_MACHINE and concatenate the PATH and PYTHONPATH variable values? Note this would have implications for set_user_env.

[mrclary]

Also, should we consider using a subprocess script like we do for posix? Getting user (or system) variables from the registry does not resolve variable references, e.g. %SystemRoot%\system32 or %HOMEPATH%\some\directory.

[mrclary]

Should Spyder get up-to-date environment variables? or should we require Spyder to restart to do this? The latter is much simpler but what do other IDEs do?

[ccordoba12]

So should we also retrieve the environment variables from HKEY_LOCAL_MACHINE and concatenate the PATH and PYTHONPATH variable values?

Yep, I think that's what we should do on Windows.

Note this would have implications for set_user_env.

What are the implications?

Also, should we consider using a subprocess script like we do for posix? Getting user (or system) variables from the registry does not resolve variable references, e.g. %SystemRoot%\system32 or %HOMEPATH%\some\directory.

Using a script would make things more complicated on Windows. Instead, we could replace %SystemRoot%, etc, by their corresponding values (which should be retrieved when getting env vars anyway).

Should Spyder get up-to-date environment variables? or should we require Spyder to restart to do this? The latter is much simpler but what do other IDEs do?

I think a restart shouldn't be required but I don't know what other IDEs do.

[mrclary]

Note this would have implications for set_user_env.

What are the implications?

Currently, the (Windows only) set_user_env adds all environment variables to HKEY_CURRENT_USER. This is okay if we only retrieve from HKEY_CURRENT_USER, but if we also retrieve from HKEY_LOCAL_MACHINE, then set_user_env would effectively duplicate all of HKEY_LOCAL_MACHINE into HKEY_CURRENT_USER. We would have to consider complicating the environment variable handling by keeping the two sources separate.

Also, should we consider using a subprocess script like we do for posix? Getting user (or system) variables from the registry does not resolve variable references, e.g. %SystemRoot%\system32 or %HOMEPATH%\some\directory.

Using a script would make things more complicated on Windows. Instead, we could replace %SystemRoot%, etc, by their corresponding values (which should be retrieved when getting env vars anyway).

I think this is also preferable to a script.

Should Spyder get up-to-date environment variables? or should we require Spyder to restart to do this? The latter is much simpler but what do other IDEs do?

I think a restart shouldn't be required but I don't know what other IDEs do.

For context, our current setup only retrieves up-to-date environment variables

• on Windows (via the registry), and only the "user" variables, not "system" variables
• on posix launched via shortcut (i.e. not launched_from_terminal), otherwise reverts to os.environ (because of issue Spyder 6.0 hangs at startup on Linux (proc.communicate never times out) #22415)

This is why I bring up the idea of not getting up-to-date environment variables on-demand. Then we could rely on os.environ for all scenarios except macOS shortcuts, but in this case just retrieve variables once at startup and cache it.

[ccordoba12]

We would have to consider complicating the environment variable handling by keeping the two sources separate.

Ok, I see. Then set_user_env needs to remain the same, i.e. it only needs to set user declared env vars. That's because on Windows it's very easy to break your installation by changing env vars. And not everyone is admin so they can set system vars.

I think this is also preferable to a script.

Agreed.

This is why I bring up the idea of not getting up-to-date environment variables on-demand. Then we could rely on os.environ for all scenarios except macOS shortcuts, but in this case just retrieve variables once at startup and cache it.

That depends on users' expectations. As a user, I'd expect that changes to PATH, for instance, are picked up by Spyder without a restart. Otherwise, doing several adjustments to that variable (e.g. when trying to run Cuda code) would be very cumbersome.