Stubs on Windows exit abnormally, includes succinct repro with busybox, related to some wider issue

[doctorpangloss]

related: rmyorston/busybox-w32#451
related: #6464

wget https://frippery.org/files/busybox/prerelease/busybox_pre64.exe
busybox_pre64.exe sh -X
mkdir uv-test
cd uv-test
uv venv --seed
source .venv/scripts/activate
# uses stub
pip install insightface

Observe pip exits abnormally under this shell. In Powershell it works fine.

uv 0.3.4 (39f3cd2 2024-08-26)
Windows 2022

All stubs, not just pip, are cursed this way. The stubs created by native pip do not have any issues.

[rmyorston]

I can see the problem, but don't have much idea why it occurs.

What I've done:

• Install Python on Windows using an installer from python.org.
• From PowerShell run pip install uv.
• From a busybox-w32 shell do the whole mkdir uv-test; ... ; pip install insightface thing. As promised, it doesn't work.

The question is why not. I was suspicious of the changes the busybox-w32 shell makes to environment variables when it reads them in as shell variables. By default it converts the variable names to uppercase and replaces backslashes with forward slashes in the value. (There are a couple of exceptions to the latter, it seemed possible more might be needed.)

I made a custom shell so those changes could be avoided. It remains necessary to change Path to PATH, or the shell won't find any external commands. pip install insightface still fails with a segfault. I do note, though, that if the failing process is stopped with an error dialog the installation seems to carry on in the background until the dialog is closed.

[zanieb]

Thanks for the report. I presume this is a bug in our "trampoline" that's used for executables on Windows. I don't see an error message in the video, what's the failure mode exactly?

[samypr100]

By default it converts the variable names to uppercase and replaces backslashes with forward slashes in the value.

Does this behave as if it was a MinGW based environment? There's been a number of MinGW related issues, especially when it comes to paths. e.g. #3573

[rmyorston]

@zanieb In my experience the failure mode is a 0xc0000005 error, which I (possibly wrongly) equate to a segfault.

[rmyorston]

@samypr100 The issue you reference seems to be somewhat confused about what MinGW means. (And I'm not sure I can be lucid enough to clear up the confusion.)

MinGW (and these days that generally means MinGW-w64) is a set of headers and libraries which make it possible to build native Windows applications.

There are various ways that can be done:

• I cross-compile on Linux.
• There's the Windows-native w64devkit toolchain.
• MSYS2 is an alternative toolchain for Windows.

busybox-w32 is built using MinGW, so it's a native Windows binary which deals in native Windows paths.

This can be opposed to binaries built using the Cygwin DLL, such as those that come with MSYS2: these see fake POSIX paths.

When the referenced issue talks of 'mingw python' it seems to mean the Python that's shipped with MSYS2. This is built using the Cygwin DLL and understands non-Windows paths. I wouldn't call that a 'MinGW Python'. Furthermore, I wouldn't be at all surprised to hear there have been issues with it's expectations regarding paths.

[samypr100]

@rmyorston Thanks for the explanation. I've made an adjustment in #6792 that would fix this issue, or at least not fault. I haven't digged into the busybox impl, but lpReserved2 reports multiple file descriptors which are invalid handles which led to the issue manifesting. For now, I've switched the implementation to ignore any handle that seems to be invalid.

uv/crates/uv-trampoline/src/bounce.rs

Lines 347 to 349 in 3be4fe5

	let crt_magic = si.lpReserved2 as *const u32;
	let handle_count = unsafe { crt_magic.read_unaligned() } as isize;
	let handle_start = unsafe { crt_magic.offset(1 + handle_count) };

[rmyorston]

@samypr100 The fix sounds promising, thanks.

The shell in busybox-w32 uses spawnve() from the Microsoft C runtime to invoke external commands. What happens after that is up to Microsoft.

It'll be interesting to know if the fix also resolves #6464.

[doctorpangloss]

Is it expected to see

Failed to close child file descriptors at 1

when invoking the stub from within sh onto stderr? otherwise things appear to be behaving normally

[samypr100]

@doctorpangloss Would you be able to confirm 0.4.5 fixes the fd issue for you?

[doctorpangloss]

$ pip install -U uv
Collecting uv
  Obtaining dependency information for uv from https://files.pythonhosted.org/packages/06/f2/0278155d4b2726765bcad89e5bc6020a1afb52d2bfe8cf1602758e44a93f/uv-0.4.8-py3-none-win_amd64.whl.metadata
  Downloading uv-0.4.8-py3-none-win_amd64.whl.metadata (11 kB)
Downloading uv-0.4.8-py3-none-win_amd64.whl (12.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.7/12.7 MB 17.2 MB/s eta 0:00:00
Installing collected packages: uv
Successfully installed uv-0.4.8
WARNING: There was an error checking the latest version of pip.
(.venv) ~/Documents/appmana $ cd appmana-comfyui/src/
(.venv) ~/Documents/appmana/appmana-comfyui/src $ uv pip install --no-deps -e .
Resolved 1 package in 18ms
Installed 1 package in 14ms
 + comfyui==0.0.1 (from file:///C:/Users/bberman/Documents/appmana/appmana-comfyui/src)
(.venv) ~/Documents/appmana/appmana-comfyui/src $ comfyui -w ../../appmana-artworkspace/src/workdir/
Failed to close child file descriptors at 1

no, unfortunately the stubs still show the issue, they do work though

[samypr100]

Could you provide steps on how to reproduce this particular case?

[doctorpangloss]

nervermind it's resolved! 0.4.8 has fixed it