pex packed binaries hanging forever at __del()__ from MktempTeardownRegistry #2641
Comments
Addressing your hang debugging 1st:The hang you observe is in a destructor that grabs no locks. It only calls:
So there is 0 Pex code in that call graph to be causing the hang. In your particular case, the dict in step 2 shows as empty; so there is not even any As such, I'm going to throw away your hang debug info - it does not appear to point usefully at anything. Pex debugging:One way to solve this or at least narrow scope to things other than Pex is to get Pex out of the picture as early as possible. If the docker image for CI is prepared by you with the packed PEX in it, you can: Then, instead of running If, instead, the packed PEX is created while running in the docker image in CI, you should create the Pex with: The venv options cause the PEX to install itself in as traditional a venv as possible on 1st run. The In either case, when running in @gebi let me know if any of that helps your debug journey. |
|
sry for the very long mail, after reading it again it sounded much harsher than intended... seems i just concentrated way too much on getting all the little details correct. many thx for your help and debugging suggestions, we'll debug further and update the issuere here if we find anything usefull. |
I didn't read it that way at all. You provided alot of facts. The more the better. The more you leave out the longer the back and forth takes. Ideally you dump all relevant state from your end so I can "see" what you see. The devil in that detail is always knowing what's relevant. Here, I think the only relevant bits you left out were:
|
|
@gebi any more data to report? |
|
@gebi I'm going to close this since there's been no further data reported. Please do re-open though if you do find more data for me to look at. |
Hi,
We wanted to start using pex for our binaries but the packed binaries are hanging on
MktempTeardownRegistry:__del()__ofte enough that none of our ci pipeline runs finish successfully.The CI builds are executed in docker containers where pex binaries are hanging sometimes.
Sadly we did not manage to reproduce it locally or in a smaller scope.
There are no concurrent pex invocations or other pex binaries running inside the container.
Binaries are created with
pex --include-tools --sh-boot --disable-cache ...but also tried without--sh-bootand without--disable-cachebut did not change behaviour.We also tried various env variables like
PEX_ROOT=$(mktemp -d)... did not helpPEX_VENV=1... did not helpPEX_MAX_INSTALL_JOBS=1... did not help_PEX_FILE_LOCK_STYLE=bsd... did not helpPEX_VERBOSE=9... still hangs, output belowAny ideas on how to debug this further?
If necessary we could execute debugging things rather quickly to help find the issue, but for the issue itself we are unsure on how to debug this any further.
environment
Debian 11 bullseye
python 3.9.2 from debian
pex packed binary executed inside docker 27.5.0
logs / outputs
lslocks just shows
access.lcklocked by the hanging pex binaryopen files from pex dir
debugging the hanging binary we got the following informations
pex verbose 9 output
requirements.txt pex is executed with
many thx for reading this till the end :) !
The text was updated successfully, but these errors were encountered: