Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT MERGE] Debug leaked semaphore #6578

Closed
wants to merge 24 commits into from
Closed

[DO NOT MERGE] Debug leaked semaphore #6578

wants to merge 24 commits into from

Conversation

tohtana
Copy link
Contributor

@tohtana tohtana commented Sep 27, 2024

Our CI tests frequently get stuck after showing the error below. You can find an example at here.

/opt/conda/envs/ptca/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

As this doesn't happen on my local env, I added logging to see what happens on the CI env.

tohtana and others added 23 commits September 27, 2024 07:16
@tohtana
add logging of multiprocessing
d094b3a
@tohtana
remove unrelated workflow yamls
49afba4
@tohtana
ZZRevert "remove unrelated workflow yamls"
5545737 
This reverts commit 49afba4.
@tohtana
display mem usage
a40d694
@tohtana
Merge branch 'master' into tohtana/debug_semaphore_leak
541bd99
@tohtana
remove grep to see errors
709cccc
@tohtana
reduce tests
bba07fc
@tohtana
update cuda ver
08b9230
@tohtana
fix printing device name
73f1716
@tohtana
add all tests
a0270a5
@tohtana
add pynvml to requirements
afc254b
@tohtana
show process count
cac28a2
@tohtana
remove fork
6d8e8e3
@tohtana
cover non dist cases
edcf622
@tohtana
fix print
8b03ef5
@tohtana
ensure exclusive allocation of master port
68f7285
@tohtana
show available memory
62989df
@tohtana
set lock file path via env var
1977569
@tohtana
fix format
da1cd32
@tohtana
fix workflow to display mem
f4aeaa9
@tohtana
revert workflow
a41863b
@tohtana
Merge branch 'tohtana/allocate_test_port' into tohtana/debug_semaphor…
e096310 
…e_leak
@tohtana
set reuse dist env
4877418
@tohtana tohtana mentioned this pull request Oct 12, 2024
Merged
github-merge-queue bot pushed a commit that referenced this pull request Oct 14, 2024
@tohtana
Ignore reuse_dist_env (#6623)
7a5bc4f 
Tests with `reuse_dist_env = True` often causes memory leaks. This PR
ignores `reuse_dist_env` and forcibly sets it to `False`. This change
might slow down the tests, but I think it is better to manually restart
runners and relaunch tests.

Memory usages (See #6578):
- `reuse_dist_env == True`:
https://github.com/microsoft/DeepSpeed/actions/runs/11302940871/job/31439471512
- `reuse_dist_env == False`:
https://github.com/microsoft/DeepSpeed/actions/runs/11303250613/job/31440137894
@tohtana
Copy link
Contributor Author

tohtana commented Oct 15, 2024

I think the cause of the memory issue in tests has been identified as reuse_dist_env and it was disabled by #6623. We still see tests sometimes stuck and opened #6627 to track it down.

@tohtana tohtana closed this Oct 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tjruwase tjruwase Awaiting requested review from tjruwase tjruwase will be requested when the pull request is marked ready for review tjruwase is a code owner

@loadams loadams Awaiting requested review from loadams loadams will be requested when the pull request is marked ready for review loadams is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@tohtana