Moar threadsafe moar better #101

JamesWrigley · 2024-05-23T15:54:55Z

This is a rebased version of #4, it should be ready to merge. Fixes #73.

(made after discussing with @jpsamaroo)

JamesWrigley · 2024-05-24T12:25:09Z

I tracked the test failures down to: JuliaLang/julia#53326
The workers are loading the builtin version of Distributed from the sysimg while the master is using the development version, and because we changed the definition of the Worker struct in this PR we get a serialization error when doing a remotecall to another worker because the master and worker are using different types.

I think this should be fixed in Base, made a PR here: JuliaLang/julia#54571
That should be merged before this one.

codecov · 2024-05-25T11:06:38Z

Codecov Report

Attention: Patch coverage is 84.74576% with 9 lines in your changes missing coverage. Please review.

Project coverage is 79.50%. Comparing base (3b9e4fd) to head (90041ca).
Report is 9 commits behind head on master.

Files with missing lines	Patch %	Lines
src/cluster.jl	86.95%	6 Missing ⚠️
src/managers.jl	50.00%	1 Missing ⚠️
src/messages.jl	0.00%	1 Missing ⚠️
src/process_messages.jl	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #101      +/-   ##
==========================================
+ Coverage   79.24%   79.50%   +0.25%     
==========================================
  Files          10       10              
  Lines        1913     1922       +9     
==========================================
+ Hits         1516     1528      +12     
+ Misses        397      394       -3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

JamesWrigley · 2024-05-25T11:17:06Z

Alrighty, after a force push to trigger CI with latest nightly we are back in the green 🥳 I think this is ready to be merged now.

jonas-schulze

I can confirm that the original code snippet from #73 (comment) is working now. 🚀

Will this be compatible to or backported, if necessary, to the next Julia LTS?

jonas-schulze · 2024-05-30T13:05:56Z

src/cluster.jl

    @async manage(w.manager, w.id, w.config, :register)
    # wait for rr_ntfy_join with timeout
    timedout = false
-    @async (sleep($timeout); timedout = true; put!(rr_ntfy_join, 1))
+    @async begin
+        sleep($timeout)
+        timedout = true
+        put!(rr_ntfy_join, 1)
+    end


Do these tasks need an errormonitor?

Yeah I think that makes sense, added in 03d7384.

jonas-schulze · 2024-05-30T13:08:08Z

test/threads.jl

+    ws = ts = product(1:2, 1:2)
+    @testset "from worker $w1 to $w2 via 1" for (w1, w2) in ws
+        @testset "from thread $w1.$t1 to $w2.$t2" for (t1, t2) in ts
+          # We want (the default) lazyness, so that we wait for `Worker.c_state`!


Suggested change

# We want (the default) lazyness, so that we wait for `Worker.c_state`!

# We want (the default) laziness, so that we wait for `Worker.c_state`!

Fixed in f4576aa.

jonas-schulze · 2024-05-30T13:12:29Z

test/threads.jl

+            end
+
+            # Wait on the spawned tasks on the owner
+            @sync begin


IMO, this sync point should fail fast, if necessary:

Suggested change

@sync begin

Base.Experimental.@sync begin

See JuliaLang/julia#42239 (comment)

Sure that makes sense, I refactored the code to use timedwait() in f4576aa.

JamesWrigley · 2024-06-02T17:58:02Z

Will this be compatible to or backported, if necessary, to the next Julia LTS?

I'll leave that for someone more qualified to properly answer, but FWIW if 1.11 is chosen as the next LTS then it'll be possible to upgrade Distributed now that it's an excised stdlib 🐙

JamesWrigley · 2024-06-03T09:21:26Z

One other thing I noticed is that this should probably be using Threads.@spawn everywhere instead of @async, but I'll leave that for another PR.

jonas-schulze

Are there any updates on this?

JamesWrigley · 2024-06-24T14:09:39Z

I believe @JBlaschke was going to have a look at it. In the meantime I see the branch is out of date, so I'll rebase it.

JBlaschke · 2024-06-24T15:09:02Z

@JamesWrigley @jonas-schulze I'll be working on this this week. Just getting back up to speed after long travel...

zsz00 · 2024-07-21T03:39:20Z

Are there any updates on this? @JamesWrigley @JBlaschke

JamesWrigley · 2024-07-21T09:34:36Z

Not from me, still need someone to review it. I believe the hesitation to merge comes from Distributed being used to run the Julia tests, so it's quite critical that this works properly.

But in the meantime you can ] dev https://github.com/JamesWrigley/Distributed.jl.git#jps/threadsafe_workerstate on 1.11 to use this branch.

gbaraldi · 2024-07-23T18:37:41Z

src/cluster.jl

-    c_state::Condition      # wait for state changes
-    ct_time::Float64        # creation time
-    conn_func::Any          # used to setup connections lazily
+    @atomic state::WorkerState


if state is always read/written from inside a lock this doesn't need to be atomic as the lock should have the correct barriers.

I don't think that's guaranteed? From a cursory grep through cluster.jl I see plenty of reads outside of a lock.

gbaraldi · 2024-07-23T18:39:23Z

Since we are making things threadsafe I would look at all @async uses because we want to deprecate it.

JamesWrigley · 2024-07-23T19:28:32Z

Ok, I replaced all uses of @async with Threads.@spawn in 7cac33f. But there are still a couple places where Base.@async_unwrap is used and I can't find an easy analogue for Threads.@spawn, I'll see if I can write a replacement.

JamesWrigley · 2024-07-23T20:33:12Z

After thinking about it for a bit, I can't come up with a decent replacement short of basically reimplementing Threads.@spawn. We could replace @async_unwrap with Threads.@spawn, but it would change the type of any exceptions thrown from whatever they currently are to a TaskFailedException, which is technically breaking.

I'd suggest keeping it in for now, we can add support for unwrapping exceptions to Threads.@spawn and switch in a future release.

jonas-schulze · 2024-07-25T08:20:05Z

test/threads.jl

+            # Wait on the spawned tasks on the owner. Note that we use
+            # timedwait() instead of @sync to avoid deadlocks.
+            t1 = Threads.@spawn fetch_from_owner(wait, recv)
+            t2 = Threads.@spawn fetch_from_owner(wait, send)
+            @test timedwait(() -> istaskdone(t1), 5) == :ok
+            @test timedwait(() -> istaskdone(t2), 5) == :ok


Sorry to chime in so late after #101 (comment); I noticed because GitHub unfolded all my previous comments.

I like the timedwait, which is what I used in JuliaLang/julia#37905. However, the timedwait has been the main reason (I think) why my first PR was reverted (JuliaLang/julia#38112). The second attempt (https://github.com/JuliaLang/julia/pull/38134/files) didn't use timedwait. I remain in favor of timedwait but wanted to refresh the information, as it has been a while.

I see, thanks. My view is that if this fails we're going to end up with a timeout somewhere no matter what, either in CI or the tests themselves. And my preference would be to have the timeout in the tests so we can have some control over it. I bumped it to 60s in abafb79 but I'm happy to increase that if people think it's too low. @vchuravy, @vtjnash, does that sound ok?

JamesWrigley · 2024-08-28T08:27:12Z

Wee progress update for those following this, after some discussion with @vchuravy on Slack I decided to:

Try to run the Julia tests in the Distributed.jl CI since that's what was failing last time.
Attempt to audit the code for correctness.

Which I'll get to at... some point.

JamesWrigley · 2025-01-20T13:20:29Z

Wee update on this, the Julia tests 'failing' last time means they were hanging. In DistributedNext I've fixed a couple of hangs related to lingering tasks, perhaps that will solve the hangs.

About the code audit, if someone from the core team commits to reviewing this then I can try to do the audit before the 1.12 release if that's desired.

IanButterworth · 2025-01-20T13:26:20Z

I'm happy to review (in addition to the core devs)

IanButterworth · 2025-01-20T13:40:38Z

Seems like there's a small conflict

JamesWrigley · 2025-01-20T13:55:03Z

(should be fixed now)

IanButterworth · 2025-01-20T18:19:42Z

src/cluster.jl

@@ -161,15 +163,16 @@ function check_worker_state(w::Worker)
        else
            w.ct_time = time()
            if myid() > w.id
-                t = @async exec_conn_func(w)
+                t = Threads.@spawn exec_conn_func(w)


I think given the tasks created by Distributed are generally orchestration and not work, I think they should stay on the same threadpool as the caller, which is likely thread 1, and may be :interactive so I think these all should be

Suggested change

t = Threads.@spawn exec_conn_func(w)

t = Threads.@spawn Threads.threadpool() exec_conn_func(w)

for which I've proposed shorthand of @spawn :same here JuliaLang/julia#57109

True, but then why ensure that they stay on the same threadpool as the caller if they're generally lightweight orchestration tasks?

I guess because the worker threads could be tied up doing blocking stuff.

Ok, I set most tasks to use Threads.threadpool() in 7f1d879. Some I didn't because they're intended to be executed on a worker, which I feel should 'own' the worker process and not have to be sensitive to any non-Distributed code.

src/cluster.jl

IanButterworth · 2025-01-20T20:46:11Z

Apart from my suggestions above, this seems like it's a big improvement, and given it's lingered a bit waiting for reviews, it may be reasonable to just go ahead and fix any issues in the 1.12 pre-release process

JamesWrigley · 2025-01-21T18:49:27Z

I agree :) And as I said almost a year ago when I first started pestering people about it, I'm willing to debug any issues in Julia CI that arise.

JamesWrigley · 2025-01-21T18:51:11Z

If this is merged I will also upstream another hang fix: JuliaParallel/DistributedNext.jl#17 (comment)

(or upstream it eventually anyway when that PR is merged)

IanButterworth · 2025-01-21T19:26:10Z

Btw, should we rename this PR to "Thread safety improvements v2"?

JamesWrigley · 2025-01-21T19:57:28Z

What do you mean the title makes perfect sense 😁

(I kid, feel free to change it if you like)

This is to avoid them accidentally running in another (potentially busy) threadpool.

IanButterworth

I've proposed that we merge this, and @JamesWrigley mentioned a couple of other fixes from DistributedNext that can be added imminently, then bumped in time for 1.12 testing.

JamesWrigley mentioned this pull request May 23, 2024

Thread safety improvements #4

Closed

JamesWrigley force-pushed the jps/threadsafe_workerstate branch from 9a8469d to ec8bce0 Compare May 25, 2024 10:57

jonas-schulze reviewed May 30, 2024

View reviewed changes

jonas-schulze approved these changes Jun 24, 2024

View reviewed changes

JamesWrigley force-pushed the jps/threadsafe_workerstate branch 2 times, most recently from e3205d8 to fa9d645 Compare June 24, 2024 14:11

JamesWrigley force-pushed the jps/threadsafe_workerstate branch from fa9d645 to b140754 Compare July 21, 2024 09:36

gbaraldi reviewed Jul 23, 2024

View reviewed changes

JamesWrigley force-pushed the jps/threadsafe_workerstate branch from 7cac33f to 01c33e9 Compare July 25, 2024 07:51

jonas-schulze reviewed Jul 25, 2024

View reviewed changes

JamesWrigley mentioned this pull request Oct 29, 2024

Such threadsafe, much wow JuliaParallel/DistributedNext.jl#4

Merged

m-fila mentioned this pull request Oct 30, 2024

use fork of distributed with fix for thread-safety key4hep/key4hep-julia-fwk#61

Closed

jpsamaroo mentioned this pull request Dec 2, 2024

Start julia with multiple threads by default JuliaLang/julia#43672

Open

JamesWrigley mentioned this pull request Jan 20, 2025

Make excluded threadunsafe tests threadsafe #121

Open

Add gitignore

1c16bc6

JamesWrigley force-pushed the jps/threadsafe_workerstate branch from abafb79 to 612e3f2 Compare January 20, 2025 13:48

IanButterworth reviewed Jan 20, 2025

View reviewed changes

src/cluster.jl Outdated Show resolved Hide resolved

vchuravy and others added 4 commits January 21, 2025 22:59

Make worker state variable threadsafe

47c9f3e

use atomic field

1b532c1

init_multi: Be more thread-safe

d7f6faf

Use errormonitor() in a few places

96e0294

JamesWrigley force-pushed the jps/threadsafe_workerstate branch from 7f1d879 to 50bdaeb Compare January 21, 2025 22:01

JamesWrigley added 3 commits January 21, 2025 18:54

Replace @async with Threads.@spawn

a460f9f

Use timedwait() in check_master_connect()

56329d5

Keep most 'client oriented' tasks in the same threadpool

90041ca

This is to avoid them accidentally running in another (potentially busy) threadpool.

IanButterworth force-pushed the jps/threadsafe_workerstate branch from 50bdaeb to 90041ca Compare January 21, 2025 23:55

IanButterworth approved these changes Jan 21, 2025

View reviewed changes

IanButterworth reviewed Jan 22, 2025

View reviewed changes

IanButterworth merged commit 0e27cd1 into JuliaLang:master Jan 22, 2025
8 checks passed

JamesWrigley deleted the jps/threadsafe_workerstate branch January 22, 2025 12:42

JamesWrigley mentioned this pull request Jan 22, 2025

Thread-safety improvements #122

Open

	# We want (the default) lazyness, so that we wait for `Worker.c_state`!
	# We want (the default) laziness, so that we wait for `Worker.c_state`!

	t = Threads.@spawn exec_conn_func(w)
	t = Threads.@spawn Threads.threadpool() exec_conn_func(w)

Moar threadsafe moar better #101

Moar threadsafe moar better #101

Conversation

JamesWrigley commented May 23, 2024

JamesWrigley commented May 24, 2024

codecov bot commented May 25, 2024 • edited Loading

Codecov Report

JamesWrigley commented May 25, 2024

jonas-schulze left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JamesWrigley commented Jun 2, 2024

JamesWrigley commented Jun 3, 2024

jonas-schulze left a comment

Choose a reason for hiding this comment

JamesWrigley commented Jun 24, 2024

JBlaschke commented Jun 24, 2024

zsz00 commented Jul 21, 2024

JamesWrigley commented Jul 21, 2024 • edited Loading

gbaraldi Jul 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gbaraldi commented Jul 23, 2024

JamesWrigley commented Jul 23, 2024

JamesWrigley commented Jul 23, 2024

Choose a reason for hiding this comment

JamesWrigley Jul 25, 2024 • edited Loading

Choose a reason for hiding this comment

JamesWrigley commented Aug 28, 2024

JamesWrigley commented Jan 20, 2025

IanButterworth commented Jan 20, 2025

IanButterworth commented Jan 20, 2025

JamesWrigley commented Jan 20, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

IanButterworth commented Jan 20, 2025

JamesWrigley commented Jan 21, 2025 • edited Loading

JamesWrigley commented Jan 21, 2025

IanButterworth commented Jan 21, 2025

JamesWrigley commented Jan 21, 2025

IanButterworth left a comment

Choose a reason for hiding this comment

codecov bot commented May 25, 2024 •

edited

Loading

JamesWrigley commented Jul 21, 2024 •

edited

Loading

gbaraldi Jul 23, 2024 •

edited

Loading

JamesWrigley Jul 25, 2024 •

edited

Loading

JamesWrigley commented Jan 21, 2025 •

edited

Loading