syncRoot barrier does not block for unfulfilled pledges

[mratsim]

syncRoot does not take into account unfulfilled pledges.

This is related to workers being able to send termination even though they have a task delayed by a pledge.
This can be avoided by workers keeping a local count of the delayed tasks they have pending.

See: https://dev.azure.com/numforge/Weave/_build/results?buildId=288&view=logs&j=6b0d97ed-2246-525a-6e2b-9532bade852d&t=90276eb0-5b44-500a-95e7-4e793047d715

[mratsim]

In particular in this finite state automaton, a new event + transition or changing the MyTreeIsIdle event to also check for delayed tasks should be sufficient:

weave/weave/stealing_fsm.nim

Lines 107 to 136 in 3bfb641

	# Decline your own steal request
	# -------------------------------------------
	onEntry(declineReqFSA, DS_TreeIdle):
	# debug:
	# log("Worker %2d: received own request (req.state: %s, left (%d): %s, right (%d): %s)\n",
	# myID(), $req.state,
	# myWorker().left,
	# if myWorker().leftIsWaiting: "waiting" else: "not waiting",
	# myWorker().right,
	# if myWorker().rightIsWaiting: "waiting" else: "not waiting")
	discard
	implEvent(declineReqFSA, DSE_MyTreeIsIdle):
	req.state == Stealing and myWorker().leftIsWaiting and myWorker().rightIsWaiting
	behavior(declineReqFSA):
	ini: DS_MyOwnReq
	event: DSE_MyTreeIsIdle
	transition: discard
	fin: DS_TreeIdle
	behavior(declineReqFSA):
	ini: DS_MyOwnReq
	transition:
	# Our own request but we still have work, so we reset it and recirculate.
	ascertain: req.victims.capacity.int32 == workforce()
	req.retry = 0
	req.victims.refill()
	fin: DS_FindVictimAndSteal

[mratsim]

Actually this is not so simple.

If a task is delayed, its owner is the Pledge handle and no threads should access it. It means that its lifetime is not tied to a thread and so a thread cannot count the delayed tasks reliable.
Once a pledge is fulfilled, all linked delayed tasks are scheduled immediately but there is no simple way for the thread that fulfilled the pledge to notify the creator thread to decrease its count.

That said:

• if a pledge is never fulfilled that is an user error. The runtime will just continue forward and never process the tasks associated and never reclaim the memory as well
• if there is no such error, that means that there is a thread making progress on the pledges, so the main thread should be blocked at sync_root, i.e. a worker can send that it is idle after delaying a task, but there is at least one worker in the runtime that will not.

So where is the bug?

A sanity check would be for each worker to count the delayed tasks created and processed. On idle, they send that count to their parent. We can then assert in the syncRootBarrier that when the runtime is quiescent the delayed tasks created/process counts are the same.