[core] [wip] Rework memory store signal checking in C++ instead of cython

[dayshah]

Why are these changes needed?

So right now the Monitor thread used for compiled graphs calls ray.get and the get call can stay alive up to the point of the python interpreter starting to shutdown. This can cause a segfault when acquiring the GIL in check_signals as mentioned by @kevin85421 here #47864 (comment). Also based on the documentation of https://docs.python.org/3/c-api/exceptions.html#c.PyErr_CheckSignals which is currently used in check_signals, it doesn't actually work on threads that aren't the main python thread. By moving the signal checking to C++ code we no longer need to acquire the gil here and can also correctly check for signals outside the main thread.

Verified that ctrl+c works with a standard ray.get(result of task with while True) and also that compiled graphs tests pass consistently with this vs. before when usually one exception throwing dag test would segfault.

Current check_signals cython implementation:

ray/python/ray/_raylet.pyx

Lines 2360 to 2365 in 9375c1f

	cdef CRayStatus check_signals() nogil:
	with gil:
	# The Python exceptions are not handled if it is raised from cdef,
	# so we have to handle it here.
	try:
	PyErr_CheckSignals()

Related issue number

#48806 #47864

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[dayshah]

this has the possibility of being called from multiple threads right, should we make this safe by making it thread_local or guarded by a mutex?

[dayshah]

@jjyao If this makes sense, I can also replace the rest of the signal checking logic in the core worker with C++ code vs. passing it down through cython code to the CoreWorkerProcess

[dentiny]

Are you sure you want to overwrite default SIGTERM handler? Which means exception thrown will not have any effect

[dayshah]

Trying to replicate the behavior of the check_signals mentioned in the pr description, I think sigterm maps to the SystemExit, will double check

[dayshah]

hmm possibly not, removed sigterm and only keeping sigint for now, we really just need to check for ctrl+c looking at the previous pr that introduced check_signals

[ruisearch42]

@dayshah the title mentioned WIP, let me know when this is ready for review

[dayshah]

@dayshah the title mentioned WIP, let me know when this is ready for review

yup, my bad on assigning before I realized a test is failing in premerge, investigating