Registering "foreign" threads

[dpwiz]

I have an API that's using forkIO to e.g. handle client requests.
How do I get back into Process context after being forked off the server loop?

The unlift for readers is basically storing env and re-running with it again, but there's quite a bit of work in spawnImpl...

In a nutshell,

Troupe.spawn do
  env <- ask
  severPid <- Troupe.self
  Server.run \connection ->
    flip Troupe.Process.runProcess env do -- naive unlifting doesn't work
      handlerPid <- Troupe.self
      traceShowM (serverPid, handlerPid) -- oops, same

If I pry open Troupe.Process a little and split spawnImpl at currentEnv <- getProcessEnv, then it appears to work:

Troupe.spawn do
  env <- ask
  severPid <- Troupe.self
  Server.run \connection ->
    void $! Troupe.spawnImplWith env Troupe.Unbound pure do
      handlerPid <- Troupe.self
      traceShowM (serverPid, handlerPid) -- looks okay

But I don't know if some stuff should be masked or something like that.

[NicolasT]

I don't think we can (or should) support using plain forkIO and then re-enter into a Process context. What could be done is allow any code to send some message to some existing ProcessId, even outside a Process context but that's about it.

An unliftio-style API won't work because we really want a ProcessId to be tied to a single Haskell thread. unliftio would get the reader env and reuse it in the thread, hence we'd end up with two Haskell threads both thinking they're the same ProcessId and, worse, receive messages from the same CQueue. Whilst sending messages to a queue is thread-safe, receiving isn't (necessarily).

Basically, your Server.run shouldn't forkIO but instead spawn new Processes to handle clients.

[dpwiz]

Basically, your Server.run shouldn't forkIO but instead spawn new Processes to handle clients.

Yeah... Unfortunately, this is not like how it works in reality. Many, if not all, APIs forkIO new threads without a thought and it is either there's a function to attach those threads or "you're welcome to reimplement everything from the ground up, good luck, don't send PRs".

[NicolasT]

Basically, your Server.run shouldn't forkIO but instead spawn new Processes to handle clients.

Yeah... Unfortunately, this is not like it works in reality. Many, if not all, APIs forkIO new threads without a thought and it is either there's a function to attach those threads or "you're welcome to reimplement everything from the ground up, good luck, don't send PRs".

I'm aware 😄 so let's see what's possible / what a good API could look like, and define the semantics of whatever the system provides.

An example (from memory):

withRunAsProcess :: (Process r a -> IO b) -> Process r c

could allow for

demo :: Process r Void
demo = withRunAsProcess $ \rap -> do
  Server.run (rap . handleConnection)
  where
    handleConnection :: Connection -> Process r a
    handleConnection conn = _

Here, the thread running rap could become the monitoring thread of a Process thread it spawns to run the actual Process r a code (registering a new ProcessId from the shared context retrieved from the parent thread where withRunAsProcess is invoked, as well as the r value etc.)

However, this is likely not sufficient: since there's no way to share ProcessIds of such processes to the parent (except for a message later on), we'll likely want to allow to monitor or link to such processes, i.e. pass in some options to withRunAsProcess.

Thinking of it, monitoring won't really work since if Server.run is within a withRunAsProcess, there's no way to receive any Down message anyway.

I understand the problem you're facing and would be happy to collaborate in finding a suitable solution!

[NicolasT]

As for unliftio, I'm not sure there's a good instance of MonadUnliftIO possible for Process r. It'd be great if there is one, opening up a lot of possibilities.

I think the problem lies in withRunInIO: we don't know whether this is invoked to run a new Process/thread, or for some other reason.

[dpwiz]

I posted my sketch at #43. Not nearly a final proposal or anything like that (=

[NicolasT]

I wrote some thoughts about unliftio at #44.