Restarting containers with arbitrary processes using SIGHUP

[ayushn21]

I'm playing around with async-container just to try and get my head around how it works and how I can use it to run multiple processes of anything.

I've created a basic project with two executables called jobs and web which write the current PID and timestamp to a file at 1 second intervals in an infinite loop.

I'm starting both these processes in a container which is within a controller. They both write to the file as expected. Then I send SIGHUP to the parent process to restart the container in a way that would simulate a "blue green" deployment, but the old processes don't seem to stop. The files will now contain timestamps from two different processes.

(On rare occasions, one of the old processes of either web or jobs does get killed, but most of the time I see both old and new processes writing to the file simultaneously.)

Not quite sure what I'm doing wrong here .... have I got something wrong with regard to notifying the status or something? I'd appreciate any guidance! I've attached a zip file with my code ... repro steps below.

To reproduce

async-container-demo.zip

• Download the attached zip file
• cd <path/to/folder>
• bundle install
• bundle exec start

You'll see a web.log and jobs.log file created with timestamps being written to them at 1 second intervals.

Grab the pid for App from the terminal:
0.0s debug: App [oid=0x500] [ec=0x514] [pid=15906] [2024-12-05 14:49:05 +1300]

Then run:
kill -HUP <pid>

Open the log files again and you should see two different PIDs writing to the file simultaneously. I'd expect the old process to stop and the new process to take over.

[ioquatix]

Something does look a bit odd, let me investigate further.

[ioquatix]

Okay, a small update. The issue is that this library expects signals to be sent to the progress group, not a specific process.

The reason for this, is that this is the default behaviour of the terminal and IIUC, things like systemd.

In other words, we expect that SIGINT (or SIGwhatever) is sent to all the processes (process group), and the controller waits for them/expects them to exit. Because that doesn't happen, we end up with the new container running alongside the old container.

This isn't an ideal situation, and we should probably handle it better, but I confirmed it does work correctly when using process groups:

#!/usr/bin/env ruby

require "async/container"
require "console"
require "debug"

# container = Async::Container.new

# container.spawn(name: "Web") do |task|
#   task.exec("bundle", "exec", "web")
# end

# container.spawn(name: "Jobs") do |task|
#   task.exec("bundle", "exec", "jobs")
# end

Console.logger.debug!

class WebController < Async::Container::Controller
  def setup(container)
    container.spawn(name: "Web") do |instance|
      instance.exec("bundle", "exec", "web")
    end

    # container.spawn(name: "Jobs") do |instance|
    #   instance.exec("bundle", "exec", "jobs")
    # end
  end
end

class App
  def self.start = self.new.start

  def start
    Console.logger.debug(self, "Starting container...")

    web_controller = WebController.new
    web_controller.run
  end
end

pgid = Process.getpgid(Process.pid)

Thread.new do
  sleep 5
  Console.debug(self, "Sending HUP signal to restart container...")
  Process.kill("HUP", -pgid)
  
  # sleep 5
  # Console.debug(self, "Sending INT signal to stop container...")
  # Process.kill("INT", Process.pid)
end

App.start

Regardless of this, I think we should improve the state handling so these situations don't occur (if possible).

The problem is there are two scenarios:

• (Normal) Pressing Ctrl-C from the terminal sends SIGINT to the foreground process group. If the controller, on receiving SIGINT, also sends it to the other processes in the container, they will receive TWO signals within rapid succession which isn't correct/ideal.
• (Alternative) Pressing Ctrl-C from the terminal sends SIGINT to the foreground process group, or the user can send SIGINT to the controller process. In both cases, we assume it was sent to the process group, but the controller will notice the children processes don't exit within a reasonable time frame, and try SIGINT at least once. At worst, this would have a slight delay in signal handling, but might be the least bad option.
• A significantly different option is to make the container to be a different process group from it's children, and ALWAYS send SIGINT to the children by hand. The problem with this approach is that if the controller hangs for any reason, the failure cases are much more tricky to deal with, and I believe terminal interaction is also problematic (the foreground process group may not be the right process).

Part of the issue here is we can't differentiate between a signal sent to a process group or an individual process.

There is one more issue to consider here: Despite some situations being multi-process, we also support multi-thread and multi-ractor. In those cases, we don't have child processes, but we still have the same semantics as a process group. I think we should add tests to confirm the behaviour here if there aren't already.

[ayushn21]

Thanks for the detailed writeup @ioquatix! I wasn't aware of "process groups" as a concept to be honest, and that the kill command is slightly different for a process group! ... still getting to grips with the Linux infra side of things ...

Given the complexities you flagged around not being able to differentiate between a signal sent to a process or a process group, and that containers could be multi-thread or multi-ractor, I wonder if this issue is better addressed with documentation rather than code? 🤔

[ioquatix]

I think it's both, probably we need to improve the documentation, and we also need to make it more robust. I'm currently working on it, but it's tricky. Process management is extremely complex.