Force kill fails to kill the process group in exec mode

[simonellefsen]

Consul Template version

As far as I can tell starting from version 0.27.0+

Configuration

No special configuration

Command

./consul-template_0.29.5 -log-level=trace -template "/tmp/config.ctmpl:/tmp/server.conf" -exec "IGNORE_SIGNALS=true python3 -u ./program.py"

#!/usr/bin/env python3
"""
Simple python example for testing envconsul/consul-template
"""

from typing import Any
import logging

import os
import sys
import time
import signal

logging.basicConfig(
    format="%(asctime)s [%(levelname)s] %(message)s", level=logging.DEBUG
)


def handler(signum: int, _frame: Any) -> None:
    logging.info("Signal handler called with signal %d", signum)
    if "IGNORE_SIGNALS" in os.environ:
        logging.info("Ignoring signal %d", signum)
        return
    sys.exit(0)


def main() -> None:
    logging.info("Program started")
    signal.signal(signal.SIGHUP, handler)
    signal.signal(signal.SIGINT, handler)
    signal.signal(signal.SIGTERM, handler)
    # loop forever
    logging.info("Looping...")
    while True:
        logging.info("Dumping _VAULT and _CONSUL environment variables")
        for _e in os.environ:
            if (
                (_e[:6] == "_VAULT" or _e[:7] == "_CONSUL")
                and not _e.endswith("_PASSWORD")
                and not _e.endswith("_SECRET_KEY")
                and not _e.endswith("_SECURITY_TOKEN")
            ):
                logging.info("%s=%s", _e, os.environ[_e])
        time.sleep(3600)
    # should not happen
    logging.info("Loop exit")
    sys.exit(1)


if __name__ == "__main__":
    main()

Debug output

https://gist.github.com/simonellefsen/206dd8476b7d0a0670fa79fdbf03a281

   PPID     PID    PGID     SID TTY        TPGID STAT   UID   TIME COMMAND
 737303  777229  777229  737303 pts/0     777229 Sl+   5004   0:00  \_ ./consul-template_0.29.5 -log-level=trace -template /tmp/config.ctmpl:/tmp/server.conf -exec IGNORE_SIGNALS=true python3 -u ./program.py
 777229  777237  777237  737303 pts/0     777229 S     5004   0:00      \_ sh -c IGNORE_SIGNALS=true python3 -u ./program.py
 777237  777238  777237  737303 pts/0     777229 S     5004   0:00          \_ python3 -u ./program.py

And after "shutdown" the application program is still running

   PPID     PID    PGID     SID TTY        TPGID STAT   UID   TIME COMMAND
      1  777238  777237  737303 pts/0     737303 S     5004   0:00 python3 -u ./program.py

Expected behavior

Expect the whole process group to be force killed.

Actual behavior

Force kill (SIGKILL) only kills the spawned subprocess shell and not the whole process group

So if I look at consul-template code

consul-template/child/child.go

Lines 354 to 356 in b0fc762

	if c.cmd != nil && c.cmd.Process != nil {
	c.cmd.Process.Kill()
	}

It very much looks like the force kill signal (SIGKILL) is not sent to the process group

But the default kill signal (SIGTERM) is sent to the process group

consul-template/child/child.go

Lines 397 to 400 in b0fc762

	if c.setpgid {
	// kill takes negative pid to indicate that you want to use gpid
	pid = -(pid)
	}

which fits with the log output from the application

References

• envconsul [options] <command>, command runs in a subshell envconsul#317

[eikenb]

Discovered in envconsul... hashicorp/envconsul#317