Synergy Supervisor

Synergy Supervisor is a Python daemon that starts, monitors, restarts and if necessary kills Workers.

Supervisor is started from launch.py script by: python launch.py start Supervisor

At this point, newly created Supervisor daemon performs two actions:

• Navigate to /etc/synergy.conf (property defined by config_file setting) and read box_id from there

• Scan box_configuration collection from the MongoDB for all process entries that should be running on the given box_id. For instance:

  {"process_name" : "SiteMonthlyAggregator", "is_on" : true, "box_id" : "DEV" }
  {"process_name" : "SiteHourlyAggregator", "is_on" : true, "box_id" : "DEV" }
  {"process_name" : "ClientDailyAggregator", "is_on" : true, "box_id" : "DEV" }

After initialization, Supervisor starts all relevant Workers. For this, it reads process name and retrieves its fully qualified classname starter from context.py. For instance: workers.site_daily_aggregator.SiteDailyAggregator.start for SiteDailyAggregator

As soon as the process is started, Supervisor updates MongoDB entry with Process ID (pid). By doing so Supervisor could itself be restarted multiple times without interrupting the running workers or spawning another copy of such processes, while continuing work of preceding Supervisor.

Every 5 seconds Supervisor checks processes by their pid, and performs one of:

• Do nothing, if process is alive and healthy
• Kill, if user has changed process state in MongoDB entry to is_on=False
• Start process, if user has changed process state in MongoDB entry is_on=True
• Clean (purge IO) and restart if process state in MongoDB is is_on=True and current process has terminated

Since Supervisor is a daemon, there is no way to perform direct interaction with it: stdio, stderr and stdout are muted. However Synergy Supervisor has build in functionality to change process' state in corresponding MongoDB entry:

• python launch.py start Supervisor starts Supervisor. All processes started by its predecessors and still healthy and running will be kept alive. All processes marked as is_on=True and not running - will be started; all processes currently running, but marked as is_on=False will be killed and cleaned.
• python launch.py stop Supervisor stops/kills Supervisor. No processes started by Supervisor will be effected - they will continue to operate unsupervised.
• python launch.py super --stop SiteDailyAggregator marks SiteDailyAggregator to be killed and never started again. Note: works only if Supervisor daemon is running.
• python launch.py super --start SiteDailyAggregator marks SiteDailyAggregator to be started (if not running already). Note: works only if Supervisor daemon is running.
• python launch.py super --query prints snapshot of processes and their states into the console.
• python launch.py super --boxid ZZZ creates a /etc/synergy.conf file with BOX_ID=ZZZ.
• python launch.py super --init initializes DB with the list of processes for this BOX_ID. Note: should be run only once, before the Scheduler is started for the first time on given box.