Monitor Linux Processes using Pushgateway & Prometheus With Custom Bash Scripts via (supervisor)

Installing Pushgateway

Update system

apt update

We can get the latest version of pushgateway from the below link.

https://prometheus.io/download/

1. Create a dedicated user first.

sudo useradd \
    --system \
    --no-create-home \
    --shell /bin/false pushgateway

2. You can use the curl or wget command to download Prometheus latest version just edit the version in the command..

wget https://github.com/prometheus/pushgateway/releases/download/v1.2.0/pushgateway-1.2.0.linux-amd64.tar.gz

3. Then, we need to extract all Pushgateway files from the archive.

tar -xvzf pushgateway-1.2.0.linux-amd64.tar.gz

4. Move pushgateway binary to to /usr/local/bin.

mv pushgateway-1.4.2.linux-amd64/pushgateway /usr/local/bin/

5. Clean up.

rm -rf pushgateway*

6. Check if Pushgateway can be executed.

pushgateway --version

7. Also, you can get configuration options by running help.

pushgateway --help

8. Create a systemd service.

vim /etc/systemd/system/pushgateway.service

pushgateway.service

[Unit]
Description=Pushgateway
Wants=network-online.target
After=network-online.target

StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
User=pushgateway
Group=pushgateway
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/pushgateway

[Install]
WantedBy=multi-user.target

9. Enable the service.

sudo systemctl enable pushgateway

10. Start Pushgateway.

sudo systemctl start pushgateway

11. Check the status.

sudo systemctl status pushgateway

12. Pushgateway can be reachable at http://<ip>:9091.

13. Let's add Pushgateway as a target to Prometheus.

sudo vim /etc/prometheus/prometheus.yml

prometheus.yml

...
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ["localhost:9091"]

14. Check Prometheus configuration. If it's valid, reload the configuration.

promtool check config /etc/prometheus/prometheus.yml
curl -X POST http://localhost:9090/-/reload

15. Make sure that the target is up and healthy http://<ip>:9090.

Installing Supervisor

1. To install supervisor , Run the below commands,

apt-get update && sudo apt-get install python-setuptools

apt get install supervisor

pip install --upgrade supervisor

2. By default apt get install supervisor will install supervisord & supervisorctl files in /usr/bin directory and pip install -upgrade supervisor command will change the directory of the files to /usr/local/bin verify it by running ls /usr/local/bin if the files aren't present in /usr/local/bin copy them to this directory by running following commands.

cp /usr/bin/supervisord /usr/local/bin/

cp /usr/bin/supervisorctl /usr/local/bin/

Remove the files from /usr/bin directory

rm /usr/bin/supervisorctl && rm /usr/bin/supervisord

3. Also change the following line in /etc/init.d/supervisor file,

DAEMON=/usr/local/bin/supervisord

run the daemon-reload command

systemctl daemon-reload

4. Then we need to setup directories for supervisor if they aren't already there.

sudo mkdir /etc/supervisor

5. Add the supervisor config.

echo_supervisord_conf > /etc/supervisor/supervisord.conf

6. Open the /etc/supervisor/supervisord.conf file uncomment the [include] section and add the below line in the [include] section.

files=conf.d/*.conf

7. To setup programs under supervisord , We need to create conf.d directory if it's not already there.

mkdir /etc/supervisor/conf.d

8. Lets setup systemd file for the supervisor. Create a file supervisord.service and add the below contents.

vi /etc/systemd/system/supervisord.service

supervisord.service

[Unit]
Description=Supervisor daemon
Documentation=http://supervisord.org
After=network.target

[Service]
ExecStart=/usr/local/bin/supervisord -n -c /etc/supervisor/supervisord.conf
ExecStop=/usr/local/bin/supervisorctl $OPTIONS shutdown
ExecReload=/usr/local/bin/supervisorctl $OPTIONS reload
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target
Alias=supervisord.service

9. Lets start the supervisor service.

systemctl start supervisord.service

10. Check the status of the service and enable it to start on system boot,

systemctl enable supervisord.service
systemctl status supervisord.service

Create and Configure Bash Scripts that will collect the Cpu and Memory metrics and send it to pushgateway.

1. Create a file named cpu.sh in /home directory or whatever directory you want and add the below contents

#!/bin/bash
z=$(ps aux)
while read -r z
do
var=$var$(awk '{print "cpu_usage{process=\""$11"\", pid=\""$2"\"}", $3z}');
done <<< "$z"
curl -X POST -H "Content-Type: text/plain" --data "$var
" http://localhost:9091/metrics/job/top/instance/machine

2. Create a file named memory.sh in /home directory or whatever directory you want and add the below contents

#!/bin/bash
z=$(ps aux)
while read -r z
do
var=$var$(awk '{print "memory_usage{process=\""$11"\", pid=\""$2"\"}", $4z}');
done <<< "$z"
curl -X POST -H "Content-Type: text/plain" --data "$var
" http://localhost:9091/metrics/job/top/instance/machine

3. make sure the bash scripts are executable by running.

chmod +x cpu.sh && chmod +x memory.sh

4. Now we need to setup a program under supervisor to manage these bash scripts.

cd /etc/supervisor/conf.d

5. Create a file named cpu.conf and add the below configurations.

[program:cpu]
command=/home/cpu.sh
autostart=true
autorestart=true
startsecs=0
redirect_stderr=true
stderr_logfile=/var/log/cpu.err.log
stdout_logfile=/var/log/cpu.out.log

6. Create a file named memory.conf and add the below configurations.

[program:memory]
command=/home/memory.sh
autostart=true
autorestart=true
startsecs=0
redirect_stderr=true
stderr_logfile=/var/log/memory.err.log
stdout_logfile=/var/log/memory.out.log

7. We need to run the below commands to let the supervisor to reread the configuration file and apply the latest changes.

supervisorctl reread
supervisorctl update

8. To check the status of the processs running under supervisor run the following commands to verify if the scripts are running

supervisorctl status

The ouput should be

cpu                              RUNNING   pid 2385122, uptime 0:00:00
memory                           RUNNING   pid 2385123, uptime 0:00:00

9. To run the supervisor client , Run

supervisorctl

Setup Dashboard on Grafana

We now have the data collected by the pushgateway are sent to the prometheus.

We are going to setup a dashboard on grafana using the prometheus data source.

Login to Grafana , Hover to + icon and Click Dashboard

You will see the following screen.

Click Add Query , Queries to , Choose Prometheus

Add this prom query , topk(10, memory_usage) for monitoring memory usage

Add this prom query , topk(10, cpu_usage) for monitoring cpu usage

You should be able to find the linux processes and the memory and cpu usage of them.