Skip to content

Commit

Permalink
Fine tune Monitoring
Browse files Browse the repository at this point in the history
  • Loading branch information
Lennonka committed Nov 20, 2023
1 parent 6fed59d commit 23d0164
Show file tree
Hide file tree
Showing 24 changed files with 222 additions and 201 deletions.
12 changes: 8 additions & 4 deletions guides/common/modules/con_metrics-data-retention.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,15 @@

The storage capacity required by PCP data logging is determined by the following factors:

* The metrics being logged.
* The logging interval.
* The retention policy.
* The logged metrics
* The logging interval
* The retention policy

The default logging (sampling) interval is 60 seconds.
The default retention policy is to compress archives older than one day and to keep archives for the last 14 days.

You can increase the logging interval or shorten the retention policy to save storage space.
If you require high-resolution sampling, you can decrease the logging interval.
In such case, ensure that you have enough storage space.

The default retention policy is to keep archives for the last 14 days, compressing archives older than one day.
PCP archive logs are stored in the `/var/log/pcp/pmlogger/_{foreman-example-com}_` directory.
19 changes: 9 additions & 10 deletions guides/common/modules/con_metrics-overview.adoc
Original file line number Diff line number Diff line change
@@ -1,19 +1,18 @@
[id='metrics-overview_{context}']
= Metrics Overview

Obtaining metrics from {Project} is useful for troubleshooting a current issue, and capacity planning.
This guide describes how to collect live metrics and archive them for a fixed period of time.
You can set up a third-party solution to collect live metrics from {ProjectServer}, archive them for a fixed period of time, and analyze them.
Obtaining metrics from {Project} is useful for capacity planning and troubleshooting performance issues.

ifdef::satellite[]
If you need to raise a support case with {Team} to resolve a performance issue, the archived data provides valuable insight.
Note that {Team} Support can only access the archived data if you upload it to a Support Case.
Note that {Team} Support can only access the archived data if you upload it to a support case.
endif::[]

You can collect the following metrics from {Project}:

* Basic statistics from the operating system, including system load, memory utilization, and input/output operations.
* Process statistics, including memory and CPU utilization.
* Apache HTTP Server activity statistics.
* PostgreSQL activity statistics.
* {Project} application statistics.

Use Performance Co-Pilot (PCP) to collect and archive {Project} metrics.
* Basic statistics from the operating system, including system load, memory utilization, and input/output operations
* Process statistics, including memory and CPU utilization
* Apache HTTP Server activity statistics
* PostgreSQL activity statistics
* {Project} application statistics
6 changes: 3 additions & 3 deletions guides/common/modules/con_pcp-metrics.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ For every metric entry, PCP stores both its data and metadata.
This includes the metrics description, data type, units, and dimensions.
For example, the metadata enables PCP to output multiple metrics with different dimensions.

The value of a counter metric only increases.
If a metric is a counter, its value can only increase.
For example, a count of disk write operations on a specific device only increases.
When you query the value of a counter metric, PCP converts this into a rate value by default.

In addition to system metrics such as CPU, memory, kernel, XFS, disk, and network, the following metrics are configured:
In addition to system metrics, such as CPU, memory, kernel, XFS, disk, and network, the following metrics are configured:

[%header,cols=2*]
[%header,cols="3,4"]
|===
|Metric
|Description
Expand Down
6 changes: 0 additions & 6 deletions guides/common/modules/con_performance-co-pilot.adoc

This file was deleted.

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[id="setting-up-the-metrics-monitoring-solution_{context}"]
= Setting Up the Metrics Monitoring Solution

Install PCP packages and configure PCP data collection.
You can use the PCP CLI tools to retrieve metrics in the command line.
ifndef::foreman-deb[]
Optionally, you can install Grafana to enable web UI access to metrics.
endif::[]
25 changes: 14 additions & 11 deletions guides/common/modules/proc_changing-data-retention-policy.adoc
Original file line number Diff line number Diff line change
@@ -1,19 +1,22 @@
[id='changing-data-retention-policy_{context}']
= Changing Data Retention Policy

This procedure describes how to change the data retention policy.
You can change the data retention policy to control after how long the PCP data are archived and deleted.

.Procedure
ifndef::foreman-deb[]
. Edit the `/etc/sysconfig/pmlogger_timers` file.
endif::[]
ifdef::foreman-deb[]
. Edit the `/etc/default/pmlogger_timers` file.
. Open the `/etc/default/pmlogger_timers` file.
endif::[]
. Find the line containing `PMLOGGER_DAILY_PARAMS`.
. If it is commented, uncomment the line.
. Ensure the default parameter `-E` is present.
. Add parameter `-x`, and add a value for the desired number of days after which data is archived.
. Add parameter `-k`, and add a value for the number of days after which data is deleted.
ifndef::foreman-deb[]
. Open the `/etc/sysconfig/pmlogger_timers` file.
endif::[]
. Locate the `PMLOGGER_DAILY_PARAMS` line.
. If the line is commented, uncomment the line.
. Configure the following parameters:
* Ensure the default `-E` parameter is present.
* Append the `-x` parameter and add a value for the required number of days after which data is archived.
* Append the `-k` parameter and add a value for the number of days after which data is deleted.

+
For example, the parameters `-x 4 -k 7` specify that data will be compressed after 4 days, and deleted after 7 days.
For example, the parameters `-x 4 -k 7` specify that data will be compressed after 4 days and deleted after 7 days.
. Save the file.
16 changes: 12 additions & 4 deletions guides/common/modules/proc_changing-default-logging-interval.adoc
Original file line number Diff line number Diff line change
@@ -1,9 +1,17 @@
[id='changing-default-logging-interval_{context}']
= Changing Default Logging Interval

This procedure describes how to change the default logging interval.
You can change the default logging interval to either increase or decrease the sampling rate, at which the PCP metrics are logged.
A larger interval results in a lower sampling rate.

.Procedure
. Edit the `/etc/pcp/pmlogger/control.d/local` configuration file.
. Edit the LOCALHOSTNAME line and append `-t __XX__s`, where _XX_ is the desired time interval, measured in seconds.
. Restart the `pmlogger` service.
. Open the `/etc/pcp/pmlogger/control.d/local` configuration file.
. Locate the `LOCALHOSTNAME` line.
. Append `-t __XX__s`, where _XX_ is the required time interval in seconds.
. Save the file.
. Restart the `pmlogger` service:
+
[options="nowrap" subs="+quotes,verbatim,attributes"]
----
# systemctl restart pmlogger
----
18 changes: 8 additions & 10 deletions guides/common/modules/proc_configuring-pcp-data-collection.adoc
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
[id='configuring-pcp-data-collection_{context}']
= Configuring PCP Data Collection

This procedure describes how to configure PCP to collect metrics about processes, {Project}, Apache HTTP Server, and PostgreSQL.
You can configure PCP to collect metrics about processes, {Project}, Apache HTTP Server, and PostgreSQL.

.Procedure
. To configure PCP to collect data about {Project} processes, configure the process monitoring PMDA to use the {Project} specific config:
. Symlink the {Project} specific configuration to PMDA process monitoring:
+
----
# ln -s /etc/pcp/proc/foreman-hotproc.conf /var/lib/pcp/pmdas/proc/hotproc.conf
----
+
By default, PCP collects basic system metrics.
By default, PCP only collects basic system metrics.
This step enables detailed metrics about the following {Project} processes:
+
* Java
Expand All @@ -21,23 +21,21 @@ This step enables detailed metrics about the following {Project} processes:
ifndef::foreman-el,foreman-deb[]
* Pulpcore
endif::[]

. Install the process monitoring PMDA.
. Install the process monitoring PMDA:
+
----
# cd /var/lib/pcp/pmdas/proc
# ./Install
----

. Configure PCP to collect metrics from Apache HTTP Server.
+
.. Enable the Apache HTTP Server extended status module.
.. Enable the Apache HTTP Server extended status module:
+
[options="nowrap", subs="verbatim,quotes,attributes"]
----
# {foreman-installer} --enable-apache-mod-status
----
.. Enable the Apache HTTP Server PMDA.
.. Enable the Apache HTTP Server PMDA:
+
----
# cd /var/lib/pcp/pmdas/apache
Expand All @@ -49,13 +47,13 @@ endif::[]
# cd /var/lib/pcp/pmdas/postgresql
# ./Install
----
. Enable telemetry functionality in {Project}:
. Enable the telemetry feature in {Project}:
+
[options="nowrap", subs="verbatim,quotes,attributes"]
----
# {foreman-installer} --foreman-telemetry-prometheus-enabled true
----
. Configure PCP to collect data from {Project}.
. Configure PCP to collect data from {Project}:
+
[options="nowrap", subs="verbatim,quotes,attributes"]
----
Expand Down
17 changes: 0 additions & 17 deletions guides/common/modules/proc_confirming-data-storage-usage.adoc

This file was deleted.

This file was deleted.

31 changes: 31 additions & 0 deletions guides/common/modules/proc_enabling-web-ui-access-to-metrics.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
[id='enabling-web-ui-access-to-metrics_{context}']
= Enabling Web UI Access to Metrics

You can enable web UI access to metrics collected by PCP by installing Grafana.

.Procedure
. Install Grafana and the Grafana PCP plug-in on your {ProjectServer}:
+
[options="nowrap", subs="verbatim,quotes,attributes"]
----
# {package-install-project} grafana grafana-pcp
----
. Start and enable the Grafana web service and the PCP proxy service:
+
----
# systemctl enable --now pmproxy grafana-server
----
. Open the firewall port to allow access to the Grafana web interface:
+
----
# firewall-cmd --permanent --add-service=grafana
----
. Reload the firewall configuration to apply the changes:
+
----
# firewall-cmd --reload
----
. Install PCP Redis and configure Grafana to load it.
For more information, see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/setting-up-graphical-representation-of-pcp-metrics_monitoring-and-managing-system-status-and-performance#configuring-pcp-redis_setting-up-graphical-representation-of-pcp-metrics[Configuring PCP Redis] in _{RHEL} 8 Monitoring and managing system status and performance_.
. Access the Grafana web UI, enable the PCP plug-in, and add PCP Redis as a data source.
For more information, see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/setting-up-graphical-representation-of-pcp-metrics_monitoring-and-managing-system-status-and-performance#accessing-the-grafana-web-ui_setting-up-graphical-representation-of-pcp-metrics[Accessing the Grafana web UI] in _{RHEL} 8 Monitoring and managing system status and performance_.
10 changes: 5 additions & 5 deletions guides/common/modules/proc_identifying-available-metrics.adoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[id=identifying-available-metrics_{context}]
= Identifying Available Metrics

* To list all metrics available via PCP, enter the following command:
* To list all metrics available through PCP, enter the following command:
+
----
# pminfo
Expand All @@ -17,16 +17,16 @@
----
# less /var/log/pcp/pmlogger/_{foreman-example-com}_/pmlogger.log
----
* The pmlogger daemon archives data as it is received, according to its configuration.
To confirm the active archive file, enter the following command:
* The `pmlogger` daemon archives data as received, according to its configuration.
To retrieve the name of the active log file, enter the following command:
+
----
# pcp | grep logger
----
+
The output includes the file name of the active archive file, for example:
The output includes the file name of the active log file, for example:
+
[options="nowrap", subs="verbatim,quotes,attributes"]
----
/var/log/pcp/pmlogger/_{foreman-example-com}_/_20230831.00.10_
pmlogger: primary logger: /var/log/pcp/pmlogger/_{foreman-example-com}_/_20230831.00.10_
----
32 changes: 0 additions & 32 deletions guides/common/modules/proc_installing-pcp-packages.adoc

This file was deleted.

31 changes: 31 additions & 0 deletions guides/common/modules/proc_installing-pcp.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
[id='installing-pcp_{context}']
= Installing PCP

Install the PCP packages on your {ProjectServer} and enable PCP daemons.

.Prerequisite
* Ensure you have the minimum of 20 GB space available in the `/var/log/pcp` directory.
+
With the default PCP data retention settings, data storage is estimated to use between 100 MB and 500 MB of disk space per day, but may use up to several gigabytes over time.
For more information, see xref:metrics-data-retention_{context}[].

.Procedure
. Install the PCP packages:
+
[options="nowrap", subs="verbatim,quotes,attributes"]
----
# {package-install-project} pcp \
ifndef::foreman-deb[]
pcp-pmda-apache \
pcp-pmda-openmetrics \
pcp-pmda-postgresql \
pcp-pmda-redis \
pcp-system-tools \
endif::[]
foreman-pcp
----
. Enable and start the Performance Metrics Collector daemon and Performance Metrics Logger daemon:
+
----
# systemctl enable --now pmcd pmlogger
----
Loading

0 comments on commit 23d0164

Please sign in to comment.