Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Running multiple agents on a single machine #803

Closed
wants to merge 28 commits into from
Closed

Running multiple agents on a single machine #803

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
6927874
Adding multiport agenr
cspetrdusak Jan 27, 2023
5fc6b51
javadoc updates
cspetrdusak Feb 14, 2023
66d7b77
Updating reading and comments
cspetrdusak May 10, 2023
47cab3d
Bump dependency versions
fstab Feb 17, 2023
e91c778
fix(examples): add missing kafka quota pattern
doxsch Feb 16, 2023
680703a
fix(examples): fix ordering for quota patterns
doxsch Feb 16, 2023
73f1f23
Fixed packaging to add version and name to BuildInfoMetric for the Ht…
dhoard Feb 8, 2023
730bb40
Bump snakeyaml version
fstab Mar 6, 2023
2dfb8b0
Use artifactId to generate debian package name
Skunnyk Nov 8, 2021
77788fa
Improve performance of duplicate sample lookup
amuraru Jul 3, 2022
d6a82b9
review comments
amuraru Jul 11, 2022
1dc95dd
[maven-release-plugin] prepare release parent-0.18.0
fstab Mar 7, 2023
2bc62d9
[maven-release-plugin] prepare for next development iteration
fstab Mar 7, 2023
626f761
Update docs for release 0.18.0
fstab Mar 7, 2023
9c7acc7
Add Doug Hoard as maintainer
fstab Mar 22, 2023
9421d16
Support Enums (#787)
speedfl Apr 21, 2023
74d7e33
feat: adding apache helix as an example (#724)
wmorgan6796 Apr 21, 2023
0e8d416
Temporarily removed ibmjava:8-jre due to CircleCI build failures (#790)
dhoard Apr 22, 2023
2858359
Refactored integration test suite (#780)
dhoard Apr 30, 2023
1161e12
Decreased test engine thread count (#794)
dhoard Apr 30, 2023
62a95c6
Changed build to use Java 11 (#795)
dhoard Apr 30, 2023
85d28d5
Added application heap memory setting (#796)
dhoard Apr 30, 2023
bab8c89
Added specific Docker container memory settings (#797)
dhoard Apr 30, 2023
e01e3d0
Changed list of smoke test Docker images to constrain memory usage wh…
dhoard Apr 30, 2023
c7254a7
Fixed incorrect test container logic that allowed tests to start befo…
dhoard May 2, 2023
eefed13
Refactored `BuildInfoCollector` for optimization (#800)
dhoard May 2, 2023
646c8c8
Merge branch 'main' into multiport-agent
dupetr May 12, 2023
6e5522c
Adding multiport agent
cspetrdusak Jan 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 73 additions & 22 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ rules:

Example configurations can be found in the `example_configs/` directory.


## Running the Standalone HTTP Server

The HTTP server is available in two versions with identical functionality:
Expand Down Expand Up @@ -66,6 +67,56 @@ rules:

As stated above, it is recommended to run JMX exporter as a Java agent and not as a standalone HTTP server.

## Running the MultiPort Java Agent

The reason for creation of this agent is monitoring and providing multiple webserver running on 1 machine. Origin of it
stems from running Apache Spark jobs on EMR clusters, where it's typical to have more than 1 executor running on 1 machine.

Differences compared to **Java Agent**
* different configuration string format
* it is capable of running more than 1 webserver on 1 machine

### Apache Spark example configuration

```bash
# version of the agent
AGENT_VERSION="0.17.2"

# path to javagent_multiport jar on S3
JMX_JAVA_AGENT_JAR_S3_PATH="s3://jmx_prometheus_javaagent_multiport-${AGENT_VERSION}.jar"

# path to configuration file for the agent on S3
JMX_JAVA_AGENT_CONFIG_S3_PATH="s3://jmx_exporter.yaml"

spark-submit --master yarn --deploy-mode cluster \
--jars "${JMX_JAVA_AGENT_JAR_S3_PATH}" \
--files "${JMX_JAVA_AGENT_CONFIG_S3_PATH}" \
--conf "spark.driver.extraJavaOptions=-javaagent:./jmx_prometheus_javaagent_multiport-${AGENT_VERSION}.jar=portStart=222,portEnd=333,configFile=./jmx_exporter.yaml" \
--conf "spark.executor.extraJavaOptions=-javaagent:./jmx_prometheus_javaagent_multiport-${AGENT_VERSION}.jar=portStart=222,portEnd=333,configFile=./jmx_exporter.yaml" \
```

### Configuration

With 6 parameters in total it has become rather impractical to extend the config line parsing with regex.

The configuration string has following shape `hostname=some_host.dc,portStart=222,portEnd=333,timeout=500,backoffMin=2000,backoffMax=4000,configFile=/some/path.yaml`.

| Parameter | value | optional | note |
|------------|-------------------------------------------------------------------------------|----------|---------------------------------------------------------------------------------------------------------------------|
| hostname | typically not used | true | defaults to 0.0.0.0 |
| portStart | port number - start of the range | false | |
| portEnd | port number - start of the range | false | the range must be continuous |
| timeout | timeout on how long will the agent try to connect to an supposedly empty port | true | default is 500ms |
| backoffMin | backoff before the agent will try to lookup and start on a given port | true | default is 2000ms |
| backoffMax | backoff before the agent will try to lookup and start on a given port | true | default is 4000ms |
| configFile | path to the config file | false | if used with `spark-submit --files`, it will be located on the same level as the jar. For example (`./config.yaml`) |



### Caveats

* You need to know how many ports you will need. This will depend on your system
* To play it safe, give enough headroom for your application
## Building

`./mvnw clean package` to build.
Expand Down Expand Up @@ -95,28 +146,28 @@ rules:
type: GAUGE
attrNameSnakeCase: false
```
Name | Description
---------|------------
startDelaySeconds | start delay before serving requests. Any requests within the delay period will result in an empty metrics set.
hostPort | The host and port to connect to via remote JMX. If neither this nor jmxUrl is specified, will talk to the local JVM.
username | The username to be used in remote JMX password authentication.
password | The password to be used in remote JMX password authentication.
jmxUrl | A full JMX URL to connect to. Should not be specified if hostPort is.
ssl | Whether JMX connection should be done over SSL. To configure certificates you have to set following system properties:<br/>`-Djavax.net.ssl.keyStore=/home/user/.keystore`<br/>`-Djavax.net.ssl.keyStorePassword=changeit`<br/>`-Djavax.net.ssl.trustStore=/home/user/.truststore`<br/>`-Djavax.net.ssl.trustStorePassword=changeit`
lowercaseOutputName | Lowercase the output metric name. Applies to default format and `name`. Defaults to false.
lowercaseOutputLabelNames | Lowercase the output metric label names. Applies to default format and `labels`. Defaults to false.
whitelistObjectNames | A list of [ObjectNames](http://docs.oracle.com/javase/6/docs/api/javax/management/ObjectName.html) to query. Defaults to all mBeans.
blacklistObjectNames | A list of [ObjectNames](http://docs.oracle.com/javase/6/docs/api/javax/management/ObjectName.html) to not query. Takes precedence over `whitelistObjectNames`. Defaults to none.
rules | A list of rules to apply in order, processing stops at the first matching rule. Attributes that aren't matched aren't collected. If not specified, defaults to collecting everything in the default format.
pattern | Regex pattern to match against each bean attribute. The pattern is not anchored. Capture groups can be used in other options. Defaults to matching everything.
attrNameSnakeCase | Converts the attribute name to snake case. This is seen in the names matched by the pattern and the default format. For example, anAttrName to an\_attr\_name. Defaults to false.
name | The metric name to set. Capture groups from the `pattern` can be used. If not specified, the default format will be used. If it evaluates to empty, processing of this attribute stops with no output.
value | Value for the metric. Static values and capture groups from the `pattern` can be used. If not specified the scraped mBean value will be used.
valueFactor | Optional number that `value` (or the scraped mBean value if `value` is not specified) is multiplied by, mainly used to convert mBean values from milliseconds to seconds.
labels | A map of label name to label value pairs. Capture groups from `pattern` can be used in each. `name` must be set to use this. Empty names and values are ignored. If not specified and the default format is not being used, no labels are set.
help | Help text for the metric. Capture groups from `pattern` can be used. `name` must be set to use this. Defaults to the mBean attribute description, domain, and name of the attribute.
cache | Whether to cache bean name expressions to rule computation (match and mismatch). Not recommended for rules matching on bean value, as only the value from the first scrape will be cached and re-used. This can increase performance when collecting a lot of mbeans. Defaults to `false`.
type | The type of the metric, can be `GAUGE`, `COUNTER` or `UNTYPED`. `name` must be set to use this. Defaults to `UNTYPED`.
| Name | Description |
|---------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| startDelaySeconds | start delay before serving requests. Any requests within the delay period will result in an empty metrics set. |
| hostPort | The host and port to connect to via remote JMX. If neither this nor jmxUrl is specified, will talk to the local JVM. |
| username | The username to be used in remote JMX password authentication. |
| password | The password to be used in remote JMX password authentication. |
| jmxUrl | A full JMX URL to connect to. Should not be specified if hostPort is. |
| ssl | Whether JMX connection should be done over SSL. To configure certificates you have to set following system properties:<br/>`-Djavax.net.ssl.keyStore=/home/user/.keystore`<br/>`-Djavax.net.ssl.keyStorePassword=changeit`<br/>`-Djavax.net.ssl.trustStore=/home/user/.truststore`<br/>`-Djavax.net.ssl.trustStorePassword=changeit` |
| lowercaseOutputName | Lowercase the output metric name. Applies to default format and `name`. Defaults to false. |
| lowercaseOutputLabelNames | Lowercase the output metric label names. Applies to default format and `labels`. Defaults to false. |
| whitelistObjectNames | A list of [ObjectNames](http://docs.oracle.com/javase/6/docs/api/javax/management/ObjectName.html) to query. Defaults to all mBeans. |
| blacklistObjectNames | A list of [ObjectNames](http://docs.oracle.com/javase/6/docs/api/javax/management/ObjectName.html) to not query. Takes precedence over `whitelistObjectNames`. Defaults to none. |
| rules | A list of rules to apply in order, processing stops at the first matching rule. Attributes that aren't matched aren't collected. If not specified, defaults to collecting everything in the default format. |
| pattern | Regex pattern to match against each bean attribute. The pattern is not anchored. Capture groups can be used in other options. Defaults to matching everything. |
| attrNameSnakeCase | Converts the attribute name to snake case. This is seen in the names matched by the pattern and the default format. For example, anAttrName to an\_attr\_name. Defaults to false. |
| name | The metric name to set. Capture groups from the `pattern` can be used. If not specified, the default format will be used. If it evaluates to empty, processing of this attribute stops with no output. |
| value | Value for the metric. Static values and capture groups from the `pattern` can be used. If not specified the scraped mBean value will be used. |
| valueFactor | Optional number that `value` (or the scraped mBean value if `value` is not specified) is multiplied by, mainly used to convert mBean values from milliseconds to seconds. |
| labels | A map of label name to label value pairs. Capture groups from `pattern` can be used in each. `name` must be set to use this. Empty names and values are ignored. If not specified and the default format is not being used, no labels are set. |
| help | Help text for the metric. Capture groups from `pattern` can be used. `name` must be set to use this. Defaults to the mBean attribute description, domain, and name of the attribute. |
| cache | Whether to cache bean name expressions to rule computation (match and mismatch). Not recommended for rules matching on bean value, as only the value from the first scrape will be cached and re-used. This can increase performance when collecting a lot of mbeans. Defaults to `false`. |
| type | The type of the metric, can be `GAUGE`, `COUNTER` or `UNTYPED`. `name` must be set to use this. Defaults to `UNTYPED`. |

Metric names and label names are sanitized. All characters other than `[a-zA-Z0-9:_]` are replaced with underscores,
and adjacent underscores are collapsed. There's no limitations on label values or the help text.
Expand Down
Loading