This project is the Hawkular Datasource plugin for Grafana. It works with:
- Hawkular Metrics (standalone)
- Hawkular Services, starting from version Alpha13
- OpenShift with Hawkular Metrics
The datasource URL must point to the Hawkular server, e.g. http://myhost:8080/hawkular (without ending /metrics)
Access: both proxy and direct modes should work in most configurations. Some earlier versions of Hawkular Metrics had a bug with CORS headers, that prevented the use of direct mode here.
If you want to use direct mode (that is, direct calls from client browser to Hawkular REST API), and if you have setup CORS restrictions in Hawkular, make sure to allow the Grafana server origin in Metrics' configuration.
If you're unsure, just use proxy mode and you should be fine.
Authentication must be set when working with a Hawkular server. Check the 'Basic Auth' box and fill the user and password fields.
Select the tenant. On Hawkular servers, use hawkular.
Alternatively, you can select Tenant per query, in which case the tenant will be prompted for every queries while editing a chart in dashboards.
Openshift-Metrics users must provide an authentication token.
Note that if you configure both Basic Authentication and a Token, only Basic Authentication will be effective.
When adding a Graph panel, the Metrics tab allows you fetch Gauges, Counters and Availability metrics in Hawkular. You can search by metric name and/or tag, assuming your version of hawkular-metrics is at least 0.20.0. Prior versions only allow searching by name.
To know your version of hawkular-metrics, check the status endpoint. E.g.
http://myhost:8080/hawkular/metrics/status
Since 0.24.0, hawkular-metrics offers a "tags query language" that allows more accurate queries on tags. If you query with tags and leave the metric name empty, the graph will display all matching metrics:
Tags queries are also useful to refine the metric name suggestions (auto-completion), to facilitate usage when Hawkular contains a lot of metrics:
To remove a tag expression, click on the tag name and select "Remove tag".
A full text editor is available, to use Hawkular's tag query language:
When used with hawkular-metrics prior to 0.24.0, tags are still available but the UI switches to the old key-value pairs system.
When using a Singlestat panel, some additional options come in. The Hawkular Datasource plugin can perform aggregations on multiple metrics, which usually the Singlestat panel doesn't. It's actually a two-steps aggregation: first, multiple series are reduced into a single one (that is either the sum of its parts, or the average). Then, the resulting series is aggregated over time through another folding: min, max, average etc.
Note that because the aggregation takes place in Hawkular, the Singlestat panel has nothing to aggregate. Thus in panel options, setting whatever in the value field on the Big value won't have any effect. However if you don't want to use the Hawkular aggregation, just set Multiple series aggregation to None.
If you selected the Tenant per query option in the Datasource configuration, you will have to provide the tenant name for every queries. You can enter either a full-text name in the Tenant field, or a variable that can holds several tenant names: just enter the variable name prefixed with $. (See section Templating variables for more information on variables). Note that, due to limitations with the templating variables, the list of tenants cannot be fetched dynamically from a remote source. It has to be defined explicitly, for instance, as a list of values with variables of type Custom.
Grafana allows you to create dashboard templates through the definition of variables. This is documented on Grafana's site. With the Hawkular Datasource Plugin, the variables of type 'Query' are mapped to the @get (url)/metrics Hawkular Metrics endpoint and can be used to retrieve tenant's metric names. Use the Query Options text field to pass query parameters, as illustrated below:
Example of query by tags to get metric ids
For instance, if you have metrics tagged "type:memory" and others tagged "type:cpu", you can write "tags=type:memory" to get only the "memory" ones, or "tags=type:cpu|memory" to get them both.
There is an exception to that rule: if the query string is prefixed with 'tags/', the variable will contain the matching tag names rather than the metric names. In this case, the Hawkular Metrics endpoint @get (url)/metrics/tags/{tags} will be used.
Example of query to get matching tag values
For instance, type "tags/type:*" to get all of the available tag values for "type".
Once you have set some variables, you can use them in graph queries by inserting the variable name prefixed with a $. It can be used either for row or graph duplication, or to display multiple series in a single graph from a single query. This is especially useful when metric names contain some dynamic parts and thus cannot be known in advance.
They can also be used in tag values, after operators = / != / is in / is not in.
Annotations are available through the use of 'string' and 'availability' metrics in Hawkular, or 'events' from Hawkular Alerts. It's a Grafana feature that allows to display custom events in timed charts.
Example with a 'string' metric:
- Setup an annotation query in Grafana. In 'Query', put the name of a 'string' metric you want to use to gather these annotations.
- Post any event (ie. string + timestamp) to some string metric.
Example, JSON posted on the Hawkular's REST API to /strings/my.timeline.metric/raw:
[
{"timestamp": 1485352477824, "value": "Starting my scenario..."}
]- Check your charts:
In the case of Hawkular Alerts events, you need to provide the trigger ID (several comma-separated IDs are allowed).
Metric names can be customized in chart legend. The Legend field can receive regular expressions in that purpose. Regex must be written between double curly braces, and include a parentheses group to point to what must be written: {{(.*)}}
Example:
From metric name
pod/c6e2a9ab-8d1c-11e7-b7be-06415eb17bbf/custom/haproxy_server_bytes_in_total{namespace=my-namespace,pod=web-45-6f464,route=web,server=10.1.10.138:8080,service=web}Legend:
HAProxy Server Bytes In {{namespace=([^,}]+)}} / {{pod=([^,}]+)}}Result:
HAProxy Server Bytes In my-namespace / web-45-6f464
Additional information on installing from sources can be found on hawkular.org.
-
Check the URL:
[host]/hawkular. Make sure there's no ending slash. When you open up this URL in a browser you should see the Hawkular logo, the installed version and a mention that the service is started. -
Make sure the credentials or token match your installation. In general, if you installed a standalone hawkular-metrics server without any specific configuration you probably don't have any authentication information to provide. If you installed hawkular-services using its installation guide you will probably have to fill-in the basic auth fields. If you are using Hawkular from OpenShift, you have to provide a Bearer token in the
Tokenfield. Tokens can be generated temporarily (go to[OpenShift host]/oauth/token/request) or from a Service account in OpenShift. -
Check the javascript debugging tool of your browser. If you see an error mentioning issues with CORS, switch to
proxymode in the datasource configuration.
Querying by tag was introduced before the plugin was properly versioned, so if you have a version >= 1.0.2 you should have it. However it is only enabled when Grafana talks to hawkular-metrics >= 0.20.0. To check your version of hawkular-metrics just open its status page in a browser ([host]/hawkular/metrics or [host]/hawkular/metrics/status).
Make sure the tenant you've configured is exactly the same than the one used to insert data. Beware that it is case sensitive. If you have any doubt about the actual presence of data in Hawkular, you can confirm with a curl command, for instance:
curl -u myUsername:myPassword \
-X GET "http://myserver/hawkular/metrics/gauges/mymetric/raw" \
-H "Content-Type: application/json" -H "Hawkular-Tenant: myTenant"`More about the REST API: http://www.hawkular.org/docs/rest/rest-metrics.html
Also note that in Hawkular, data has a retention period of 7 days by default (it can be configured). So if no data has been produced since that time, you won't be able to see anything.
Check your version of hawkular-metrics ([host]/hawkular/metrics or [host]/hawkular/metrics/status). Prior to 0.16.0, metric names containing slashes, like in OpenShift, were unfortunately not showing up in Grafana. You can consider upgrading metrics.