-
Notifications
You must be signed in to change notification settings - Fork 803
Configuration
Burrow is configured using viper, which handles configurations specified in multiple formats. This includes TOML, JSON, YAML, and others, but this configuration guide will focus on TOML. If you are not familiar with this format, please take a look at the reference documentation for TOML. It looks similar to INI configs, but there are some thing that might trip you up, like needing to have quotes around string values.
The configuration is organized into several subheadings. There is an example configuration file in the "config" directory of the Burrow distribution.
The [general] configuration heading covers the location of the PID file, as well as an optional place to put the stdout/stderr output.
[general]
pidfile="burrow.pid"
stdout-logfile="burrow.out"
access-control-allow-origin="mysite.example.com"
| Key | Value Type | Required | Default Value | Description |
|---|---|---|---|---|
| pidfile | path | no | burrow.pid | The path and filename to store the PID of the running process in. |
| stdout-logfile | path | no | (empty) | If specified, it is the path and filename to redirect stdout and stderr into. If not specified, stdout/stderr are not redirected. |
| access-control-allow-origin | string | no | (empty) | The value to put in the Access-Control-Allow-Origin header in responses from the HTTP server in Burrow. |
The [logging] configuration heading covers the configuration for logging. If no filename config is provided, all logs are written to stdout. If it is specified, logs are written out to the file configured using lumberjack and periodicially rolled.
[logging]
filename="logs/burrow.log"
level="info"
maxsize=100
maxbackups=30
maxage=10
use-localtime=false
use-compression=true
| Key | Value Type | Required | Default Value | Description |
|---|---|---|---|---|
| filename | path | no | (empty) | The path and filename to write logs to. If not specified, logs are written to stdout. |
| level | string | no | info | The minimum log level to output. Must be one of "debug", "info", "warn", "error", "panic", "fatal". |
| maxsize | integer | no | 100 | The maximum size for a log file, in MB. |
| maxbackups | integer | no | 10 | The maximum number of log files to maintain. |
| maxage | integer | no | 30 | The maximum number of days to keep log files for. |
| use-localtime | boolean | no | false | If true, write log entries using the system's local time. Otherwise, use UTC. |
| use-compression | boolean | no | false | If true, compress rotated log files. |
The [zookeeper] heading specifies a Zookeeper ensemble to use in order to store metadata for modules and provide locking between multiple copies of Burrow. This allows you to run multiple copies of Burrow with only one being allowed to execute the notifiers at any time.
[zookeeper]
servers=["zkhost01.example.com:2181", "zkhost02.example.com:2181", "zkhost03.example.com:2181" ]
timeout=6
root-path=/mypath/burrow
| Key | Value Type | Required | Default Value | Description |
|---|---|---|---|---|
| servers | list of hostname:port | yes | (none) | A list of strings in the form "hostname:port" that point to the servers in the Zookeeper ensemble. At least one is required. |
| timeout | integer | no | 6 | The expiration timeout for Zookeeper sessions, in seconds. |
| root-path | znode | no | /burrow/notifier | A full path to the znode that Burrow is allow to write under. The path will be created if it does not already exist. |
Profiles are used to generalize some configurations for the modules. These are items like TLS and SASL configs that may be used in multiple modules, to prevent duplication and error in the configs. While none of these are required, when they are provided they must each have a unique subheading (as described).
The [client-profile] heading defines profiles that are used for connecting to Kafka clusters. In this section you can group together a client version, TLS config, and a SASL config (both described below), along with a customized client ID string. It must have a unique subheading specified, such as [client-profile.myprofile], where "myprofile" is the name it will be referred to as. Multiple subheadings may be specified to have different client configurations.
[client-profile.myclient]
kafka-version="0.10.2"
client-id="burrow-myclient"
tls="mytlsprofile"
sasl="mysaslprofile"
| Key | Value Type | Required | Default Value | Description |
|---|---|---|---|---|
| kafka-version | string | yes | 0.8 | The client version of Kafka to support. Valid versions are: 0.8, 0.8.0, 0.8.1, 0.8.2, 0.9, 0.9.0, 0.9.0.0, 0.9.0.1, 0.10, 0.10.0, 0.10.0.0, 0.10.0.1, 0.10.1, 0.10.1.0, 0.10.2, 0.10.2.0, 0.11.0, 0.11.0.1, 0.11.0.2 |
| client-id | string | no | burrow-lagchecker | A string to be passed to Kafka as the client ID. |
| tls | string | no | (empty) | If specified, it is the name of a TLS profile to use in this client configuration. |
| sasl | string | no | (empty | If specified, it is the name of a SASL profile to use in this client configuration. |
The [tls] heading sets up a TLS (or SSL, if you're old school) profile that can be referenced by multiple modules to set up secure connections. It must have a unique subheading specified, such as [tls.myprofile], where "myprofile" is the name it will be referred to as. Multiple subheadings may be specified to have different TLS configurations.
[tls.mytlsprofile]
certfile="certificate.crt"
keyfile="keyfile.key"
cafile="rootcerts.crt"
no-verify=false
| Key | Value Type | Required | Default Value | Description |
|---|---|---|---|---|
| certfile | path | yes | (empty) | The path to the certificate file to be used (either a client certificate for client connections, or a server certificate for the HTTP server). |
| keyfile | path | yes | (empty) | The path to the key file that matches the certfile. |
| cafile | path | no | (empty) | If specified, it is the path to a bundle of certificates that provide the trusted root certificates for the connection. |
| no-verify | boolean | no | false | If true, do not validate the trust of the certificate sent by the other side of this connection. |
The [sasl] heading sets up a SASL profile that can be referenced by multiple modules for authenticating a connection. It must have a unique subheading specified, such as [sasl.myprofile], where "myprofile" is the name it will be referred to as. Multiple subheadings may be specified to have different SASL configurations.
[sasl.mysaslprofile]
username="myname"
password="sekrit"
handshake-first=false
| Key | Value Type | Required | Default Value | Description |
|---|---|---|---|---|
| username | string | yes | (empty) | The username to use for authentication. |
| password | string | yes | (empty) | The password to use for authentication. |
| handshake-first | boolean | no | true | If false, do not send SASL handshake first (only used for Sarama connections to Kafka when using a non-Kafka proxy). |
The [httpserver] heading configures the HTTP server in Burrow. If this heading is not specified, Burrow will start the server on a random available port (which will be printed in the log output). If it is specified, it must have a unique subheading specified, such as [httpserver.mylistener] (where "mylistener" is the subheading). Multiple subheadings may be specified in order to start the server listening on multiple addresses.
[httpserver]
address=":8080"
timeout=300
tls="tls-profile-name"
| Key | Value Type | Required | Default Value | Description |
|---|---|---|---|---|
| address | string | yes | (empty) | This is a string of the form ip:port that specifies what the server should listen on. If the ip part is left blank, the listener attaches to all IPs on the host. If the port part is left blank, a random port is chosen. |
| timeout | integer | no | 300 | The timeout and keepalive setting for connections, in seconds. |
| tls | string | no | (empty) | If specified, it must be the name of a configured TLS profile and the listener is started with this configuration. |
The [storage] heading configures the storage subsystem in Burrow. It is not required to specify this section, as Burrow will properly configure a storage module with the default values. If it is specified, it must have a unique subheading specified, such as [storage.mystorage] (where "mystorage" is the subheading). Only one storage section is permitted, however
[storage.mystorage]
class-name="inmemory"
intervals=10
expire-group=604800
workers=20
min-distance=1
group-whitelist=".*"
| Key | Value Type | Required | Default Value | Description |
|---|---|---|---|---|
| class-name | string | yes | (empty) | This is the name of the module type. The only permitted value is "inmemory". |
| intervals | integer | no | 10 | The number of offsets to store for each partition. |
| expire-group | integer | no | 604800 | The number of seconds after which a group will be purged if it has not committed offsets. |
| workers | integer | no | 20 | The number of worker goroutines to start up. This number can be increased to provide more parallelism in the storage module. |
| min-distance | integer | no | 0 | Coalesce offset commits that are more frequent than this number of seconds. Used with the intervals configuration, this can be used to set a minimum time for the offset window (10 offsets x 15s = 150s minimum window). |
| group-whitelist | string (regex) | no | (empty) | If specified, ignore data for groups that do not match the regular expression. |
The [evaluator] heading configures the evaluator subsystem in Burrow. It is not required to specify this section, as Burrow will properly configure a evaluator module with the default values. If it is specified, it must have a unique subheading specified, such as [evaluator.myeval] (where "myeval" is the subheading). Only one evaluator section is permitted, however
[evaluator.mystorage]
class-name="caching"
expire-cache=10
| Key | Value Type | Required | Default Value | Description |
|---|---|---|---|---|
| class-name | string | yes | (empty) | This is the name of the module type. The only permitted value is "caching". |
| expire-cache | integer | no | 604800 | The number of seconds to cache evaluation results for. |
The [cluster] heading configures a single Kafka cluster to fetch topic and offset information from. It is not required to specify this section, but if you don't you won't have Burrow doing any work. When it is specified, it must have a unique subheading specified, such as [cluster.myclustername]. The "myclustername" part is the name by which this cluster will be referred to in other parts of the configuration, as well as in HTTP requests and notifications. You may configure as many cluster sections as you want (and Burrow is efficient at handling multiple Kafka clusters). Note that the cluster configuration only retrieves the topic list and cluster offsets. To get consumer group information, you must also configure a consumer section.
[cluster.myclustername]
class-name="kakfa"
(add additional class configs here)
| Key | Value Type | Required | Default Value | Description |
|---|---|---|---|---|
| class-name | string | yes | (empty) | This is the name of the cluster module type. The available classes are:
|
The [consumer] heading configures a single repository to retrieve consumer group offsets and information from. It is not required to specify this section, but if you don't you won't have Burrow doing much work. When it is specified, it must have a unique subheading specified, such as [consumer.myconsumers]. The "myconsumers" part is the name that will be used when writing logs about this module. You may configure as many consumer sections as you want for a given cluster.
[consumer.myconsumers]
class-name="kakfa"
cluster="myclustername"
(add additional class configs here)
| Key | Value Type | Required | Default Value | Description |
|---|---|---|---|---|
| class-name | string | yes | (empty) | This is the name of the cluster module type. The available classes are:
|
| cluster | string | yes | (empty) | This is the name of the cluster to which the consumers in this module belong. |
The [notifier] heading configures a method for sending out notifications of group status. It is not required to specify this section - Burrow can be used in a poll-only mode via the HTTP server. When it is specified, however, it must have a unique subheading specified, such as [notifier.mynotify]. The "mynotify" part is the name that will be used when writing logs about this module. You may configure as many notifier sections as you need to provide the alerts that you need.
[notifier.mynotify]
class-name="http"
interval=30
threshold=3
group-whitelist="^important-group-prefix.*$"
group-blacklist="^not-this-group$"
extras={ key1="value1", key2="value2" }2" }
send-close=true
template-open="path/to/file1.tmpl"
template-close="path/to/file2.tmpl"
(add additional class configs here)
| Key | Value Type | Required | Default Value | Description |
|---|---|---|---|---|
| class-name | string | yes | (empty) | This is the name of the cluster module type. The available classes are:
|
| interval | integer | no | 60 | The number of seconds to wait between sending notifications for a single group. |
| threshold | integer | no | 2 | The minimum group status to send out notifications for (refer to StatusConstant for values). |
| group-whitelist | string (regex) | no | (empty) | If specified, only send notifications for groups that match this regular expression. |
| group-blacklist | string (regex) | no | (empty) | If specified, only send notifications for groups that DO NOT match this regular expression. This is processed after the whitelist (if specified). |
| extras | map (string -> string) | no | (empty) | A map of values that will be available when compiling the template for the message to send. |
| send-close | boolean | no | false | If true, use the template-close template to send a notification when the group status drops to OK (after it has been WARN or above). |
| template-open | path | yes | (empty) | The path and filename of a template file that is used to compile the message to send for groups that pass the whitelist and status threshold (see Templates for more information on building the template). |
| template-close | path | no | (empty) | The path and filename of a template file that is used to compile the message to send for groups that transition to OK (if send-close is true). |