Skip to content

Configuration

Jump to bottom
Todd Palino edited this page Dec 1, 2017 · 25 revisions

Burrow is configured using viper, which handles configurations specified in multiple formats. This includes TOML, JSON, YAML, and others, but this configuration guide will focus on TOML. If you are not familiar with this format, please take a look at the reference documentation for TOML. It looks similar to INI configs, but there are some thing that might trip you up, like needing to have quotes around string values.

The configuration is organized into several subheadings. There is an example configuration file in the "config" directory of the Burrow distribution.

General

The [general] configuration heading covers the location of the PID file, as well as an optional place to put the stdout/stderr output.

[general]
pidfile="burrow.pid"
stdout-logfile="burrow.out"
access-control-allow-origin="mysite.example.com"
Key Value Type Required Default Value Description
pidfile path no burrow.pid The path and filename to store the PID of the running process in.
stdout-logfile path no (empty) If specified, it is the path and filename to redirect stdout and stderr into. If not specified, stdout/stderr are not redirected.
access-control-allow-origin string no (empty) The value to put in the Access-Control-Allow-Origin header in responses from the HTTP server in Burrow.

Logging

The [logging] configuration heading covers the configuration for logging. If no filename config is provided, all logs are written to stdout. If it is specified, logs are written out to the file configured using lumberjack and periodicially rolled.

[logging]
filename="logs/burrow.log"
level="info"
maxsize=100
maxbackups=30
maxage=10
use-localtime=false
use-compression=true
Key Value Type Required Default Value Description
filename path no (empty) The path and filename to write logs to. If not specified, logs are written to stdout.
level string no info The minimum log level to output. Must be one of "debug", "info", "warn", "error", "panic", "fatal".
maxsize integer no 100 The maximum size for a log file, in MB.
maxbackups integer no 10 The maximum number of log files to maintain.
maxage integer no 30 The maximum number of days to keep log files for.
use-localtime boolean no false If true, write log entries using the system's local time. Otherwise, use UTC.
use-compression boolean no false If true, compress rotated log files.

Zookeeper

The [zookeeper] heading specifies a Zookeeper ensemble to use in order to store metadata for modules and provide locking between multiple copies of Burrow. This allows you to run multiple copies of Burrow with only one being allowed to execute the notifiers at any time.

[zookeeper]
servers=["zkhost01.example.com:2181", "zkhost02.example.com:2181", "zkhost03.example.com:2181" ]
timeout=6
root-path=/mypath/burrow
Key Value Type Required Default Value Description
servers list of hostname:port yes (none) A list of strings in the form "hostname:port" that point to the servers in the Zookeeper ensemble. At least one is required.
timeout integer no 6 The expiration timeout for Zookeeper sessions, in seconds.
root-path znode no /burrow/notifier A full path to the znode that Burrow is allow to write under. The path will be created if it does not already exist.

Profiles

Profiles are used to generalize some configurations for the modules. These are items like TLS and SASL configs that may be used in multiple modules, to prevent duplication and error in the configs. While none of these are required, when they are provided they must each have a unique subheading (as described).

TLS Profile

The [tls] heading sets up a TLS (or SSL, if you're old school) profile that can be referenced by multiple modules to set up secure connections. It must have a unique subheading specified, such as [tls.myprofile], where "myprofile" is the name it will be referred to as. Multiple subheadings may be specified to have different TLS configurations.

[tls.myprofile]
certfile="certificate.crt"
keyfile="keyfile.key"
cafile="rootcerts.crt"
no-verify=false
Key Value Type Required Default Value Description
certfile path yes (empty) The path to the certificate file to be used (either a client certificate for client connections, or a server certificate for the HTTP server).
keyfile path yes (empty) The path to the key file that matches the certfile.
cafile path no (empty) If specified, it is the path to a bundle of certificates that provide the trusted root certificates for the connection.
no-verify boolean no false If true, do not validate the trust of the certificate sent by the other side of this connection.

SASL Profile

The [sasl] heading sets up a SASL profile that can be referenced by multiple modules for authenticating a connection. It must have a unique subheading specified, such as [sasl.myprofile], where "myprofile" is the name it will be referred to as. Multiple subheadings may be specified to have different SASL configurations.

[sasl.myprofile]
username="myname"
password="sekrit"
handshake-first=false
Key Value Type Required Default Value Description
username string yes (empty) The username to use for authentication.
password string yes (empty) The password to use for authentication.
handshake-first boolean no true If false, do not send SASL handshake first (only used for Sarama connections to Kafka when using a non-Kafka proxy).

HTTPServer

The [httpserver] heading configures the HTTP server in Burrow. If this heading is not specified, Burrow will start the server on a random available port (which will be printed in the log output). If it is specified, it must have a unique subheading specified, such as [httpserver.mylistener] (where "mylistener" is the subheading). Multiple subheadings may be specified in order to start the server listening on multiple addresses.

[httpserver]
address=":8080"
timeout=300
tls="tls-profile-name"
Key Value Type Required Default Value Description
address string yes (empty) This is a string of the form ip:port that specifies what the server should listen on. If the ip part is left blank, the listener attaches to all IPs on the host. If the port part is left blank, a random port is chosen.
timeout integer no 300 The timeout and keepalive setting for connections, in seconds.
tls string no (empty) If specified, it must be the name of a configured TLS profile and the listener is started with this configuration.

Storage

The [storage] heading configures the storage subsystem in Burrow. It is not required to specify this section, as Burrow will properly configure a storage module with the default values. If it is specified, it must have a unique subheading specified, such as [storage.mystorage] (where "mystorage" is the subheading). Only one storage section is permitted, however

[storage.mystorage]
class-name="inmemory"
intervals=10
expire-group=604800
workers=20
min-distance=1
group-whitelist=".*"
Key Value Type Required Default Value Description
class-name string yes (empty) This is the name of the module type. The only permitted value is "inmemory".
intervals integer no 10 The number of offsets to store for each partition.
expire-group integer no 604800 The number of seconds after which a group will be purged if it has not committed offsets.
workers integer no 20 The number of worker goroutines to start up. This number can be increased to provide more parallelism in the storage module.
min-distance integer no 0 Coalesce offset commits that are more frequent than this number of seconds. Used with the intervals configuration, this can be used to set a minimum time for the offset window (10 offsets x 15s = 150s minimum window).
group-whitelist string (regex) no (empty) If specified, ignore data for groups that do not match the regular expression.

Evaluator

The [evaluator] heading configures the evaluator subsystem in Burrow. It is not required to specify this section, as Burrow will properly configure a evaluator module with the default values. If it is specified, it must have a unique subheading specified, such as [evaluator.myeval] (where "myeval" is the subheading). Only one evaluator section is permitted, however

[evaluator.mystorage]
class-name="caching"
expire-cache=10
Key Value Type Required Default Value Description
class-name string yes (empty) This is the name of the module type. The only permitted value is "caching".
expire-cache integer no 604800 The number of seconds to cache evaluation results for.

Clusters

The [cluster] heading configures a single Kafka cluster to fetch topic and offset information from. It is not required to specify this section, but if you don't you won't have Burrow doing any work. When it is specified, it must have a unique subheading specified, such as [cluster.myclustername]. The "myclustername" part is the name by which this cluster will be referred to in other parts of the configuration, as well as in HTTP requests and notifications. You may configure as many cluster sections as you want (and Burrow is efficient at handling multiple Kafka clusters). Note that the cluster configuration only retrieves the topic list and cluster offsets. To get consumer group information, you must also configure a consumer section.

[cluster.myclustername]
class-name="kakfa"
(add additional class configs here)
Key Value Type Required Default Value Description
class-name string yes (empty) This is the name of the cluster module type. The available classes are:

Consumers

The [consumer] heading configures a single repository to retrieve consumer group offsets and information from. It is not required to specify this section, but if you don't you won't have Burrow doing much work. When it is specified, it must have a unique subheading specified, such as [consumer.myconsumers]. The "myconsumers" part is the name that will be used when writing logs about this module. You may configure as many consumer sections as you want for a given cluster.

[consumer.myconsumers]
class-name="kakfa"
cluster="myclustername"
(add additional class configs here)
Key Value Type Required Default Value Description
class-name string yes (empty) This is the name of the cluster module type. The available classes are:

Notifiers

The [notifier] heading configures a method for sending out notifications of group status. It is not required to specify this section - Burrow can be used in a poll-only mode via the HTTP server. When it is specified, however, it must have a unique subheading specified, such as [notifier.mynotify]. The "mynotify" part is the name that will be used when writing logs about this module. You may configure as many notifier sections as you need to provide the alerts that you need.

[notifier.mynotify]
class-name="http"
interval=30
threshold=3
group-whitelist="^important-group-prefix.*$"
group-blacklist="^not-this-group$"
extras={ key1="value1", key2="value2" }2" }
send-close=true
template-open="path/to/file1.tmpl"
template-close="path/to/file2.tmpl"
(add additional class configs here)
Key Value Type Required Default Value Description
class-name string yes (empty) This is the name of the cluster module type. The available classes are:
  • http - Send outbound HTTP request Click here for additional configurations needed
  • email - Send email. Click here for additional configurations needed | | interval | integer | no | 60 | The number of seconds to wait between sending notifications for a single group. | | threshold | integer | no | 2 | The minimum group status to send out notifications for (refer to StatusConstant for values). | | group-whitelist | string (regex) | no | (empty) | If specified, only send notifications for groups that match this regular expression. | | group-blacklist | string (regex) | no | (empty) | If specified, only send notifications for groups that DO NOT match this regular expression. This is processed after the whitelist (if specified). | | extras | map (string -> string) | no | (empty) | A map of values that will be available when compiling the template for the message to send. | | send-close | boolean | no | false | If true, use the template-close template to send a notification when the group status drops to OK (after it has been WARN or above). | | template-open | path | yes | (empty) | The path and filename of a template file that is used to compile the message to send for groups that pass the whitelist and status threshold (see Templates for more information on building the template). | | template-close | path | no | (empty) | The path and filename of a template file that is used to compile the message to send for groups that transition to OK (if send-close is true). |
Clone this wiki locally