Skip to content

LTSV Decoder

Jump to bottom Edit New page
Frank Denis edited this page Sep 23, 2015 · 7 revisions

Label Tab-separated Values is a simple and clever format to encode a set of key/value pairs.

A key/value pair is represented by: key:value. The split happens right after the first column (:), so it is perfectly valid for a value to contain columns.

Key/value pairs are delimited by TAB characters. Therefore, keys can contain any characters with the exception of : and TAB, whereas values can contain any characters with the exception of TAB and the records delimiter (usually \n).

Keys and values must be valid UTF-8 strings.

Generating LTSV records is extremely fast and easy. Values containing TAB characters can usually just replace these with spaces.

Flowgger expects LTSV records to match some very basic constraints:

  • A message MUST include a timestamp, under a key named time. The timestamp can be expressed in RFC 3339 format or in English format.
  • A message MUST include a source host name, under the key host.
  • A message MAY include a description, under the key message.
  • A message MAY include a severity level, as a number (between 0 and 7, matching syslog severity levels), under the key level.
  • A message MAY include any number of additional key/value pairs.

Here is an example of valid LTSV record (\t has to be replaced with actual TAB characters):

time:[2015-08-05T15:53:45.637824Z]\thost:testhostname\tname1:value1\tname 2: value 2\tn3:v3

In addition to being easy to generate, LTSV is also the fastest option in Flowgger.

LTSV schemas

By design, and unlike JSON-based formats, values in LTSV records are not typed, and are assumed to be strings by default.

However, it may be desirable to enforce type constraints, and to retains the types when converting LTSV to typed formats such as JSON.

In order to do so, a schema can be defined for LTSV inputs, in an [input.ltsv_schema] section of the Flowgger configuration file:

[input.ltsv_schema]
counter = "u64"
amount = "f64"

Supported types are:

  • string
  • bool (boolean value)
  • f64 (floating-point number)
  • i64 (signed integer)
  • u64 (unsigned integer)

Pay attention to the fact that some of these values may not have a representation in the target format. For example, Javascript, hence JSON (hence GELF) can only represent values up to 2^53-1 without losing precision.

/EOF

Clone this wiki locally