fluent-plugin-forest

This is a plugin for Fluentd.

Component

ForestOutput

ForestOutput creates sub plugin instance of a output plugin dynamically per tag, from template configurations. In template configurations, you can write configuration lines for overall tags by <template>, and for specified tags by <case TAG_PATTERN>, and you can use __TAG__ (or ${tag}) placeholder at anywhere in <template> and <case>.

This plugin helps you if you are writing very long configurations by copy&paste with a little little diff for many tags.

Other supported placeholders:

• __HOSTNAME__ (or ${hostname})
  • replaced with string specified by 'hostname' configuration value, or (default) result of 'hostname' command
• __ESCAPED_TAG__ (or ${escaped_tag})
  • replaced with escaped tag. Escaped tag is replaced '.' with a character specified by 'escape_tag_separator' (default: '_')
• __TAG_PARTS[n]__ (or ${tag_parts[n]})
  • it acts accessing the index which split the tag with '.' (dot). It will get 'td' by ${tag_parts[0]}, 'apache' by ${tag_parts[1]} and 'access' by ${tag_parts[-1]} when the tag was td.apache.access.
  • you can also use range index like '1..4' or '1...3' (e.g. ${tag_parts[1..-1]} will return 'apache.access' on giving tag same as previous.)

You SHOULD NOT use ForestOutput for tags increasing infinitly.

Configuration

NOTICE: If you configure fluent-plugin-forest with buffer_type file (or plugins, default buffer type is file), you should modify buffer_path with __TAG__ (or ${tag}) to help to use buffer files for each tags.

ForestOutput

If you are writing long long configurations like below:

<match service.blog>
  @type file
  time_slice_format %Y%m%d%H
  compress gz
  path /var/log/blog.*.log
</match>
<match service.portal>
  @type file
  time_slice_format %Y%m%d%H
  compress gz
  path /var/log/portal.*.log
</match>
<match service.news>
  @type file
  time_slice_format %Y%m%d%H
  compress gz
  path /var/log/news.*.log
</match>
<match service.sns>
  @type file
  time_slice_format %Y%m%d%H
  compress gz
  path /var/log/sns.*.log
</match>
# ...

You can write configuration with ForestOutput like below:

<match service.*>
  @type forest
  subtype file
  remove_prefix service
  <template>
    time_slice_format %Y%m%d%H
    compress gz
    path /var/log/${tag}.*.log
  </template>
</match>

If you want to place logs /var/archive for service.search.** as filename with hostname, without compression, case directive is useful:

<match service.*>
  @type forest
  subtype file
  remove_prefix service
  <template>
    time_slice_format %Y%m%d%H
  </template>
  <case search.**>
    path /var/archive/__TAG__.__HOSTNAME__.*.log
  </case>
  <case *>
    compress gz
    path /var/log/__TAG__.*.log
  </case>
</match>

case configuration overwrites template configuration, so you can also write like this:

<match service.*>
  @type forest
  subtype file
  remove_prefix service
  <template>
    time_slice_format %Y%m%d%H
    compress gz
    path /var/log/__TAG__.*.log
  </template>
  <case search.**>
    path /var/archive/__TAG__.*.log
  </case>
</match>

Version 0.2.0 or later, subsections adding/overwriting are supported. About the case below, three <store> subsections are defined for search.** pattern.

<match service.*>
  @type forest
  subtype copy
  <template>
    <store>
      @type file
      path /path/to/copy1
    </store>
    <store>
      @type file
      path /path/to/copy2
    </store>
  </template>
  <case search.**>
    <store>
      @type file
      path /path/to/copy3
    </store>
  </case>
</match>

Subsections with same arguments will be overwritten. See this example:

<match service.*>
  @type forest
  subtype route
  <template>
    <route {search,admin}.a>
      add_prefix first
    </route>
    <route {search,admin}.b>
      add_prefix second
    </route>
    <route {search,admin}.c>
      add_prefix third
    </route>
    <route {search,admin}.*>
      add_prefix extra
    </route>
  </template>
  <case admin.*>
    <route {search,admin}.*>
      add_prefix other
    </route>
  </case>
</match>

In this case, <route {search,admin}.*> subsection will be overwritten to add prefix 'other' for tag service.admin.*.

For Configuration DSL

In DSL configurations, case is reserved by Ruby itself. Use pattern instead of case. pattern works just as same as case. (v0.3.0 or later.)

TODO

• patches welcome!

Copyright

• Copyright (c) 2012- TAGOMORI Satoshi (tagomoris)
• License
  • Apache License, Version 2.0