Set up template output plugin that writes to a file.

[davemarco]

Description

Added basic file output plugin. Plugin will serve as a template onto which IR and S3 output functionality will be added.

Plugin

File plugin is based on example stdout plugin provided by fluent-bit. The main file which implements required callbacks is plugins/out_clp_s3/out_clp_s3.go. I made the following changes to enhance functionality

1. Support for additional user-defined configuration values in config/config.go
2. Temporary modification so writes to file instead of stdout in plugins/out_clp_s3/flush/flush.go(code will be deleted on next PR)
3. Basic error handling in internal/utils/error.go

Development aids

In addition to plugin, I added

• Dockerfile plugins/out_clp_s3/Dockerfile
• Makefile plugins/out_clp_s3/Makefile

Docker

In addition to plugin, I added a dockerfile so new users/developers can test quickly. The dockerfile first builds the plugin in a go container and then copies the binary to a fluentbit container. The fluentbit container starts fluentbit with the output plugin and writes logs to the current directory using a docker volume.

The repo also contains two example fluent-bit configuration files. plugins/out_clp_s3/fluent-bit.conf is the main configuration file. plugins/out_clp_s3/plugin-config.conf provides the path to plugin binary. Both of these configuration are setup with paths that exists on the docker container, but can be modified for local use (however, modification may break docker setup)

Make

basic makefile to build plugin

Validation performed

I built plugin and tested can write to file locally, and with docker.

[davemarco]

FYI... Later on I noticed their "example stdout plugin" (which i copied the logic for this PR) can print strings as a byte array. There is no issue with cpu logs where values are ints tested with the configuration in this PR. I have looked into a solution for the next PR.

[davemarco]

• removed error.go and put error handling inline.
• Swapped makefile with taskfile

[davidlion]

Try to do a pass on the doc strings + comments. Many are not formatted correctly.

[davemarco]

Try to do a pass on the doc strings + comments. Many are not formatted correctly.

Made updates to comments and doc strings for line wrapping, capitalization and periods

[davidlion]

Lets tweak the title + commit message to:
Set up template output plugin that writes to a file. (#2)