Upload batches of records to S3 instead of writing to file; Upgrade input validation using Go validator package.

[davemarco]

Description

Plugin can now send chunks to aws s3. In addition, the input validation has been heavily upgraded. Now supports default settings, optional settings, and input parsing. I also cleaned up READMEs and added a table for settings and defaults.

AWS S3

Plugin loads credentials using standard hierarchy described here

In addition to standard chain, I added the ability to assume a role. The fluent bit s3 plugin can do this, and I was already familiar with roles for clp cloud, so i added this feature.

I added some checks during startup to make sure credentials are good, and bucket exists . If checks fail, wont start collecting logs and will exit.

The file name is as follows: $bucketPrefix/$tag_$time_$ID.zst
The time is the system time in nanoseconds. the ID can be set by user, but if they leave it out, it is a UUID. This should help prevent collision if multiple collectors are sending to same path.

I removed file stuff, but may add back as another plugin later. In general, I think it is just useful for debugging and not having to send stuff to s3.

Input validation

Using go validator package to help with input validation. Fancier features being used are timezone and string parsing. Timezone checks if the IR timezone is a valid timezone. String parsing is being used to confirm role_arn has the correct prefix. More can be added later on.

There were some issues in that fluent-bit only supplies string type, and does not supply boolean type. Some of the code is confusing because of this, let me know if questions.

Validation performed

• Sent logs to test bucket using regular setup and docker.
• Tested role_arn (assume IAM role) setting works as expected with a test role.
• Tested startup checks with invalid credentials/buckets

[kirkrodrigues]

Approving on behalf of @davidlion.