Merge pull request #108 from bendrucker/metadata-reduce-namespace

Polish "reduce metadata" feature

Author: frankreno

Gemfile.lock

@@ -66,7 +66,7 @@ PLATFORMS
   ruby
 
 DEPENDENCIES
-  bundler (~> 1.3)
+  bundler (~> 2)
   codecov
   fluent-plugin-kubernetes_sumologic!
   rake
@@ -75,4 +75,4 @@ DEPENDENCIES
   webmock
 
 BUNDLED WITH
-   1.16.3
+   2.0.1

README.md

@@ -100,7 +100,7 @@ Environment | Variable Description
 `FLUENTD_USER_CONFIG_DIR`|A directory of user-defined fluentd configuration files, which must be in the  `*.conf` directory in the container.
 `FLUSH_INTERVAL` |How frequently to push logs to Sumo.<br/><br/>Default: `5s`
 `KUBERNETES_META`|Include or exclude Kubernetes metadata such as `namespace` and `pod_name` if using JSON log format. <br/><br/>Default: `true`
-`KUBERNETES_META_REDUCE`| Reduces Kubernetes metadata, check [Reducing Kubernetes Metadata](#####reduce-kubernetes-metadata-using-annotations). <br></br>Default: `false`
+`KUBERNETES_META_REDUCE`| Reduces redundant Kubernetes metadata, see [_Reducing Kubernetes Metadata_](#reducing-kubernetes-metadata). <br></br>Default: `false`
 `LOG_FORMAT`|Format in which to post logs to Sumo. Allowable values:<br/><br/>`text`—Logs will appear in SumoLogic in text format.<br/>`json`—Logs will appear in SumoLogic in json format.<br/>`json_merge`—Same as json but if the container logs in json format to stdout it will merge in the container json log at the root level and remove the log field.<br/><br/>Default: `json`
 `MULTILINE_START_REGEXP`|The regular expression for the `concat` plugin to use when merging multi-line messages. Defaults to Julian dates, for example, Jul 29, 2017.
 `NUM_THREADS`|Set the number of HTTP threads to Sumo. It might be necessary to do so in heavy-logging clusters. <br/><br/>Default: `1`
@@ -147,10 +147,29 @@ The following table show which  environment variables affect which Fluentd sourc
 ### FluentD stops processing logs
 When dealing with large volumes of data (TB's from what we have seen), FluentD may stop processing logs, but continue to run.  This issue seems to be caused by the [scalability of the inotify process](https://github.com/fluent/fluentd/issues/1630) that is packaged with the FluentD in_tail plugin.  If you encounter this situation, setting the `ENABLE_STAT_WATCHER` to `false` should resolve this issue.
 
-### Override environment variables using annotations
-You can override the `LOG_FORMAT`, `KUBERNETES_META_REDUCE`, `SOURCE_CATEGORY` and `SOURCE_NAME` environment variables, per pod, using [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/). For example:
+### Reducing Kubernetes metadata
 
-```
+You can use the `KUBERNETES_META_REDUCE` environment variable (global) or the `sumologic.com/kubernetes_meta_reduce` annotation (per pod) to reduce the amount of Kubernetes metadata included with each log line under the `kubernetes` field. 
+
+When set, FluentD will remove the following properties:
+
+* `pod_id`
+* `container_id`
+* `namespace_id`
+* `master_url`
+* `labels`
+* `annotations`
+
+Logs will still include:
+
+* `pod_name`
+* `container_name`
+* `namespace_name`
+* `host`
+
+These fields still allow you to uniquely identify a pod and look up additional details with the Kubernetes API.
+
+```yaml
 apiVersion: v1
 kind: ReplicationController
 metadata:
@@ -165,10 +184,7 @@ spec:
       labels:
         app: mywebsite
       annotations:
-        sumologic.com/format: "text"
         sumologic.com/kubernetes_meta_reduce: "true"
-        sumologic.com/sourceCategory: "mywebsite/nginx"
-        sumologic.com/sourceName: "mywebsite_nginx"
     spec:
       containers:
       - name: nginx
@@ -177,9 +193,10 @@ spec:
         - containerPort: 80
 ```
 
-### Reduce Kubernetes Metadata using annotations
-You can also use the "sumologic.com/kubernetes_meta_reduce" annotation to exclude `pod_id`, 
-`container_id`, `namespace_id`, `namespace_name`, `master_url` and `labels` from Kubernetes Metadata.
+
+### Override environment variables using annotations
+You can override the `LOG_FORMAT`, `KUBERNETES_META_REDUCE`, `SOURCE_CATEGORY` and `SOURCE_NAME` environment variables, per pod, using [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/). For example:
+
 ```
 apiVersion: v1
 kind: ReplicationController
@@ -195,7 +212,10 @@ spec:
       labels:
         app: mywebsite
       annotations:
+        sumologic.com/format: "text"
         sumologic.com/kubernetes_meta_reduce: "true"
+        sumologic.com/sourceCategory: "mywebsite/nginx"
+        sumologic.com/sourceName: "mywebsite_nginx"
     spec:
       containers:
       - name: nginx

fluent-plugin-kubernetes_sumologic.gemspec

@@ -20,7 +20,7 @@ Gem::Specification.new do |gem|
 
   gem.required_ruby_version = '>= 2.0.0'
 
-  gem.add_development_dependency "bundler", "~> 1.3"
+  gem.add_development_dependency "bundler", "~> 2"
   gem.add_development_dependency "rake"
   gem.add_development_dependency 'test-unit', '~> 3.1.0'
   gem.add_development_dependency "codecov", ">= 0.1.10"

lib/fluent/plugin/filter_kubernetes_sumologic.rb

@@ -183,7 +183,6 @@ def filter(tag, time, record)
           record["kubernetes"].delete("namespace_id")
           record["kubernetes"].delete("labels")
           record["kubernetes"].delete("master_url")
-          record["kubernetes"].delete("namespace_name")
           record["kubernetes"].delete("annotations")
         end
         if @add_stream == false

test/plugin/test_filter_kubernetes_sumologic.rb

@@ -458,6 +458,7 @@ def create_driver(conf = CONFIG)
             "container_name" => "log-format-labs",
             "pod_name" => "log-format-labs-54575ccdb9-9d677",
             "host" => "docker-for-desktop",
+            "namespace_name" => "default",
         },
         "_sumo_metadata" => {
             :category => "kubernetes/default/log/format/labs",
@@ -506,6 +507,7 @@ def create_driver(conf = CONFIG)
             "container_name" => "log-format-labs",
             "pod_name" => "log-format-labs-54575ccdb9-9d677",
             "host" => "docker-for-desktop",
+            "namespace_name" => "default",
         },
         "_sumo_metadata" => {
             :category => "kubernetes/default/log/format/labs",
@@ -557,6 +559,7 @@ def create_driver(conf = CONFIG)
             "container_name" => "log-format-labs",
             "pod_name" => "log-format-labs-54575ccdb9-9d677",
             "host" => "docker-for-desktop",
+            "namespace_name" => "default",
         },
         "_sumo_metadata" => {
             :category => "kubernetes/default/log/format/labs",