From c37e4e06b2351487a227a51cf294e4cd8bc892e5 Mon Sep 17 00:00:00 2001 From: Phantom-Intruder Date: Thu, 8 Feb 2024 14:35:22 +0530 Subject: [PATCH 1/7] FluentBit integration --- Logging101/filebeat-sidecar.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/Logging101/filebeat-sidecar.md b/Logging101/filebeat-sidecar.md index adf49f3e..a2826591 100644 --- a/Logging101/filebeat-sidecar.md +++ b/Logging101/filebeat-sidecar.md @@ -111,4 +111,8 @@ kubectl describe pod --watch You should see two containers being described by this command under the `Containers` section. Watch as the state of both containers goes from `pending` to `running`. When the container running the sleep command goes to a `successful` state, the container running filebeat should immediately. Both pods will then go into a `Terminating` state before the pod itself terminates and leaves. -This brings us to the end of this section on logging with filebeat sidecars. You can use the same concept with similar tools such as fluentd if you plan to scale up your jobs/logs massively. Just make sure that there are no bottlenecks in any other points such as logstash and elasticsearch. \ No newline at end of file +This brings us to the end of this section on logging with filebeat sidecars. You can use the same concept with similar tools such as fluentd if you plan to scale up your jobs/logs massively. Just make sure that there are no bottlenecks in any other points such as logstash and elasticsearch. + +We have already covered fluent bit, so you know that it is way more lightweight than either filebeat or fluentd. In fact, according to [benchmarks by AWS](https://aws.amazon.com/blogs/opensource/centralized-container-logging-fluent-bit/), fluent bit uses 5 times less memory compared to fluentd, and 2 times less than filebeat. So in the case where we run hundreds of jobs at the same time, it makes a lot of sense to use a logger that pushes all your log lines with as little resource consumption as possible since we will be creating a logger instance per each pod. + +In the next section, let's take a look at how we can use fluentbit as a sidecar container to push logs. \ No newline at end of file From 5b2d1295f83aed78df5a3310259d636006d31474 Mon Sep 17 00:00:00 2001 From: Phantom-Intruder Date: Fri, 9 Feb 2024 14:10:08 +0530 Subject: [PATCH 2/7] FluentBit integration --- Logging101/filebeat-sidecar.md | 4 +++- Logging101/fluentbit-sidecar.md | 40 +++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-) create mode 100644 Logging101/fluentbit-sidecar.md diff --git a/Logging101/filebeat-sidecar.md b/Logging101/filebeat-sidecar.md index a2826591..24319b60 100644 --- a/Logging101/filebeat-sidecar.md +++ b/Logging101/filebeat-sidecar.md @@ -115,4 +115,6 @@ This brings us to the end of this section on logging with filebeat sidecars. You We have already covered fluent bit, so you know that it is way more lightweight than either filebeat or fluentd. In fact, according to [benchmarks by AWS](https://aws.amazon.com/blogs/opensource/centralized-container-logging-fluent-bit/), fluent bit uses 5 times less memory compared to fluentd, and 2 times less than filebeat. So in the case where we run hundreds of jobs at the same time, it makes a lot of sense to use a logger that pushes all your log lines with as little resource consumption as possible since we will be creating a logger instance per each pod. -In the next section, let's take a look at how we can use fluentbit as a sidecar container to push logs. \ No newline at end of file +In the next section, let's take a look at how we can use fluentbit as a sidecar container to push logs. + +[Next: fluentbit as a sidecar container](./fluentbit-sidecar.md) \ No newline at end of file diff --git a/Logging101/fluentbit-sidecar.md b/Logging101/fluentbit-sidecar.md new file mode 100644 index 00000000..542324eb --- /dev/null +++ b/Logging101/fluentbit-sidecar.md @@ -0,0 +1,40 @@ +## fluentbit as a sidecar container + +The concept behind the fluentbit sidecar container will be basically the same as with the filebeat sidecar. The differences will be in the fluentbit conf since that will obviously use a different syntax, and the way the fluentbit container will be loaded into the pod. + +``` +apiVersion: v1 +kind: ConfigMap +metadata: + name: fluentbit-configmap +data: + fluent-bit.conf: | + [SERVICE] + Flush 1 + Log_Level info + Daemon off + + [INPUT] + Name tail + Path /data/*.log + Tag mixlog + + [OUTPUT] + Name http + Match mixlog + Host logstash-logstash + Port 5044 +``` + +``` +- name: fluent-bit-sidecar + image: cr.fluentbit.io/fluent/fluent-bit:2.2.2 + volumeMounts: + - name: fluent-bit-config + mountPath: /fluent-bit/etc/ + readOnly: true + - name: shared-data + mountPath: /data/ + command: ["/fluent-bit/bin/fluent-bit"] + args: ["-c", "/fluent-bit/etc/fluent-bit.conf"] + ``` \ No newline at end of file From 67bb875f5e494a3909f5d6bd2a53784f80c1e683 Mon Sep 17 00:00:00 2001 From: Phantom-Intruder Date: Sat, 10 Feb 2024 11:39:34 +0530 Subject: [PATCH 3/7] flient bit sidecar cont. --- Logging101/fluentbit-sidecar.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Logging101/fluentbit-sidecar.md b/Logging101/fluentbit-sidecar.md index 542324eb..424b42de 100644 --- a/Logging101/fluentbit-sidecar.md +++ b/Logging101/fluentbit-sidecar.md @@ -1,6 +1,6 @@ ## fluentbit as a sidecar container -The concept behind the fluentbit sidecar container will be basically the same as with the filebeat sidecar. The differences will be in the fluentbit conf since that will obviously use a different syntax, and the way the fluentbit container will be loaded into the pod. +The concept behind the fluentbit sidecar container will be basically the same as with the filebeat sidecar. The differences will be in the fluentbit conf since that will obviously use a different syntax and the way the fluentbit container will be loaded into the pod. We will continue to use the same Ubuntu job that we were using before, and the same concept of using a completion flag to tell when the container should stop will continue to apply. We will also be using the same shared volume, and we will be using a ConfigMap to load the fluent bit conf as well. Below is the fluent bit conf that matches the filebeat config that we had in the previous section: ``` apiVersion: v1 @@ -26,6 +26,8 @@ data: Port 5044 ``` +A + ``` - name: fluent-bit-sidecar image: cr.fluentbit.io/fluent/fluent-bit:2.2.2 From da769db4c885019986acc065b2ee3b0203de2ec9 Mon Sep 17 00:00:00 2001 From: Phantom-Intruder Date: Sun, 11 Feb 2024 11:32:11 +0530 Subject: [PATCH 4/7] fluent bit cont --- Logging101/fluentbit-sidecar.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Logging101/fluentbit-sidecar.md b/Logging101/fluentbit-sidecar.md index 424b42de..6f31f823 100644 --- a/Logging101/fluentbit-sidecar.md +++ b/Logging101/fluentbit-sidecar.md @@ -26,7 +26,7 @@ data: Port 5044 ``` -A +The first five lines are already familiar to you. We then start the fluentbit config. We first have some information on the service, followed by the definition of the input. As with before, we use the tail plugin to get all the log files found in /data/ and tag them with the tag "mixlog". We then match these tagged items in the output plugin and stream the logs into the logstash service. You will notice that while filebeat natively had an input source to logstash called "beats", fluent bit does not. However, we can use "http" to do this instead. From the logstash side, you will have to change the input to point to use "http" instead of "beats", but apart from that, everything should work just fine. ``` - name: fluent-bit-sidecar From f327dc9fc33cbfebbf5000037a6d66ee4d244b53 Mon Sep 17 00:00:00 2001 From: Phantom-Intruder Date: Tue, 13 Feb 2024 12:09:58 +0530 Subject: [PATCH 5/7] fluentbit sidecar --- Logging101/fluentbit-sidecar.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/Logging101/fluentbit-sidecar.md b/Logging101/fluentbit-sidecar.md index 6f31f823..443138a3 100644 --- a/Logging101/fluentbit-sidecar.md +++ b/Logging101/fluentbit-sidecar.md @@ -28,6 +28,8 @@ data: The first five lines are already familiar to you. We then start the fluentbit config. We first have some information on the service, followed by the definition of the input. As with before, we use the tail plugin to get all the log files found in /data/ and tag them with the tag "mixlog". We then match these tagged items in the output plugin and stream the logs into the logstash service. You will notice that while filebeat natively had an input source to logstash called "beats", fluent bit does not. However, we can use "http" to do this instead. From the logstash side, you will have to change the input to point to use "http" instead of "beats", but apart from that, everything should work just fine. +Now let's look at what should be done from the Kubernetes manifest side. It will be basically the same thing as what we had with filebeat, except we will use the fluent bit image. We will also be pointing the overriding config to fluent-bit.conf which will be mounted in a shared volume, the same as the filebeat yaml. Apart from that, everything will be the same. + ``` - name: fluent-bit-sidecar image: cr.fluentbit.io/fluent/fluent-bit:2.2.2 @@ -39,4 +41,16 @@ The first five lines are already familiar to you. We then start the fluentbit co mountPath: /data/ command: ["/fluent-bit/bin/fluent-bit"] args: ["-c", "/fluent-bit/etc/fluent-bit.conf"] - ``` \ No newline at end of file + ``` + +Now that we have covered both areas that need to be changed, let's go ahead and give this a test run. First off, deploy the ConfigMap: + +``` +kubectl apply -f fluentbit-configmap.yaml +``` + +Next, apply the deployment.yaml: + +``` +kubectl apply -f non-parallel-job.yml +``` \ No newline at end of file From 1c66234f2a96ff2454bf6fc72bf784358872c0d9 Mon Sep 17 00:00:00 2001 From: Phantom-Intruder Date: Wed, 14 Feb 2024 12:07:31 +0530 Subject: [PATCH 6/7] fluentbit sidecar --- Logging101/filebeat-sidecar.md | 2 +- Logging101/fluentbit-sidecar.md | 18 ++++++++++++++++-- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/Logging101/filebeat-sidecar.md b/Logging101/filebeat-sidecar.md index 24319b60..0ef1a250 100644 --- a/Logging101/filebeat-sidecar.md +++ b/Logging101/filebeat-sidecar.md @@ -109,7 +109,7 @@ Note the name of the pod, and use it in the below command: kubectl describe pod --watch ``` -You should see two containers being described by this command under the `Containers` section. Watch as the state of both containers goes from `pending` to `running`. When the container running the sleep command goes to a `successful` state, the container running filebeat should immediately. Both pods will then go into a `Terminating` state before the pod itself terminates and leaves. +You should see two containers being described by this command under the `Containers` section. Watch as the state of both containers goes from `pending` to `running`. When the container running the sleep command goes to a `successful` state, the container running filebeat should immediately stop. Both pods will then go into a `Terminating` state before the pod itself terminates and leaves. This brings us to the end of this section on logging with filebeat sidecars. You can use the same concept with similar tools such as fluentd if you plan to scale up your jobs/logs massively. Just make sure that there are no bottlenecks in any other points such as logstash and elasticsearch. diff --git a/Logging101/fluentbit-sidecar.md b/Logging101/fluentbit-sidecar.md index 443138a3..25e312d8 100644 --- a/Logging101/fluentbit-sidecar.md +++ b/Logging101/fluentbit-sidecar.md @@ -40,7 +40,7 @@ Now let's look at what should be done from the Kubernetes manifest side. It will - name: shared-data mountPath: /data/ command: ["/fluent-bit/bin/fluent-bit"] - args: ["-c", "/fluent-bit/etc/fluent-bit.conf"] + args: ["-c", "/fluent-bit/etc/fluent-bit.conf & while [ ! -f /data/completion-flag ]; do sleep 1; done && exit 0"] ``` Now that we have covered both areas that need to be changed, let's go ahead and give this a test run. First off, deploy the ConfigMap: @@ -53,4 +53,18 @@ Next, apply the deployment.yaml: ``` kubectl apply -f non-parallel-job.yml -``` \ No newline at end of file +``` + +Now let's observe the containers in the same way we did with the filebeat sidecars. + +``` +kubectl get po +``` + +Note the name of the pod, and use it in the below command: + +``` +kubectl describe pod --watch +``` + +You should see two containers being described by this command under the `Containers` section. Watch as the state of both containers goes from `pending` to `running`. When the container running the sleep command goes to a `successful` state, the container running fluentbit should immediately stop. Both pods will then go into a `Terminating` state before the pod itself terminates and leaves. \ No newline at end of file From 40a944e74caca6c20e889a2ee3b1fb4afcdffdb6 Mon Sep 17 00:00:00 2001 From: Phantom-Intruder Date: Thu, 15 Feb 2024 13:16:58 +0530 Subject: [PATCH 7/7] fluentbit sidecar --- Logging101/fluentbit-sidecar.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/Logging101/fluentbit-sidecar.md b/Logging101/fluentbit-sidecar.md index 25e312d8..beef4ce1 100644 --- a/Logging101/fluentbit-sidecar.md +++ b/Logging101/fluentbit-sidecar.md @@ -67,4 +67,10 @@ Note the name of the pod, and use it in the below command: kubectl describe pod --watch ``` -You should see two containers being described by this command under the `Containers` section. Watch as the state of both containers goes from `pending` to `running`. When the container running the sleep command goes to a `successful` state, the container running fluentbit should immediately stop. Both pods will then go into a `Terminating` state before the pod itself terminates and leaves. \ No newline at end of file +You should see two containers being described by this command under the `Containers` section. Watch as the state of both containers goes from `pending` to `running`. When the container running the sleep command goes to a `successful` state, the container running fluentbit should immediately stop. Both pods will then go into a `Terminating` state before the pod itself terminates and leaves. + +## Conclusion + +This brings us to the end of the section on running fluent bit as a sidecar container. Now, you may be asking the question: if fluentbit does the same things as filebeat with a much smaller resource footprint, why use filebeat at all? The answer to this is features. For example, logstash supports the Beats protocol natively. However, it does not do this for fluentbit. Instead, you will have to use HTTP, which might mess up the output that is presented in Kibana. Larger loggers such as fluentd support in-built grok parsing which fluentbit doesn't. Instead, you will have to push logs from fluent bit to fluentd (or logstash as we do here), which adds another resource that acts as a mediator. Since logstash also handles buffering so that elasticsearch doesn't get overwhelmed, this isn't a particularly terrible idea. Additionally, you might notice that fluent bit does not have tools like bash or sh, which means that if you want to look inside the fluent bit container for some reason, you won't be able to do so. + +So there is a trade-off and you will have to consider what is best for your use case. \ No newline at end of file