From 50ba7100a59345e337554c7400e2652737d97d7c Mon Sep 17 00:00:00 2001 From: Matej Gera Date: Mon, 2 Oct 2023 16:45:27 +0200 Subject: [PATCH 1/2] Fix README link Signed-off-by: Matej Gera --- otel-ecs-fargate/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/otel-ecs-fargate/README.md b/otel-ecs-fargate/README.md index 302d483a..feeb8743 100644 --- a/otel-ecs-fargate/README.md +++ b/otel-ecs-fargate/README.md @@ -74,7 +74,7 @@ Example container declaration within a Task Definition: ] ``` -In the example above, you'll need to set two instances each of and . The logConfiguration section included in the example will forward OTEL logs to the Coralogix platform, as documented in our fluentbit log processing configuration instructions [here](../../logs/fluent-bit/ecs-fargate/README.md). If you don't want to have them submitted to the Coralogix platform, you can replace the logConfiguration with whichever logDriver configuration you would prefer. To submit to Cloudwatch, you can configure as so: +In the example above, you'll need to set two instances each of and . The logConfiguration section included in the example will forward OTEL logs to the Coralogix platform, as documented in our fluentbit log processing configuration instructions [here](../logs/fluent-bit/ecs-fargate/README.md). If you don't want to have them submitted to the Coralogix platform, you can replace the logConfiguration with whichever logDriver configuration you would prefer. To submit to Cloudwatch, you can configure as so: ``` "logConfiguration": { From d8ea49eb2468c75cce6eec514e40ae4403fd0c0b Mon Sep 17 00:00:00 2001 From: Matej Gera Date: Mon, 2 Oct 2023 16:55:32 +0200 Subject: [PATCH 2/2] Format all docs Signed-off-by: Matej Gera --- logs/fluent-bit/ecs-fargate/README.md | 5 +++-- logs/fluent-bit/eks-fargate/CHANGELOG.md | 2 +- logs/fluent-bit/eks-fargate/README.md | 6 +++--- otel-agent/k8s-helm/CHANGELOG.md | 1 - otel-ecs-fargate/README.md | 15 ++++++++++++--- otel-eks-fargate/CHANGELOG.md | 2 +- otel-eks-fargate/README.md | 14 +++++++------- 7 files changed, 27 insertions(+), 18 deletions(-) diff --git a/logs/fluent-bit/ecs-fargate/README.md b/logs/fluent-bit/ecs-fargate/README.md index 77e18997..3cf8a31d 100644 --- a/logs/fluent-bit/ecs-fargate/README.md +++ b/logs/fluent-bit/ecs-fargate/README.md @@ -8,7 +8,7 @@ The aws-for-fluent-bit image, [maintained here by AWS](https://github.com/aws/aw The base_filters.conf file includes a set of filters to ensure proper ingestion by our backend. This should be included as the first configuration file for your instance deployment. Ensure you upload this to an S3 bucket in your AWS account. -As I just alluded to, you can load multiple configuration files from S3 to build your final configuration. This is done by setting custom environment variables within the task definition. +As I just alluded to, you can load multiple configuration files from S3 to build your final configuration. This is done by setting custom environment variables within the task definition. Example container declaration within a Task Definition: @@ -71,6 +71,7 @@ In order to allow container access to the S3 object, you'll need to provide the ] } ``` + Note: Don't confuse Task Execution Role for Task Role, this permission needs to be added to the Task Role. (Contrary to the ADOT (OTEL) Metrics and Traces integration) After you've added the above container to your existing Task Definition, you need to adjust the logConfiguration for the containers you wish to forward to Coralogix. @@ -92,4 +93,4 @@ To do so, you'd add this "logConfiguration" section to each of your application "compress": "gzip" } } -``` \ No newline at end of file +``` diff --git a/logs/fluent-bit/eks-fargate/CHANGELOG.md b/logs/fluent-bit/eks-fargate/CHANGELOG.md index 2caa0327..8458c0d0 100644 --- a/logs/fluent-bit/eks-fargate/CHANGELOG.md +++ b/logs/fluent-bit/eks-fargate/CHANGELOG.md @@ -4,4 +4,4 @@ ### v0.0.1 / 2023-08-17 -* [DOC] Documented EKS Fargate fluent-bit (log_router) integration \ No newline at end of file +* [DOC] Documented EKS Fargate fluent-bit (log_router) integration diff --git a/logs/fluent-bit/eks-fargate/README.md b/logs/fluent-bit/eks-fargate/README.md index 4c70b110..12f5e19a 100644 --- a/logs/fluent-bit/eks-fargate/README.md +++ b/logs/fluent-bit/eks-fargate/README.md @@ -97,7 +97,7 @@ data: Time_Keep true ``` -Details on the workings of the AWS log router can be found in the configuration documentation on the AWS docs site here: +Details on the workings of the AWS log router can be found in the configuration documentation on the AWS docs site here: [Fargate logging - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html) @@ -105,5 +105,5 @@ Details on the workings of the AWS log router can be found in the configuration - The log_router will only be attached to workloads started after the manifest has been applied. **You will need to restart your pods** in order for the log_router to start forwarding the logs to your Firehose. - You can add additional filters, but they are limited to the following types: -`grep, parser, record_modifier, rewrite_tag, throttle, nest, modify, kubernetes` -- If you wish to add a Cloudwatch output, you’ll have to add additional permissions to your Fargate profile. Please review the above linked AWS documentation for details. \ No newline at end of file + `grep, parser, record_modifier, rewrite_tag, throttle, nest, modify, kubernetes` +- If you wish to add a Cloudwatch output, you’ll have to add additional permissions to your Fargate profile. Please review the above linked AWS documentation for details. diff --git a/otel-agent/k8s-helm/CHANGELOG.md b/otel-agent/k8s-helm/CHANGELOG.md index 40032af9..2085eb9d 100644 --- a/otel-agent/k8s-helm/CHANGELOG.md +++ b/otel-agent/k8s-helm/CHANGELOG.md @@ -8,7 +8,6 @@ * [FEATURE] Set integrationName as resource attribute * [CHORE] Upgrading upstream chart. (v0.71.2) - ### v0.0.33 / 2023-09-04 * [CHORE] Upgrading upstream chart. (v0.71.1) diff --git a/otel-ecs-fargate/README.md b/otel-ecs-fargate/README.md index feeb8743..33896728 100644 --- a/otel-ecs-fargate/README.md +++ b/otel-ecs-fargate/README.md @@ -2,7 +2,7 @@ The OpenTelemetry collector offers a vendor-agnostic implementation of how to receive, process and export telemetry data. -In this document, we'll explain how to add the OTEL collector as a sidecar agent to your ECS Task Definitions. We use an AWS customized OpenTelemetry image called AWS Distro for OpenTelemetry (ADOT), as it has several features that allow for more convenient management of the configuration. We also have an example cloudformation template for review [here](https://github.com/coralogix/cloudformation-coralogix-aws/tree/master/aws-integrations/ecs-fargate) +In this document, we'll explain how to add the OTEL collector as a sidecar agent to your ECS Task Definitions. We use an AWS customized OpenTelemetry image called AWS Distro for OpenTelemetry (ADOT), as it has several features that allow for more convenient management of the configuration. We also have an example cloudformation template for review [here](https://github.com/coralogix/cloudformation-coralogix-aws/tree/master/aws-integrations/ecs-fargate) The ADOT image, [maintained here by AWS](https://github.com/aws-observability/aws-otel-collector), allows for loading of the OpenTelemetry configuration via Systems Manager Parameter Stores. This makes adjusting your configuration more convenient and more dynamic than baking a static configuration into your container image. @@ -74,7 +74,15 @@ Example container declaration within a Task Definition: ] ``` -In the example above, you'll need to set two instances each of and . The logConfiguration section included in the example will forward OTEL logs to the Coralogix platform, as documented in our fluentbit log processing configuration instructions [here](../logs/fluent-bit/ecs-fargate/README.md). If you don't want to have them submitted to the Coralogix platform, you can replace the logConfiguration with whichever logDriver configuration you would prefer. To submit to Cloudwatch, you can configure as so: +In the example above, you'll need to set two instances each of + + + +and + + + +. The logConfiguration section included in the example will forward OTEL logs to the Coralogix platform, as documented in our fluentbit log processing configuration instructions [here](../logs/fluent-bit/ecs-fargate/README.md). If you don't want to have them submitted to the Coralogix platform, you can replace the logConfiguration with whichever logDriver configuration you would prefer. To submit to Cloudwatch, you can configure as so: ``` "logConfiguration": { @@ -106,6 +114,7 @@ In order to allow container access to the Systems Manager Parameter Store, you'l ] } ``` + Note: Don't confuse Task Role for Task Execution Role, this permission needs to be added to the Task Execution Role. (Contrary to the fluentbit Logs integration) -After adding the above container to your existing Task Definition, your applications can submit their traces and metrics exports to http://localhost:4318/v1/traces and /v1/metrics. It will also collect container metrics from all containers in the Task Definition. \ No newline at end of file +After adding the above container to your existing Task Definition, your applications can submit their traces and metrics exports to http://localhost:4318/v1/traces and /v1/metrics. It will also collect container metrics from all containers in the Task Definition. diff --git a/otel-eks-fargate/CHANGELOG.md b/otel-eks-fargate/CHANGELOG.md index 5033d1b2..3ee42f4e 100644 --- a/otel-eks-fargate/CHANGELOG.md +++ b/otel-eks-fargate/CHANGELOG.md @@ -5,4 +5,4 @@ ### v0.0.1 / 2023-08-17 * [NEW] Added EKS Fargate related configuration files -* [DOC] Documented EKS Fargate OTEL integration \ No newline at end of file +* [DOC] Documented EKS Fargate OTEL integration diff --git a/otel-eks-fargate/README.md b/otel-eks-fargate/README.md index 5424926a..597596ca 100644 --- a/otel-eks-fargate/README.md +++ b/otel-eks-fargate/README.md @@ -9,19 +9,19 @@ The Coralogix EKS Fargate integration for Metrics and Traces leverages OpenTelem ## Requirements: - `cx-eks-fargate-otel` namespace declared in your EKS cluster - - This namespace need not be hosted by a Fargate profile, but if it is desired, you’ll need to create one. + - This namespace need not be hosted by a Fargate profile, but if it is desired, you’ll need to create one. - A Secret containing your Coralogix API Key, in the `cx-eks-fargate-otel` namespace. ## Creating Secret 1. Export your API key to a local variable: - 1. `export PRIVATE_KEY=` + 1. `export PRIVATE_KEY=` 2. Set your namespace variable: - 1. `export NAMESPACE=cx-eks-fargate-otel` + 1. `export NAMESPACE=cx-eks-fargate-otel` 3. Create the secret using kubectl: - 1. `kubectl create secret generic coralogix-keys -n $NAMESPACE --from-literal=PRIVATE_KEY=$PRIVATE_KEY` + 1. `kubectl create secret generic coralogix-keys -n $NAMESPACE --from-literal=PRIVATE_KEY=$PRIVATE_KEY` 4. Confirm it’s been set - 1. `kubectl get secret coralogix-keys -o yaml -n $NAMESPACE` + 1. `kubectl get secret coralogix-keys -o yaml -n $NAMESPACE` ## Create ServiceAccount @@ -52,7 +52,7 @@ eksctl create iamserviceaccount \ ## Configure and Deploy OTEL Collector Service: -The attached yaml manifest will deploy an OTEL collector, a clusterIP service for submission of application traces and metrics, and the cluster permissions required to query the Kubernetes API. +The attached yaml manifest will deploy an OTEL collector, a clusterIP service for submission of application traces and metrics, and the cluster permissions required to query the Kubernetes API. There are a few container environment variables that need to be set, detailed at the top of the yaml file. @@ -76,4 +76,4 @@ This manifest also has some required environmental variables that need to be set Again, after setting the environment variables, deploy using kubectl apply: -`kubectl apply -f cx-eks-fargate-otel-self-monitoring.yaml` \ No newline at end of file +`kubectl apply -f cx-eks-fargate-otel-self-monitoring.yaml`