diff --git a/otel-ecs-ec2/CHANGELOG.md b/otel-ecs-ec2/CHANGELOG.md index c6de5657..3c426372 100644 --- a/otel-ecs-ec2/CHANGELOG.md +++ b/otel-ecs-ec2/CHANGELOG.md @@ -2,6 +2,9 @@ ## OpenTelemtry-Agent +### v0.0.32 / 2024-03-05 +* [FEAT] Add batch size limits configuration + ### v0.0.31 / 2023-12-13 * [FIX] fix config example files diff --git a/otel-ecs-ec2/README.md b/otel-ecs-ec2/README.md index 2e2a1211..0ef14729 100644 --- a/otel-ecs-ec2/README.md +++ b/otel-ecs-ec2/README.md @@ -62,6 +62,23 @@ This repo provides example of the following configuration files (you can create - [logging](logging.yaml) - [traces & metrics](config.yaml) +#### About batch sizing + +[Batch processor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor) ensures that the telemetry being sent to Coralogix backend is batched into bigger requests, ensuring lower networking overhead and better performance. The batching processor is enabled by default and we strongly recommend to use it. By default, the `otel-integration` chart uses the following recommended settings for batch processors in all collectors: + + ```yaml + batch: + send_batch_size: 1024 + send_batch_max_size: 2048 + timeout: "1s" + ``` + +These settings imposes a hard limit of 2048 units (spans, metrics, logs) on the batch size, ensuring a balance between the recommended size of the batches and networking overhead. + +You may adjust these settings according to your needs, but when configuring the batch processor by yourself, it is important to be mindful of the size limites imposed by the Coraloigx endpoints (currently **max. 10 MB** after decompression - see [documentation](https://coralogix.com/docs/opentelemetry/#limits--quotas)). + +More information on how to configure the batch processor can be found [here](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor#batch-processor). + ### Deploying Open Telemetry Once you have your configuration files and ECS Cluster ready, you can deploy Open Telemetry to ECS using the cloudformation templates available [here](https://github.com/coralogix/cloudformation-coralogix-aws/blob/master/opentelemetry/ecs-ec2/README.md). diff --git a/otel-ecs-ec2/config.yaml b/otel-ecs-ec2/config.yaml index edf4c754..9e819309 100644 --- a/otel-ecs-ec2/config.yaml +++ b/otel-ecs-ec2/config.yaml @@ -52,7 +52,10 @@ processors: - set(attributes["image_id"], attributes["image.id"]) - delete_key(attributes, "image.id") - batch: + batch: + send_batch_size: 1024 + send_batch_max_size: 2048 + timeout: "1s" # otel-collector resource detection for collector resourcedetection/otel-collector: