Configure batch for EC2 (#379)

Signed-off-by: Matej Gera <[email protected]>

otel-ecs-ec2/CHANGELOG.md

@@ -2,6 +2,9 @@
 
 ## OpenTelemtry-Agent
 
+### v0.0.32 / 2024-03-05
+* [FEAT] Add batch size limits configuration
+
 ### v0.0.31 / 2023-12-13
 * [FIX] fix config example files
 

otel-ecs-ec2/README.md

@@ -62,6 +62,23 @@ This repo provides example of the following configuration files (you can create
 - [logging](logging.yaml)
 - [traces & metrics](config.yaml)
 
+#### About batch sizing
+
+[Batch processor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor) ensures that the telemetry being sent to Coralogix backend is batched into bigger requests, ensuring lower networking overhead and better performance. The batching processor is enabled by default and we strongly recommend to use it. By default, the `otel-integration` chart uses the following recommended settings for batch processors in all collectors:
+
+  ```yaml
+      batch:
+        send_batch_size: 1024
+        send_batch_max_size: 2048
+        timeout: "1s"
+  ```
+
+These settings imposes a hard limit of 2048 units (spans, metrics, logs) on the batch size, ensuring a balance between the recommended size of the batches and networking overhead. 
+
+You may adjust these settings according to your needs, but when configuring the batch processor by yourself, it is important to be mindful of the size limites imposed by the Coraloigx endpoints (currently **max. 10 MB** after decompression - see [documentation](https://coralogix.com/docs/opentelemetry/#limits--quotas)). 
+
+More information on how to configure the batch processor can be found [here](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor#batch-processor).
+
 ### Deploying Open Telemetry
 
 Once you have your configuration files and ECS Cluster ready, you can deploy Open Telemetry to ECS using the cloudformation templates available [here](https://github.com/coralogix/cloudformation-coralogix-aws/blob/master/opentelemetry/ecs-ec2/README.md).

otel-ecs-ec2/config.yaml

@@ -52,7 +52,10 @@ processors:
           - set(attributes["image_id"], attributes["image.id"])
           - delete_key(attributes, "image.id")
 
-  batch:
+    batch:
+      send_batch_size: 1024
+      send_batch_max_size: 2048
+      timeout: "1s"
 
   # otel-collector resource detection for collector
   resourcedetection/otel-collector: