Skip to content

Commit

Permalink
fix: CDS-1810 Added Json support for Appname and Sub
Browse files Browse the repository at this point in the history
  • Loading branch information
@coralogix-concourse
coralogix-concourse authored Feb 6, 2025
1 parent 6e483da commit e5b4d74
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 34 deletions.
3 changes: 3 additions & 0 deletions aws-integrations/aws-shipper-lambda/CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
# Changelog
## v1.3.1 / 2025-02-04
### 🧰 Bug fixes 🧰
- Added support for dynamic allocation of Application and Subsystem names based on json key from log.

### v1.3.0 / 2025-01-20
### 💡 Enhancements 💡
Expand Down
44 changes: 10 additions & 34 deletions aws-integrations/aws-shipper-lambda/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,14 @@ To extract only a specific portion of the metadata value, you can utilize a regu
{{ s3.object.key | r'AWSLogs\/.+\/(.*)$' }}
```

This results in a `SubsystemName` value of `elb.log` as this is the part of the regex that is captured by the group `(.*)`.
This would result in a SubsystemName value of `elb.log` as this is the part of the regex that is captured by the group `(.*)`.

If you want to use a json key value as the application name, you would set the `ApplicationName` parameter to:

```
{{ $.eventSource }}
```
Assume the log is a CloudTrail log and the eventSource is `s3.amazonaws.com` then the application name will be `s3.amazonaws.com`.

!!! Important

Expand All @@ -344,7 +351,7 @@ If you want to bypass using the public internet, you can use AWS PrivateLink to

!!! note

The following method for using dynamic values will change to the method defined above in `coralogix-aws-shipper v1.1.0` and later. This approach will no longer be supported.
> Note the following method for using dynamic values will change to the method defined above in `coralogix-aws-shipper v1.1.0` and later. This approach will no longer be supported. Please check the new method in the [Metadata](#metadata) section.

If you wish to use dynamic values for the Application and Subsystem Name parameters, consider the following:

Expand Down Expand Up @@ -406,7 +413,7 @@ If the deployment fails while assigning the trigger, ensure that no notification

Set the `MAX_ELAPSED_TIME` variable for the default change (default = 250). The `BATCHES_MAX_SIZE` (in MB) defines the maximum batch size before sending data to Coralogix. This value is limited by the maximum payload accepted by the Coralogix endpoint (default = 4). The `BATCHES_MAX_CONCURRENCY` sets the maximum number of concurrent batches that can be sent.

# Cloudwatch Metrics Stream via Firehose PrivateLink (beta)
# Cloudwatch Metrics Stream via PrivateLink (beta)

As of version `v1.3.0`, the Coralogix AWS Shipper supports streaming **Cloudwatch Metrics to Coralogix via Firehose over a PrivateLink**.

Expand Down Expand Up @@ -438,37 +445,6 @@ To enable the Cloudwatch Metrics Stream via Firehose (PrivateLink) you must prov
| LambdaSecurityGroupID | Specify the ID of the Security Group where the integration should be deployed. | | :heavy_check_mark: |
| StoreAPIKeyInSecretsManager | Enable this to store your API Key securely. Otherwise, it will remain exposed in plain text as an environment variable in the Lambda function console. | True | |

# Cloudwatch Metrics Stream via Firehose PrivateLink (beta)

As of version `v1.3.0`, the Coralogix AWS Shipper supports streaming **Cloudwatch Metrics to Coralogix via Firehose over a PrivateLink**.

This workflow is designed for scenarios where you need to stream metrics from a CloudWatch Metrics stream to Coralogix via a PrivateLink endpoint.

#### Why Use This Workflow?

AWS Firehose does not support PrivateLink endpoints as a destination because Firehose cannot be connected to a VPC, which is required to reach a PrivateLink endpoint. To overcome this limitation, the Coralogix AWS Shipper acts as a transform function. It is attached to a Firehose instance that receives metrics from the CloudWatch Metrics stream and forwards them to Coralogix over a PrivateLink.

#### When to Use This Workflow

This workflow is specifically for bypassing the limitation of using Firehose with the Coralogix PrivateLink endpoint. If there is no requirement for PrivateLink, we recommend using the default Firehose Integration for CloudWatch Stream Metrics found [here](https://coralogix.com/docs/integrations/aws/amazon-data-firehose/aws-cloudwatch-metric-streams-with-amazon-data-firehose/).

#### How does it work?

![Cloudwatch stream via PrivateLink Workflow](./static/cloudwatch-metrics-pl-workflow.png)

To enable the Cloudwatch Metrics Stream via Firehose (PrivateLink) you must provide the required parameters outlined below.

| Parameter | Description | Default Value | Required |
|-----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|--------------------|
| TelemetryMode | Specify the telemetry collection modes, supported values (`metrics`, `logs`). Note that this value must be set to `metrics` for the Cloudwatch metric stream workflow | logs | :heavy_check_mark: |
| ApiKey | The Send-Your-Data [API Key](https://coralogix.com/docs/send-your-data-api-key/) validates your authenticity. This value can be a direct Coralogix API Key or an AWS Secret Manager ARN containing the API Key.<br>*Note the parameter expects the API Key in plain text or stored in secret manager.* | | :heavy_check_mark: |
| ApplicationName | The name of the application for which the integration is configured. [Advanced Configuration](#advanced-configuration) specifies dynamic value retrieval options. | | :heavy_check_mark: |
| SubsystemName | Specify the [name of your subsystem](https://coralogix.com/docs/application-and-subsystem-names/). For a dynamic value, refer to the Advanced Configuration section. For CloudWatch, leave this field empty to use the log group name. | | :heavy_check_mark: |
| CoralogixRegion | Your data source should be in the same region as the integration stack. You may choose from one of [the default Coralogix regions](https://coralogix.com/docs/coralogix-domain/): [Custom, EU1, EU2, AP1, AP2, US1, US2]. If this value is set to Custom you must specify the Custom Domain to use via the CustomDomain parameter. | Custom | :heavy_check_mark: |
| S3BucketName | The S3Bucket that will be used to store records that have failed processing | | :heavy_check_mark: |
| LambdaSubnetID | Specify the ID of the subnet where the integration should be deployed. | | :heavy_check_mark: |
| LambdaSecurityGroupID | Specify the ID of the Security Group where the integration should be deployed. | | :heavy_check_mark: |
| StoreAPIKeyInSecretsManager | Enable this to store your API Key securely. Otherwise, it will remain exposed in plain text as an environment variable in the Lambda function console. | True | |

## Support

Expand Down

0 comments on commit e5b4d74

Please sign in to comment.