Skip to content

Commit

Permalink
cds-1565 Fix Publish Error with v0.3.0 release
Browse files Browse the repository at this point in the history
  • Loading branch information
@coralogix-concourse
coralogix-concourse authored Feb 1, 2025
1 parent 422911d commit 082395f
Show file tree
Hide file tree
Showing 3 changed files with 73 additions and 10 deletions.
1 change: 1 addition & 0 deletions aws-integrations/aws-shipper-lambda/CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
### v1.3.0 / 2025-01-20
### 💡 Enhancements 💡
- New intergration workflow added for ingesting Cloudwatch Stream Metrics via Firehose over PrivateLink
- Add Cloudwatch Metrics Stream creation to custom resource function

### v1.2.0 / 2025-01-7
### 🧰 Bug fixes 🧰
Expand Down
32 changes: 32 additions & 0 deletions aws-integrations/aws-shipper-lambda/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,38 @@ To enable the Cloudwatch Metrics Stream via Firehose (PrivateLink) you must prov
| LambdaSecurityGroupID | Specify the ID of the Security Group where the integration should be deployed. | | :heavy_check_mark: |
| StoreAPIKeyInSecretsManager | Enable this to store your API Key securely. Otherwise, it will remain exposed in plain text as an environment variable in the Lambda function console. | True | |

# Cloudwatch Metrics Stream via Firehose PrivateLink (beta)

As of version `v1.3.0`, the Coralogix AWS Shipper supports streaming **Cloudwatch Metrics to Coralogix via Firehose over a PrivateLink**.

This workflow is designed for scenarios where you need to stream metrics from a CloudWatch Metrics stream to Coralogix via a PrivateLink endpoint.

#### Why Use This Workflow?

AWS Firehose does not support PrivateLink endpoints as a destination because Firehose cannot be connected to a VPC, which is required to reach a PrivateLink endpoint. To overcome this limitation, the Coralogix AWS Shipper acts as a transform function. It is attached to a Firehose instance that receives metrics from the CloudWatch Metrics stream and forwards them to Coralogix over a PrivateLink.

#### When to Use This Workflow

This workflow is specifically for bypassing the limitation of using Firehose with the Coralogix PrivateLink endpoint. If there is no requirement for PrivateLink, we recommend using the default Firehose Integration for CloudWatch Stream Metrics found [here](https://coralogix.com/docs/integrations/aws/amazon-data-firehose/aws-cloudwatch-metric-streams-with-amazon-data-firehose/).

#### How does it work?

![Cloudwatch stream via PrivateLink Workflow](./static/cloudwatch-metrics-pl-workflow.png)

To enable the Cloudwatch Metrics Stream via Firehose (PrivateLink) you must provide the required parameters outlined below.

| Parameter | Description | Default Value | Required |
|-----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|--------------------|
| TelemetryMode | Specify the telemetry collection modes, supported values (`metrics`, `logs`). Note that this value must be set to `metrics` for the Cloudwatch metric stream workflow | logs | :heavy_check_mark: |
| ApiKey | The Send-Your-Data [API Key](https://coralogix.com/docs/send-your-data-api-key/) validates your authenticity. This value can be a direct Coralogix API Key or an AWS Secret Manager ARN containing the API Key.<br>*Note the parameter expects the API Key in plain text or stored in secret manager.* | | :heavy_check_mark: |
| ApplicationName | The name of the application for which the integration is configured. [Advanced Configuration](#advanced-configuration) specifies dynamic value retrieval options. | | :heavy_check_mark: |
| SubsystemName | Specify the [name of your subsystem](https://coralogix.com/docs/application-and-subsystem-names/). For a dynamic value, refer to the Advanced Configuration section. For CloudWatch, leave this field empty to use the log group name. | | :heavy_check_mark: |
| CoralogixRegion | Your data source should be in the same region as the integration stack. You may choose from one of [the default Coralogix regions](https://coralogix.com/docs/coralogix-domain/): [Custom, EU1, EU2, AP1, AP2, US1, US2]. If this value is set to Custom you must specify the Custom Domain to use via the CustomDomain parameter. | Custom | :heavy_check_mark: |
| S3BucketName | The S3Bucket that will be used to store records that have failed processing | | :heavy_check_mark: |
| LambdaSubnetID | Specify the ID of the subnet where the integration should be deployed. | | :heavy_check_mark: |
| LambdaSecurityGroupID | Specify the ID of the Security Group where the integration should be deployed. | | :heavy_check_mark: |
| StoreAPIKeyInSecretsManager | Enable this to store your API Key securely. Otherwise, it will remain exposed in plain text as an environment variable in the Lambda function console. | True | |

## Support

**Need help?**
Expand Down
50 changes: 40 additions & 10 deletions aws-integrations/aws-shipper-lambda/template.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -946,6 +946,18 @@ Resources:
- !Ref Secret
- !Ref ApiKey
TelemetryMode: !Ref TelemetryMode
CWStreamFirehoseDestinationARN: !If
- TelemetryModeIsMetrics
- !GetAtt FireHoseDestination.Arn
- !Ref AWS::NoValue
CWStreamFirehoseAccessRoleARN: !If
- TelemetryModeIsMetrics
- !GetAtt FirehoseAccessRole.Arn
- !Ref AWS::NoValue
CWMetricStreamName: !If
- TelemetryModeIsMetrics
- !Sub '${AWS::StackName}-coralogix-metric-stream'
- !Ref AWS::NoValue
Metadata:
SamResourceId: ConfigureLambda
LambdaLogGroup:
Expand Down Expand Up @@ -1186,16 +1198,9 @@ Resources:
- "firehose:PutRecord"
- "firehose:PutRecordBatch"
Resource: !GetAtt FireHoseDestination.Arn
CloudWatchMetricStream:
Type: AWS::CloudWatch::MetricStream
Condition: TelemetryModeIsMetrics
Properties:
FirehoseArn: !GetAtt FireHoseDestination.Arn
RoleArn: !GetAtt FirehoseAccessRole.Arn
OutputFormat: opentelemetry1.0
##################################
# -- Custom Resource Definition -#
##################################
##################################
# -- Custom Resource Definition -#
##################################
CustomResourceFunction:
Type: AWS::Serverless::Function
Metadata:
Expand All @@ -1206,6 +1211,14 @@ Resources:
Timeout: 900
Policies:
- Statement:
- !If
- TelemetryModeIsMetrics
- Sid: IamPassRole
Effect: Allow
Action:
- iam:PassRole
Resource: !GetAtt FirehoseAccessRole.Arn
- !Ref AWS::NoValue
- Sid: EC2Access
Effect: Allow
Action:
Expand Down Expand Up @@ -1235,6 +1248,23 @@ Resources:
Resource:
- !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:*
- !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:event-source-mapping:*
- Sid: CloudWatchMetrics
Effect: Allow
Action:
- cloudwatch:PutMetricStream
- cloudwatch:DeleteMetricStream
- cloudwatch:GetMetricStream
- cloudwatch:ListMetricStreams
- cloudwatch:StartMetricStreams
- cloudwatch:StopMetricStreams
- cloudwatch:DescribeAlarmsForMetric
- cloudwatch:PutMetricData
- cloudwatch:GetMetricData
- cloudwatch:GetMetricStatistics
- cloudwatch:ListMetrics
- cloudwatch:GetMetricWidgetImage
Resource:
- !Sub arn:aws:cloudwatch:${AWS::Region}:${AWS::AccountId}:*
- Statement:
- Sid: S3NotificationPolicy
Effect: Allow
Expand Down

0 comments on commit 082395f

Please sign in to comment.