error=service error with CloudWatch Metrics sink

[PeteLeeds]

Hi, I have a Vector Aggregator deployment in ECS/Fargate, which receives logs and metrics from other Vector deployments and sends them to CloudWatch. This works well for logs, however when deploying using metrics I get the following error:

2024-12-05T17:04:12.628092Z ERROR sink{component_kind="sink" component_id=cloudwatch_metrics component_type=aws_cloudwatch_metrics}:request{request_id=1551}: vector::sinks::util::sink: Request failed. error=service error

This error starts pretty much as soon as the cluster starts up. Sometimes some metrics make it into Cloudwatch and sometimes they don't (one hypothesis is that they may have started reaching cloudwatch after I reduced the rate limit, but that hasn't stopped the error).

My config is as below:

[sources.vector]
address = "0.0.0.0:6000" 
type = "vector"

[transforms.vector_logs_metrics]    
inputs = [ "vector" ]
type = "route"
route.logs = { type = "is_log" }
route.metrics = { type = "is_metric" }

[sinks.cloudwatch_metrics]
type = "aws_cloudwatch_metrics"
default_namespace = "default-namespace"
request.rate_limit_num = 50
inputs = ["vector_logs_metrics.metrics"]

There isn't much to go on on this error, so I'm wondering if anyone has seen anything similar or has some steps to help debug?

Vector version: 0.42.0

[PeteLeeds]

I've been debugging this by taking metrics out and seeing what happens, and it seems that the error only occurs when trying to send the filesystem_used_ratio host_metric to Cloudwatch. Are there any known quirks with this metric?

[jszwedko]

I've been debugging this by taking metrics out and seeing what happens, and it seems that the error only occurs when trying to send the filesystem_used_ratio host_metric to Cloudwatch.

That's odd. That is a just a simple gauge metric so I can't imagine why it would result in different behavior than other gauge metrics. Maybe taking it out brought it below some limit of number of metrics? Are there any other ERROR logs before this one? You could try setting VECTOR_LOG= aws_smithy_runtime=debug to get debug AWS logs that may have more info.

[PeteLeeds]

Thanks for the response - just filtering down to that metric and no others produces the error, so unless that metric by itself is reaching some limit it seems to be something to do with the metric itself. By using vector tap I can see that the value is sometimes 'null' for this gauge (I can't see this being the case for others), I wonder if that might be causing the issue? I can't seem to be able to test as it doesn't look like I can access .gauge.value in VRL (I can see this ticket if this is still the case? #5521)

[jszwedko]

By using vector tap I can see that the value is sometimes 'null' for this gauge (I can't see this being the case for others)

Ah, that could be it. That is a bit strange though 🤔 Here is where it is emitted:

https://github.com/vectordotdev/vector/blob/master/src/sources/host_metrics/filesystem.rs#L129-L134

I can't see how the value could end up being null.

I can't seem to be able to test as it doesn't look like I can access .gauge.value in VRL (I can see this ticket if this is still the case? #5521)

Yeah, unfortunately. You could use a lua transform though.