Splunk HEC Metrics AggregatedSummary Support

[clmaestas-work]

Good afternoon,

I am testing out vector to see if it fulfill some of our needs. In our tests so far we are going prometheus -> vector -> splunk enterprise. So far it looks like it does what we need it to do, but I've hit one roadblock. It appears the AggregatedSummary metric type is not supported by the splunk_hec_metrics sink. Is this the case, have I not configured it correctly, or am I misreading the error message?

For example here is the prometheus config:

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'prometheus_master'
    static_configs:
      - targets: ['localhost:9090']

Here is the vector config:

api:
  enabled: true

sources:
  promScrape:
    type: prometheus_scrape
    endpoints:
      - http://localhost:9090/metrics

sinks:
  promToSplunk:
    type: splunk_hec_metrics
    inputs:
      - promScrape
    default_token: [my token]
    endpoint: [my endpoint]
    index: "promvecmetrics"
    tls:
      verify_certificate: false

With the above I get the following error:

ERROR sink{component_kind="sink" component_id=promToSplunk component_type=splunk_hec_metrics}: vector::internal_events::splunk_hec::sink: Invalid metric received. error="Metric kind not supported." error_type="invalid_metric" stage="processing" value=AggregatedSummary { quantiles: [Quantile { quantile: 0.0, value: 5.0525e-5 }, Quantile { quantile: 0.25, value: 8.2525e-5 }, Quantile { quantile: 0.5, value: 0.000108933 }, Quantile { quantile: 0.75, value: 0.000187652 }, Quantile { quantile: 1.0, value: 0.001971355 }], count: 2869, sum: 0.454619628 } kind=Absolute internal_log_rate_limit=true

Because of the vector::internal_events::splunk_hec::sink: Invalid metric received. I assume this means that the splunk_hec_metrics sink does not support AggregatedSummary metrics. Is that interpretation correct. I ask because I see the open issue Support AggregatedSummary for gcp_stackdriver sink #9530 So I am thinking this may be another case where one of the sinks does not support AggregatedSummary.

If it is a supported metric type could you point me in the direction of a working example?

If it is not supported, is there a workaround to get this metric type through to splunk? I am guessing I would have to transform it, remap it, then send it too splunk_hec_logs where I could use the built in log2metric sourcetypes.

Finally is there a page I can go to too see all of the supported metric types for each sync for future reference.

[jszwedko]

Hey! Yes, unfortunately it looks like that metric kind isn't supported yet. Looking at

vector/src/sinks/splunk_hec/metrics/sink.rs

Lines 124 to 145 in d90a95c

	impl HecMetricsProcessedEventMetadata {
	fn extract_metric_name(metric: &Metric, default_namespace: Option<&str>) -> String {
	encode_namespace(metric.namespace().or(default_namespace), '.', metric.name())
	}
	fn extract_metric_value(metric: &Metric) -> Option<f64> {
	match *metric.value() {
	MetricValue::Counter { value } => Some(value),
	MetricValue::Gauge { value } => Some(value),
	_ => {
	emit!(SplunkInvalidMetricReceivedError {
	value: metric.value(),
	kind: &metric.kind(),
	error: "Metric kind not supported.".into(),
	});
	None
	}
	}
	}
	}
	pub type HecProcessedEvent = ProcessedEvent<Metric, HecMetricsProcessedEventMetadata>;

it seems like only counters and gauge are supported at the moment.

As a workaround you could probably use metric_to_log and then transform the output. Looking at https://docs.splunk.com/Documentation/Splunk/9.3.1/Metrics/GetMetricsInOther it seems like only raw points are accepted so I think you'd need to send metrics like <some metric>_max, etc.

[clmaestas-work]

Understood, thank you for the prompt response and suggestions. I'll be getting out of your hair now. Thank you again.