Using sendDistributionMetricWithDate gives runtime error on flush

[timotheeg]

Expected Behavior

Lambda using sendDistributionMetricWithDate() should work and metrics should be sent

Actual Behavior

In my Lambda, I have 2 calls to send metrics. I initially used sendDistributionMetric(), but decided to provide both the same date with sendDistributionMetricWithDate() instead. The metrics are accepted, I can see the log entries:

{
    "e": 1728552052.824,
    "m": "cx.helpdesk.call.rating",
    "t": [
        "dd_lambda_layer:datadog-nodev20.17.0",
        "region:ap-southeast-1",
        "memorysize:128",
        "cold_start:true",
        "datadog_lambda:v9.115.0",
        "runtime:nodejs20.x",
        // ... plus private tags
    ],
    "v": 6
}

(why is this even printed in the log btw? did someone forgot to remove a console log?)

But at the end of processing, this error is logged:

2024-10-10T09:20:53.845Z	76aa8b68-d34d-4561-8181-faa7b0cd8cf9	ERROR	[dd.trace_id=3371094672245296109 dd.span_id=7681230763497218352] 
{
    "status": "error",
    "message": "Cannot read properties of undefined (reading 'flush')",
    "name": "TypeError",
    "stack": "TypeError: Cannot read properties of undefined (reading 'flush')
    at MetricsListener.<anonymous> (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:134:56)
    at step (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:33:23)
    at Object.next (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:14:53)
    at fulfilled (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:5:58)"
}

Steps to Reproduce the Problem

1. Make a lambda in ap-southeast-1
2. Use the 2 DD layers:
   • arn:aws:lambda:ap-southeast-1:464622532012:layer:Datadog-Node20-x:115
   • arn:aws:lambda:ap-southeast-1:464622532012:layer:Datadog-Extension:65
3. set up the usual DD env vars to retrieve API key and everything
4. use sendDistributionMetric() for 2 metrics, and verify data appears in DD
5. change to use sendDistributionMetricWithDate() for the 2 metrics, and see the runtime errors in cloudwatch logs

Specifications

* `arn:aws:lambda:ap-southeast-1:464622532012:layer:Datadog-Node20-x:115`
* `arn:aws:lambda:ap-southeast-1:464622532012:layer:Datadog-Extension:65`

Stacktrace

2024-10-10T09:20:53.845Z	76aa8b68-d34d-4561-8181-faa7b0cd8cf9	ERROR	[dd.trace_id=3371094672245296109 dd.span_id=7681230763497218352] 
{
    "status": "error",
    "message": "Cannot read properties of undefined (reading 'flush')",
    "name": "TypeError",
    "stack": "TypeError: Cannot read properties of undefined (reading 'flush')
    at MetricsListener.<anonymous> (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:134:56)
    at step (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:33:23)
    at Object.next (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:14:53)
    at fulfilled (/opt/nodejs/node_modules/datadog-lambda-js/metrics/listener.js:5:58)"
}

[astuyve]

Hi @timotheeg - thanks for reaching out!

We print metrics to stdout for customers using the Datadog Lambda Log Forwarder as a data egress mechanism. You can see this logic here. Given that you're using the Datadog Lambda Extension, that shouldn't be enabled unless log forwarding was inadvertently enabled as well with DD_FLUSH_TO_LOG: true.

Can you share the configuration and code for your Lambda function which causes this issue to appear?

Thanks!

[timotheeg]

Thanks @astuyve

We print metrics to stdout for customers using the Datadog Lambda Log Forwarder as a data egress mechanism.

Noted, but so why is there a difference between sendDistributionMetricWithDate() and sendDistributionMetric() on the logging behaviour? When I use sendDistributionMetric(), it does not log, and the data gets sent to DD (no run time error) 🤔 .

that shouldn't be enabled unless log forwarding was inadvertently enabled as well with DD_FLUSH_TO_LOG: true

I can see the lambda DOES have DD_FLUSH_TO_LOG: true in the env vars, I could turn that off and try again 🤔

But also I'll try making a minimal repro case to send you (so there's nothing related to my company in there).

[cwill-smg]

@timotheeg Any updated on this? I am facing the same issue, even though DD_FLUSH_TO_LOG is set to false.

[cwill-smg]

@astuyve As posted above, I am facing the same issue. This is my configuration:

package.json:

"datadog-lambda-js": "9.118.0"

CDK:

let ddProps: DatadogProps = {
    addLayers: true,
    env: envUtils.getEnvStage(),
  	nodeLayerVersion: 118,
    extensionLayerVersion: 68,
    apiKeySecretArn: ddSecretArn,
    flushMetricsToLogs: false, // rely on dd-forwarder only
    injectLogContext: false, // rely on dd-forwarder only
    enableDatadogLogs: true,
    enableDatadogTracing: true,
    service: 'unified-importer',
    site: 'datadoghq.eu',
    version: process.env.GITHUB_SHA || '1'
  };

Example lambda envs:

Error

[cwill-smg]

I debugged a bit further: I downloaded the Datadog-Node20-x-layer and added some debug-logs to it. With that changes I created a new version of the layer. With that debug-logs I can see, that this code-path is never reached

 private async createProcessor(config: MetricsConfig, apiKey: Promise<string>) {
    if (!this.config.logForwarding) {
     // nope :-(
    }
  }  

Even though

• the lambda has DD_FLUSH_TO_LOG set to false (as the attached screenshot above shows)
• and both flushMetricsToLogs and injectLogContext are set to false during creation of the layer in CDK

[cwill-smg]

Debug log of this.config at this position reveals that log_forwarding is indeed true 🤔

And logs for process.env (relevant DD-properties):

I post the full configuration to check if maybe any other setting leads to log_forwarding being true:

let ddProps: DatadogProps = {
    addLayers: true,
    enableMergeXrayTraces: false,
    env: envUtils.getEnvStage(),
    nodeLayerVersion: 118, // https://github.com/DataDog/datadog-lambda-js/releases
    service: 'my-service',
    site: 'datadoghq.eu',
    version: process.env.GITHUB_SHA || '1',
    apiKeySecretArn: ddSecretArn,
    flushMetricsToLogs: false, // rely on dd-forwarder only
    injectLogContext: false, // rely on dd-forwarder only
    enableDatadogLogs: true,
    enableDatadogTracing: true,
    extensionLayerVersion: 68
  };

[cwill-smg]

Further, this looks suspicious to me:

datadog:Using StatsD client

which is very likely not to be expected when using nodeLayerVersion in conjunction with extensionLayerVersion no?

[astuyve]

Hey, the nodeLayerVersion is our tracing layer an is meant to be used alongside the extensionLayerVersion. In that case I would expect the statsd client to be enabled to send distribution metrics to the datadog extension.

That config appears to only read from the environment variable or a datadog.yaml file, and defaults to false. Since the environment variable is ruled out, maybe there's a config file?

Can you confirm if a datadog.yaml file is in the function zip payload?

Thanks!

[cwill-smg]

Thanks @astuyve for your reply 🙏

the nodeLayerVersion is our tracing layer an is meant to be used alongside the extensionLayerVersion. In that case I would > expect the statsd client to be enabled to send distribution metrics to the datadog extension.

Ah, Ok.

Can you confirm if a datadog.yaml file is in the function zip payload?

No, there is no datadog.yml neither within the Lambda nor as additional layer.

[cwill-smg]

When I add debug-logs to getConfig then I receive these settings:

So it actually never calls the code in the if-block:

if (userConfig === undefined || userConfig.logForwarding === undefined) {
        var result = getEnvValue(exports.logForwardingEnvVar, "false").toLowerCase();
        config.logForwarding = result === "true";
}

[cwill-smg]

@astuyve I finally found the issue: The project I'm currently working on is using an internal library to instantiate its lambdas. And this library was setting logForwarding: true internally 🤦
Sorry for the trouble and your time.

[astuyve]

No problem, glad you got it tracked down!

I'm going to close this issue for now as it does seem to be related to unintentional configurations, although I'll ask someone to investigate if we can catch this error a little more gracefully.

Thanks again!