Datadog extension

[tlfbrito]

Create a layer extension with Datadog APM for PHP.

[mnapoli]

Same comment as #34 (comment)

[fgilio]

I don't know about New Relic, but Datadog has first party support for Lambda and apparently has layers for other languages https://docs.datadoghq.com/integrations/amazon_lambda/?tab=awsconsole#datadog-lambda-layer

[bradleyess]

We are working on this at Linktree currently. The first party support for Lambda via Datadog does not show support for PHP. @fgilio - are you able to get this working with your PHP application?

[fgilio]

We are working on this at Linktree currently. The first party support for Lambda via Datadog does not show support for PHP. @fgilio - are you able to get this working with your PHP application?

No, we didn't. Tried a few things but had to abandon it

[bradleyess]

@fgilio - we have recently gotten this working with our Bref lambdas.

We are able to view logs and APM traces - but there is some work to be done to make it truly viable for production here.

1. We need to send REPORT logs from Lambda to enable Datadog Invocation metrics in the Serverless function view.
2. We need to enable Enhanced Lambda Metrics - we are unsure on how to do this right now as the recommendation is to use their Lambda layer, which supports Python and NodeJS. Worth a go as the metrics should have nothing to do with the runtime.

[reganjohnson]

Any updates here? Would love to have logging and tracing support for DataDog in our bref functions. @bradleyess Any resources available to view on how you accomplished this?

[danieleperot]

@ramsey would you like some help with the feature? I would like to contribute too if it helps 😊

[ramsey]

@danieleperot I believe Datadog themselves are working on some improvements to their Lambda / serverless support. Maybe @bwoebi can provide some insight? 🙂

[bwoebi]

So, the current state is that we have a big PR under review which will enable sending traces directly to Datadog, without an agent in between. You'll then provide the DD_API_KEY directly to php. We're looking towards releasing this first iteration in August.

However, the direct sending is still lacking the critical components of normalization/obfuscation and stats computation. It will work, but the experience may be lacking a bit then :-D We're looking at filling that in over the next few months.

Once we have that, we're looking at providing an integration for the Bref code itself, but that probably is still half a year away.

[tcarrio]

Hey @bwoebi, I'm curious why the architectural shift in this direction. It sounds like there are hurdles around re-implementing what the agent already provides and dd-trace-php already supports.

Are there designs around this change that could shed some light as to some of the advantages and disadvantages of the approach?

Is this something that's being applied across all languages that Datadog supports (thus a change in every dd-trace-$lang package per se).

I'm sure my mental map of these changes isn't 100% correct so any further information would be useful 🙂

[bwoebi]

The simple reason is: for very-short-running scenarios, like a function is invoked once, then thrown away, the overhead of the agent is not insignificant. And it will simplify the setup, obviously.

And yes, over time, we're looking at applying such a solution across all tracers.

The main disadvantage is that some of the code needs to be re-implemented. Apart from that, there are hopefully no significant disadvantages.

[danieleperot]

Thanks for the updates! 😊
Inspired by the work done by @ramsey I have been able to instrument datadog in the meantime by:

• moving the Datadog PHP INI file in php/conf.d/98-datadog.ini
• moving all Datadog extensions and content of /opt/datadog/ into a folder in my project called layers/datadog
• adding the following snippet to my serverless.yml:

  layers:
    datadog:
      path: layers/datadog
      name: ${sls:stage}-datadog
      description: Datadog PHP tracing extension layer
      retain: false

• adding the datadog custom layer to my functions in serverless.yml:

  # sample function
  layers:
    - ${bref:layer.php-82-fpm}
    - ${bref-extra:gd-php-82}
    - !Ref DatadogLambdaLayer

• adding serverless-plugin-datadog according to the instructions 😊

This pretty much took care of it for the time being 👍

[mnapoli]

@bwoebi would you be able to put me in touch with your contacts at Datadog? (or send me their email?)

I'd love to work on a native integration with Bref (proper support for Datadog). Here's my email: [email protected] Thanks!

[bwoebi]

@mnapoli You've been sent an email :-)

@danieleperot Nice that you got it to work - if you have any feedback about the current serverless experience, we're always interested in hearing it.

[ramsey]

Just to share the approach I took, here are my notes. I hope these (in addition to @danieleperot's notes) can help others.

I had created a Bref extension a while back, and it was working okay for us, except it wasn't properly sending metrics (statsd) back to Datadog, and since it was written for Bref v1, I decided to close it.

I note that someone else managed to get a Datadog extension merged for Bref, so maybe it's a good option for folks.

Nevertheless, here's the route I ended up going...

Here's a summary:

1. We build a custom layer that's kept in a zip file in our repo
2. We include this layer in our serverless.yaml file
3. We also include the serverless-plugin-datadog to get the Datadog and DogStatsD agents running on the Lambda

To build the custom layer, we have a directory in our repo at resources/layers/datadog (directory name doesn't matter, just including a note about it here because my examples use this path).

In this directory, we have:

1. a Dockerfile for building the layer
2. a script that we run to build the zip file
3. the zip file itself

Our Dockerfile looks very close to what was merged in #442:

Click to view the Dockerfile

ARG PHP_VERSION
FROM bref/build-php-$PHP_VERSION AS ext

ENV DDTRACE_BUILD_DIR=${BUILD_DIR}/ddtrace

ARG DATADOG_VERSION
RUN set -xe; \
    mkdir -p ${DDTRACE_BUILD_DIR}; \
    curl -Ls -o ${DDTRACE_BUILD_DIR}/datadog-setup.php \
        https://github.com/DataDog/dd-trace-php/releases/download/${DATADOG_VERSION}/datadog-setup.php

WORKDIR ${DDTRACE_BUILD_DIR}

RUN php datadog-setup.php --php-bin=all

RUN cp "$(php-config --extension-dir)/ddtrace.so" /tmp/ddtrace.so
RUN cp "$(php-config --extension-dir)/ddappsec.so" /tmp/ddappsec.so
RUN cp "$(php-config --extension-dir)/datadog-profiling.so" /tmp/datadog-profiling.so
RUN cp "$(php-config --ini-dir)/98-ddtrace.ini" /tmp/ext.ini

RUN sed -i 's/extension = ddtrace\.so/extension = \/opt\/bref-extra\/ddtrace.so/' /tmp/ext.ini
RUN sed -i 's/extension = ddappsec\.so/extension = \/opt\/bref-extra\/ddappsec.so/' /tmp/ext.ini
RUN sed -i 's/extension = datadog-profiling\.so/;extension = \/opt\/bref-extra\/datadog-profiling.so/' /tmp/ext.ini
RUN sed -i 's/datadog\.appsec\.enabled = On/datadog.appsec.enabled = Off/' /tmp/ext.ini

FROM scratch

COPY --from=ext /tmp/ddtrace.so /opt/bref-extra/ddtrace.so
COPY --from=ext /tmp/ddappsec.so /opt/bref-extra/ddappsec.so
COPY --from=ext /tmp/datadog-profiling.so /opt/bref-extra/datadog-profiling.so
COPY --from=ext /tmp/ext.ini /opt/bref/etc/php/conf.d/98-ddtrace.ini
COPY --from=ext /opt/datadog/ /opt/datadog

Then, our script to build the layer looks something like this (it's more robust, but I grabbed the essentials for these notes):

Click to view the script

#!/usr/bin/env bash

LAYER_BUILD_PATH="/path/to/resources/layers/datadog"
BREF_PHP_VERSION="82"
DATADOG_VERSION="0.90.0"
TAG="my/lambda-datadog-php-layer"
ZIP_FILE="$LAYER_BUILD_PATH/datadog.zip"

# Clean up any previous builds
rm "$ZIP_FILE"
rm -rf "$LAYER_BUILD_PATH/opt"

docker build \
    -t "$TAG" \
    --build-arg "PHP_VERSION=$BREF_PHP_VERSION" \
    --build-arg "DATADOG_VERSION=$DATADOG_VERSION" \
    --platform "linux/amd64" \
    "$LAYER_BUILD_PATH"

CONTAINER_ID=$(docker create --entrypoint=scratch "$TAG")

docker cp "$CONTAINER_ID:/opt" "$LAYER_BUILD_PATH"

zip -X --recurse-paths "$ZIP_FILE" "$LAYER_BUILD_PATH/opt"

# Clean up this build
rm -rf "$LAYER_BUILD_PATH/opt"
docker rm "$CONTAINER_ID"
docker rmi "$TAG"

echo "Layer zip file is available at $ZIP_FILE"

That builds the zip file, which we commit to our repo (mainly to save time during deployments).

We also have a custom INI file, using Bref's approach to customizing php.ini, at php/conf.d/datadog.ini. We turned off generation of the root span because it was showing weird results in our Datadog spans (more on this below).

Click to view datadog.ini

datadog.trace.enabled = On
datadog.trace.cli_enabled = On
datadog.trace.auto_flush_enabled = On
datadog.trace.generate_root_span = Off
datadog.trace.startup_logs = Off

In order to generate spans, we set up custom instrumentation. We created a script at src/Datadog/instrumentation.php, with the following contents. This installs a Datadog hook on Bref\Runtime\Invoker::invoke() and also on the handler it receives, and it creates spans.

Click to view instrumentation.php

<?php

/**
 * Creates APM spans for the Bref invoker and the handler that it invokes
 */

declare(strict_types=1);

namespace App\Datadog;

use Bref\Event\Handler;
use Closure;
use DDTrace\HookData;
use Psr\Http\Server\RequestHandlerInterface;

use function DDTrace\install_hook;
use function count;
use function extension_loaded;
use function is_array;
use function is_object;
use function is_string;
use function method_exists;

(static function (): void {
    if (!extension_loaded('ddtrace')) {
        return;
    }

    install_hook(
        'Bref\Runtime\Invoker::invoke',
        static function (HookData $hookData): void {
            $span = $hookData->span();

            /** @var RequestHandlerInterface | Handler | callable $handler */
            $handler = $hookData->args[0];

            $handlerHook = function (HookData $hookData) use ($span): void {
                $hookData->span($span);
            };

            if ($handler instanceof Closure) {
                install_hook($handler, $handlerHook);
            } elseif (is_object($handler) && method_exists($handler, 'handle')) {
                install_hook($handler::class . '::handle', $handlerHook);
            } elseif (is_array($handler) && count($handler) === 2) {
                $className = is_object($handler[0]) ? $handler[0]::class : $handler[0];
                install_hook($className . '::' . $handler[1], $handlerHook);
            } elseif (is_string($handler)) {
                install_hook($handler, $handlerHook);
            }
        },
    );
})();

To make this work, we updated composer.json to load this script as part of the autoloader. Like in this snippet:

Click to view snippet from composer.json

{
  "autoload": {
    "psr-4": {
      "App\\": "src/"
    },
    "files": [
      "src/Datadog/instrumentation.php"
    ]
  }
}

Lastly, we add the layer and Datadog plugin (and configuration) to our serverless.yml config:

Click to view snippets from serverless.yml

# ...

custom:

  # ...

  # For details on these configuration properties, see https://github.com/DataDog/serverless-plugin-datadog
  datadog:
    addExtension: true
    addLayers: false # Because we aren't using JavaScript/Python
    apiKey: ${env:DD_API_KEY}
    captureLambdaPayload: true
    enabled: ${strToBool(${env:DD_PLUGIN_ENABLED, false})}
    # We already send Cloudwatch logs to Datadog via the Datadog Forwarder, so
    # we set enableDDLogs to false so that we do not send the same logs twice.
    # See: https://docs.datadoghq.com/logs/guide/forwarder/
    enableDDLogs: false
    enableDDTracing: true
    enableSourceCodeIntegration: false # Because we aren't using JavaScript/Python
    env: ${self:custom.envStage}
    logLevel: ${env:DD_LOG_LEVEL, 'critical'}
    service: ${self:custom.serviceName}
    version: ${self:custom.version}

# ...

plugins:
  - serverless-plugin-datadog
  - ./vendor/bref/bref

layers:
  datadog:
    name: "our-datadog-lambda-layer"
    description: "Datadog layer"
    compatibleRuntimes:
      - provided.al2
    allowedAccounts:
      - ${aws:accountId}
    package:
      artifact: resources/layers/datadog/datadog.zip

provider:
  name: aws
  region: us-east-1
  runtime: php-82
  layers:
    - Ref: DatadogLambdaLayer # This name is magically created through having layers.datadog above.

# ...

That's a lot, but I hope it helps!

With this set up, we're able to capture logging, tracing, and metrics from our Bref Lambdas.