OpenTelemetry support

[RaviVaranasi]

New Feature Summary

Can I ship logs to DataDog instead of logging inside Salesforce? The advantage is to have one logging system across multiple systems

[jamessimone]

@RaviVaranasi I would look at the Slack plugin for the canonical way to publish log entries off-platform (as far as architecture is concerned; this isn't to say that you must send log entries to Slack if they're going off-platform, just that the way that this plugin is built should answer the question of "how to do this.")

I'm sure @jongpie can add additional info - and perhaps an official DataDog plugin can also be supported - but if you're looking to build out this functionality and the question is just "is it possible," the architecture in the Slack plugin should be enough to get going!

[RaviVaranasi]

Thanks for the feedback. If we ship the logs to DataDog, can we not create Log objects in SF?

[jamessimone]

You still can!

[jongpie]

@RaviVaranasi I've previously done some integrations other tools like Loggly, but I haven't used Datadog myself before, so I'm not familiar with their API or how to best integrate with them. But to expand on what @jamessimone has said, I think there are 2-3 ways you could approach this type of integration:

Option 1: Create a Nebula Logger plugin that sends Log__c and LogEntry__c data to Datadog

As @jamessimone pointed out, the Slack plugin is a great example of how plugins for Nebula Logger are structured. This plugin in particular also leverages Log__c and LogEntry__c data, so if you want to still store data in Salesforce (in addition to storing it in Datadog), then I would mimic the Slack plugin's approach

• Pro: this approach would keep your existing usage of Nebula Logger the same, the plugin would just be an additional async job that sends data from the custom objects to Datadog
• Con: this will continue to use your org's data storage in the custom objects Log__c and LogEntry__c

Option 2: Create a Nebula Logger plugin that can be configured to send either LogEntryEvent__e or Log__c/LogEntry__c data to Datadog

If you're worried about the storage usage of the custom objects in your org, and you want to control if some or all Nebula Logger instead goes just to Datadog, then you can take a look at the Big Object Archiving plugin. The Big Object Archiving plugin lets you choose per user if Nebula Logger's LogEntryEvent__e data is stored in Log__c/LogEntry__c, or only stored in a custom big object (LogEntryArchive__b). You could have a similar plugin for Datadog that lets you control per user if logging data is stored in Salesforce, Datadog, or both

• Pro: this approach let you decide per user if their logging data lives in Salesforce and/or Datadog
• Con: this would be the most complex option to implement

Option 3: Don't Create a Nebula Logger plugin at all - instead, integrate using pub/sub API

I'm not sure what's involved with setting up this option (because I haven't used Datadog before), but since Nebula Logger uses platform events to create LogEntryEvent__e data, you can probably setup Datadog to subscribe directly to LogEntryEvent__e object using the pub/sub API.

• Pro: this approach would not require creating a Nebula Logger plugin (as far as I'm aware) - this would be configured directly in Datadog (assuming they support it)
• Con: external pub/sub subscribers count towards your org's daily limits for platform event allocations - so if you have a lot of logging occurring in your org, this could cause you to exceed your limits

Hope this helps, but let us know if you have questions!

[sjurgis]

@jongpie any chance you know any resources / examples how to log apex metrics (mostly around performance)?

[sjurgis]

FWIW Datadog does have a connector for Event Monitoring https://docs.datadoghq.com/integrations/salesforce/

[RaviVaranasi]

FWIW Datadog does have a connector for Event Monitoring https://docs.datadoghq.com/integrations/salesforce/

Does this do logging?

[jonathankretzmer]

Instead of actively targeting a single provider like DataDog, it would possibly be more beneficial to target an open standard, e.g. OpenTelemetry (which DataDog supports).

[jongpie]

@jonathankretzmer I agree, I think it makes more sense to target the OpenTelemetry standard, instead of vendor-specific integrations. There was recently a discussion about this too, so I think a lot of people would benefit from using OpenTelemetry. I've started some initial analysis & prototyping on this already - I'm hoping to work on it more over the next several weeks.