duration_μs cannot be handled by DataDog

[pirvudoru]

Hello,

Seems that DataDog does not like the "duration_μs" as path anywhere -- cannot remap, cannot create a metric from it. -- From LoggerJSON.Plug

Can we change it to something else ?

I've just updated from 5.1.3 to latest 6.03

[AndrewDryga]

I think a good way to solve this would be to sanitize all the metadata key names in DataDog adapter, a PR would be welcome.

[AndrewDryga]

Hello @bvobart @pirvudoru and @guzishiwo. Sorry for taking so long to reply. I pushed the simplest possible fix to main - replace μs with us. Most destinations do not support unicode so I feel like there is no good reason trying to fight to keep it.

Please give it a try. I will push the release to hex in a few days.

[bvobart]

Alright that sounds like a good solution for the general case, @AndrewDryga, but for the Elastic formatter, duration_us is still not a valid ECS field, so it would still be best to map that to event.duration. I don't think Elastic and Kibana will crash or error when they encounter duration_us, but they might well ignore it or at least not do anything useful with it. I'll rebase my MR (#129) and reopen it to fix that.

I see DataDog has a similar predefined format for a duration field (they recommend just using duration with the value in nanoseconds), so a similar remap will need to happen for the DataDog formatter. Same for the Google Cloud formatter too, though I'm not sure what field they use.

To be clear, I do think it's good to keep the generic duration field as duration_us because that at least specifies which unit the duration is in. But the product-specific formatters will need to adhere to those products' log specs.

EDIT: see #132

[AndrewDryga]

@bvobart I'm sorry for missing that out, lets merge your fix.

[AndrewDryga]

TODO for myself:

• Remap duration to HTTPRequest latency for GCP: https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#HttpRequest

[AndrewDryga]

The fix will be released in 6.2.0.