CP-44752: propagate System.Diagnostics tracing information using W3C traceparent header

[edwintorok]

The newer System.Net.Http would propagate this automatically, but the older WebRequest doesn't.

Retrieve the span identifier and add the header ourselves. The WebRequest object is fresh for each request, so setting the headers shouldn't cause race conditions.

This adds a dependency on a library that is part of .Net source repository, but distributed outside of it on NuGet: System.Diagnostics. Lock the dependency to a given hash to avoid accidental/malicious changes.
This dependency is not available on .Net45, so all the code and build dependencies are wrapped with appropriate conditional compilation directives (which may look kind of ugly, they can be dropped once XenServer.dll drops support for .Net45 and switches to .Net462 minimum).

No new instrumentation is created, so if the application using XenServer.dll is not configured/set up to use OpenTelemetry or System.Diagnostics.DiagnosticSource, then this is a no-op.

I've tested this using the Powershell 7 module.

Draft PR because:

• I haven't tested the version with the locked dependencies yet
• There is one TODO in the code about obtaining a version number

[edwintorok]

Internal build succeeded with locked package dependencies too.

[edwintorok]

I think I'll also need NET462_OR_GREATER otherwise XC won't pick this up.

[danilo-delbusso]

haven't tested it yet but I spotted a couple of things on the first run

for SDK PRs also please request reviews manually, because we don't monitor the repo as often as toolstack does, so it helps to see PR reviews under https://github.com/pulls

[kc284]

As above.

[edwintorok]

I've addressed the review comments, and tested a build, the traceparent is sent correctly:

POST /jsonrpc HTTP/1.1
Host: lcy2-dt72.xenrt.citrite.net
Accept: application/json
User-Agent: XenServerPSModule/25.2.0.0
Connection: Keep-Alive
traceparent: 00-a29a037790a77a956ab8ce8887d1287d-900ac8ca433ee4f4-01
Content-Type: application/json
Content-Length: 100

[edwintorok]

I'll add a small self-contained example here on how the output looks with OpenTelemetry instrumentation enabled.

[edwintorok]

dotnet new console -o sample1
dotnet add package OpenTelemetry.Exporter.Console --version 1.9.0
dotnet add package OpenTelemetry --version 1.9.0
dotnet nuget add source --name 'Offline Packages' E:\myfolder\xen-api\sdk\sdk\XenServer-SDK\XenServer.NET\
dotnet add package XenServer.NET -s 'Offline Packages'

// See https://aka.ms/new-console-template for more information
using System.Net;
using System;
using System.Collections.Generic;
using XenAPI;
using System.Runtime.InteropServices;
using System.ComponentModel.DataAnnotations;
using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using System.Diagnostics;

ActivitySource source = new ActivitySource("Sample.DistributedTracing", "1.0.0");

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
                .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("MySample"))
                .AddSource("*")
                .AddConsoleExporter()
                .Build();

using (Activity activity = source.StartActivity("SomeWork"))
{
    //Trust all certificates. This is a test workaround. DO NOT USE IN PRODUCTION CODE!
    ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
    ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;

    string hostUrl = args[0], username = args[1], password = args[2];
    Session session = new Session(hostUrl);
    try
    {
        session.login_with_password(username, password, "", "XenSdkSample");

        var hosts = Host.get_all(session);
        System.Console.WriteLine("Got hosts: " + hosts.Count);
    }
    finally
    {
        session.logout();
    }
}

The output looks like this (the application can choose which activity sources to enable, if they want just the XenServer on, they can enable 'XenServer.*'):
https://gist.github.com/edwintorok/d1528660c588d6916363c8bb0565ba50

[edwintorok]

Tested using a slightly tweaked version of the above sample (using OTLP to send to jaeger), and confirmed that we get a joined Client + Server trace in Jaeger.

Review comments addressed in new commits. Removing stale review marker, and requested new review.

[mg12]

The code changes and output looks good to me, and the joined Client + Server trace in Jaeger shows it's working as intended. Perhaps someone more used to the SDK build system could check the dependency changes @kc284 @danilo-delbusso

[edwintorok]

It has been discussed that we should switch to .Net 8, so I'll see if I can add an additional build target for that here (that'll ensure that the right DLL gets picked out of the Nuget).

[edwintorok]

Looks like I need to tweak the preprocessor defines, because this doesn't cover .Net 8.0:
#if (NET462_OR_GREATER || NETSTANDARD2_0_OR_GREATER)

NET462 refers to .Net framework I think, so .Net 8.0 is not covered by it, and .Net Core and regular .Net probably have no overlap either.
I'll see if I can find a simpler preprocessor define to use, and if not add .Net 6.0+ to that.

[edwintorok]

The xenserver-samples fail to run:

\r\nException: System.IO.FileNotFoundException: Could not load file or assembly &apos;System.Diagnostics.DiagnosticSource, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51&apos;. The system cannot find the file specified.\r\nFile name: &apos;System.Diagnostics.DiagnosticSource, Version=8.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51&apos;\r\n   at XenAPI.JsonRpcClient.Rpc[T](String callName, JToken parameters, JsonSerializer serializer)\r\n   at XenAPI.JsonRpcClient.session_login_with_password(String _uname, String _pwd, String _version, String _originator)\r\n   at XenAPI.Session.login_with_password(String username, String password, String version, String originator)\r\n   at Citrix.XenServer.Commands.ConnectXenServerCommand.ProcessRecord()\r\n   at System.Management.Automation.CommandProcessor.ProcessRecord()\r\n</log>\r\n</test>\r\n<test>\r\n<name>Create SR create_nfs_sr 10.63.96.18 /mia11_export/4288734-QzbMXt PowerShellAutoTestSR True</name>\r\n<state>Fail</state>\r\n<log>\r\nCmd: 'create_nfs_sr 10.63.96.18 /mia11_export/4288734-QzbMXt PowerShellAutoTestSR3jigbrbkso6'\r\nException: System.Exception: Could not find open sessions to any XenServers.\r\n   at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)\r\n</log>\r\n</test>\r\n<test>\r\n<name>Install VM install_vm PowerShellAutoTestVM PowerShellAutoTestSR True</name>\r\n<state>Fail</state>\r\n<log>\r\nCmd: 'install_vm PowerShellAutoTestVM42q95qmrdwg PowerShellAutoTestSR3jigbrbkso6'\r\nException: System.Exception: Could not find open sessions to any XenServers.\r\n   at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)\r\n</log>\r\n</test>\r\n<test>\r\n<name>Start VM start_vm PowerShellAutoTestVM Running</name>\r\n<state>Fail</state>\r\n<log>\r\nCmd: 'start_vm PowerShellAutoTestVM42q95qmrdwg'\r\nException: System.Exception: Could not find open sessions to any XenServers.\r\n   at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)\r\n</log>\r\n</test>\r\n<test>\r\n<name>Shutdown VM shutdown_vm PowerShellAutoTestVM Halted</name>\r\n<state>Fail</state>\r\n<log>\r\nCmd: 'shutdown_vm PowerShellAutoTestVM42q95qmrdwg'\r\nException: System.Exception: Could not find open sessions to any XenServers.\r\n   at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)\r\n</log>\r\n</test>\r\n<test>\r\n<name>Uninstall VM uninstall_vm &apos;PowerShellAutoTestVM&apos; True</name>\r\n<state>Fail</state>\r\n<log>\r\nCmd: 'uninstall_vm &apos;PowerShellAutoTestVM42q95qmrdwg&apos;'\r\nException: System.Exception: Could not find open sessions to any XenServers.\r\n   at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)\r\n</log>\r\n</test>\r\n<test>\r\n<name>Destroy SR detach_nfs_sr PowerShellAutoTestSR True</name>\r\n<state>Fail</state>\r\n<log>\r\nCmd: 'detach_nfs_sr PowerShellAutoTestSR3jigbrbkso6'\r\nException: System.Exception: Could not find open sessions to any XenServers.\r\n   at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)\r\n</log>\r\n</test>\r\n<test>\r\n<name>Disconnect Server</name>\r\n<state>Pass</state>\r\n<log />\r\n</test>\r\n</group>\r\n</results>\r\n" (

Maybe the dependencies don't get propagated correctly from the SDK to the app, although I would've expected Nuget to do the right thing.

I'll try another build where I only enable tracing when you're on .Net 8.0+ and not in other cases (that means that XenCenter likely won't get it, and we'll also lack tracing in Powershell -- which was an easy way to test that the tracing works).

The C# sample fails to run too, but that probably needs the environment updated to have .Net 8.0 available (I switched the sample from 6.0 to 8.0):

You must install .NET to run this application.

App: /root/csharp/XenSdkSample
Architecture: x64
App host version: 8.0.13
.NET location: Not found

Learn more:
https://aka.ms/dotnet/app-launch-failed

Download the .NET runtime:
https://aka.ms/dotnet-core-applaunch?missing_runtime=true&arch=x64&rid=linux-x64&os=debian.12&apphost_version=8.0.13

[edwintorok]

With the latest changes here (enable tracing only for .NET 8.0), and updating the tests to install the .Net 8 runtime when testing the .Net 8 sample -- the tests have passed, so I think this is ready.

Addressed comments