Address guest tracing improvements

[dblnz]

The current version using opentelemetry is an incremental step, after which additional tunables can be added to improve usability.

Improvements to be considered:

• Fix max tracing level based on the max log level the guest is provided
• The guest logging shall use the same logic as tracing for reporting back to host. Maybe we can have spans/events emitted as logs in the guest.
• Change the tracing underlying storage to a circular buffer for traces/events to avoid context switches. This ensures the guest doesn't do host calls to further degrade performance. Trace/logs loss is an acceptable compromise
• Provide a mechanism on the guest to enable/disable the tracing support, configure the memory used and log level at runtime
• Consider changing to a heap allocated circular buffer, which is runtime configurable

[simongdavies]

Trace/logs loss is an acceptable compromise

I think this is only true if the host can make the decision that they are willing to suffer log/trace loss.

[dblnz]

Having the configuration option on the guest side, the guest can say how much space it is acceptable to use for traces/logs buffering.
Making VMExits to report traces has a performance penalty.

[simongdavies]

Having the configuration option on the guest side, the guest can say how much space it is acceptable to use for traces/logs buffering. Making VMExits to report traces has a performance penalty.

Understood, I am saying that we should not make the decision on if to trade off data loss for perf for a host, they should make that decision themselves.