Question: how to enable TracedObjectStore?

[yanghua]

I found we have a TracedObjectStore. However, this feature is not used. My question is: How do I enable it? Or can it work?

[yanghua]

cc @westonpace @wjones127

[westonpace]

TracedObjectStore is used by default. I actually don't think you can disable it.

So perhaps the question is more, how can we consume the events emitted by TracedObjectStore?

Right now, there is only one outlet for span traces in pylance and that is to use trace_to_chrome:

from lance.tracing import trace_to_chrome

trace_to_chrome(level="debug", file="/some/file.json")

This will dump all tracing spans into a file using the chrome event viewer format (you can view this file with https://ui.perfetto.dev/)

Is there another way you would like to consume these spans? Perhaps via log messages or some kind of otel integration?

[yanghua]

Oh, I thought it was disabled by default.

If it is open by default, I would prefer to see it in the log message.

[yanghua]

@westonpace It seems the generated file with trace_to_chrome can not be parsed by the popular flamegraph tool.

It throws this exception:

Traceback (most recent call last):
  File "./stackcollapse-chrome-tracing.py", line 139, in <module>
    all_events = load_events(args.input_file)
  File "./stackcollapse-chrome-tracing.py", line 83, in load_events
    get_trace_events(trace_file, events)
  File "./stackcollapse-chrome-tracing.py", line 69, in get_trace_events
    for entry in json_data['traceEvents']:
TypeError: list indices must be integers, not str

Is there a recommended way to generate flame diagrams based on this JSON file?

[yanghua]

Seems there is a library tracing-flame, after importing this, we can add a FlameLayer like this:

use tracing_flame::FlameLayer;
use tracing_subscriber::prelude::*;

let (flame_layer, _guard) = FlameLayer::with_file("./tracing.folded").unwrap();
tracing_subscriber::registry()
    .with(flame_layer)
    .init();

[westonpace]

Yeah, I haven't bothered much with flamegraphs with traces. Normally I look at trace graphs with https://ui.perfetto.dev/ instead.

I think adding a trace_to_flame (or whatever to name it) is fine if you want to add a PR with that. There are many possible outputs for this kind of data. Another one I hope to have added at some point is tools like open telemetry or prometheus.

I do think I tried some experiments with FlameLayer and found that it only reported the active time (e.g. CPU time) which wasn't as useful for the kind of profiling I was hoping to do.

[westonpace]

Hmm, I was just trying to use some traces on main and it seems that chrome tracing is broken. I'll investigate. That may explain why stackcollapse-chrome-tracing didn't work 😦