Ghidra Trace Format

[pjsoberoi]

@d-millar,

Similar to this discussion on the GADP protocol, I would like to request the creation of an open source Ghidra trace format. I can see a lot of ideas for tools that can use the trace format.

[d-millar]

@pjsoberoi going for the easy requests, I see. :)

Would love to see an open-source trace format, but better programmers than I have tried and failed on this front. Again, just for the sake of discussion, what would you want for the proposed format:

• optimized for collection speed
• optimized for trace size
• optimized for post-processing speed
• optimized for random access
• should it record only the instruction sequence
• record enough context to recreate state
• record enough timing information to recreate multithreaded state
• should it allow for custom events
• should it be reversible or quasi-reversible
• should it be optimized for differential analysis
• should it be compressed or human-readable
• should it be optimized for ease of use and customization

Obviously many of these options are mutually exclusive, and I’m not sure you could even option your way through them. Also, there’s the whole issue of adoption - if you’re shooting for widespread adoption, the format has to at least be able to match the constraints of possible generators.

Just my two cents. Would be interested in ideas/discussion of the possibilities and possible use cases.

[gourryinverse]

@d-millar i realize my post was long, and maybe beyond the intent of @pjsoberoi 's post, but having gone in loops around trace and coverage formats for probably going on 5 years, adding "yet another format" to the pile is far less important than understanding what people are trying to do.

Maybe more succinctly - this task / discussion should probably be around Ghidra Trace formatS (plural).

[d-millar]

@gourryinverse Been there as well and agree completely with everything you’ve said. That’s why, for instance, we’ve implemented access to windbg/kd TTD traces through their API, rather than importing their traces directly. Am considering doing the same for the Intel PIN API and (if it exists - I haven’t done enough research yet) for the Intel PT API. (Any pointers on the latter would be greatly appreciated!)

I do believe tracing is one of those areas like visualization - it has a lot of promise and is very seductive, but is easy to overestimate. Targeted tracing has, for me at least, been much more protective in the past. That said, I do understand PJ’s use case, and it’s quite possible none of the aforementioned technologies really solves his problem. I do think the more opinions/insights we can get on this the better. It’s an extremely interesting, although definitely research-required problem.

[d-millar]

I should also mention having a few more test cases, such as Intel PT or RR, would allow us to flesh out our ideas about interaction between the Ghidra “traces”, which are more of a session record in database format, and more global trace formats, particularly in the area of time-scale information and reversible debugging interfaces.

[gourryinverse]

@d-millar if you want a crash-course on Intel PT, I'd be happy to give you 20-30 minutes and do a quick walk through of what I know. I would say it's a pretty well documented binary stream, the issue with it is that if you have real Intel PT data (produced by the chip, rather than from an emulated system or such), you might not necessarily be able to "jump to" any particular part of the trace and make sense of it, due to 1) Variable packet length, and 2) a variety of compression cheats they use that make decoding particularly expensive.

[d-millar]

@gourryinverse That would be extremely useful! Am traveling this week, but am definitely interested. Will give you a shout when I get back.

[pjsoberoi]

Obviously you have spent much more time researching this area than I have. I have not considered possible format optimizations at all.
One use case I'm interested in is processor and processor module verification. For that I would need all instructions, all registers, and memory deltas.

The first idea that came to mind is to maybe have a simple raw trace format that can be optimized post collection into another format. The goals would be to optimize for collection speed, simple format to implement (for reading and writing), and tools to convert to other optimized formats for different uses cases.

• optimized for collection speed

Yes.

• optimized for trace size

No. Another program can optimize the trace afterwards into another format.

• optimized for post-processing speed

No. Another program can optimize the trace afterwards into another format.

• optimized for random access

No. Another program can optimize the trace afterwards into another format.

• should it record only the instruction sequence

Maybe have flags for instructions, instructions + regs, instructions + regs + memory deltas. I can do useful stuff with just instructions + regs, I don't know what I could do with just instructions.

• record enough context to recreate state

All state encapsulated by instructions + regs + memory deltas seems doable. State such as other processes, kernel, network stack, etc seem really hard. I would skip that for now.

• record enough timing information to recreate multithreaded state

Sure. But that can be an option. Seems hard so skip it for now.

• should it allow for custom events

No. Another program can optimize the trace afterwards into another format and that can handle custom events.

• should it be reversible or quasi-reversible

Yes.

• should it be optimized for differential analysis

No. Another program can optimize the trace afterwards into another format.

• should it be compressed or human-readable

No. Another program can take the trace as input and transform it to something human readable.

• should it be optimized for ease of use and customization

Yes.

[d-millar]

A bit random, but wanted to say I haven't forgotten about this - just haven't had time to really take it on.

[nsadeveloper789]

Related: #2730 (comment)