Add support for splitting Perfetto traces

[tingfengwanxiao]

When I use as ./traceconv.exe systrace 1.pftrace 2.ftrace,I get a empty 2.ftrace,
it seems because my 1.pftrace is too large as 8G,so How can I change so large perfetto trace to ftrace,
or how can I directly capture a ftrace by som command as large as 8G?

[LalitMaganti]

Are you sure it's actually because the file is large? Have you tried with a smaller file and things work correctly?

I would be a bit surprised if the file being large was an issue albeit it's not impossible I guess.

Just to be clear, if you have an 8GB Perfetto file, that will turn into a 20GB or more systrace file. What are you planning on doing with that in the end?

[tingfengwanxiao]

I want to capture a long-term trace and then divide it into smaller ones at certain time points, but perfetto trace seems unable to be directly divided, so I want to convert it into ftrace and use text processing to divide it. Do you have any good suggestions

[LalitMaganti]

Why not just collect many small traces in the first place? Would be a lot easier than capturing a big trace and then trying to split it.

[tingfengwanxiao]

Why not just collect many small traces in the first place? Would be a lot easier than capturing a big trace and then trying to split it.

I don't know when I need the trace,so I need to collect it all

[LalitMaganti]

Fine I don't think it's worth discussing more in that direction.

Back to the actual topic at hand: would still be useful to know if this is only an issue on big traces or also happening on smaller traces. Especially interesting is if the cut-off is 4GB.

[tingfengwanxiao]

The small ones won't happen, the large ones will happen multiple times, and the converted files will be empty

[LalitMaganti]

Do you have enough memory on your system to have the full trace in memory i.e. at least 20GB of free memory? Because systrace conversion works by first parsing the full trace into trace processor tables before then doing the conversion.

[LalitMaganti]

Specifically, what happens if you just try and just parse the trace with trace_processor_shell? Does it work?

[tingfengwanxiao]

Do you have enough memory on your system to have the full trace in memory i.e. at least 20GB of free memory? Because systrace conversion works by first parsing the full trace into trace processor tables before then doing the conversion.

I only have 16G,maybe its reason

[tingfengwanxiao]

Then how can I do for this large trace? split it by time for some small trace

[LalitMaganti]

There's not really a good solution for this today unfortunately. Everyone who needs this sort of thing today just collects smaller traces in the first place.

[tingfengwanxiao]

ok ,thank you !

[LalitMaganti]

I've changed this bug to track a better implementation of trace splitting: we might implement it at some point but for us it's not a super high priority.

[tingfengwanxiao]

Can I use trace_processor_shell query the trace such as "select * from slice where slice > start.ts and slice.ts<end.ts" to get the trace between them and save the new trace？

[tingfengwanxiao]

Or whether it is possible to read in batches instead of all at once

Do you have enough memory on your system to have the full trace in memory i.e. at least 20GB of free memory? Because systrace conversion works by first parsing the full trace into trace processor tables before then doing the conversion.

[LalitMaganti]

No neither is possible.

[LalitMaganti]

Also to use trace processor with such a large trace in the first place still requires you to have enough memory to hold the full trace in memory.