Implement ftrace pid filtering support in traced_probes

[wangjiyang]

Hi Perfetto team.

I am looking for a ftrace filter feature. Background is I am trying to tracing sched:sched_switch ftrace event of some specified processes for a very very long time, eg, serveral days. Enabling sched_switch events for all processes will produce too much data and may consumes CPU and disk. The overhead is too much for our usecases. So I wonder what's the best practice of tracing limited number of processes?

[LalitMaganti]

We don't support any such feature natively in Perfetto. If you want to setup filtering outside Perfetto (i.e. by creating an ftrace synthetic event or directly writing to the /sys files) you can do that and just use Perfetto as a recording mechanism.

Note that any filtering may lead to traces which don't make sense in trace processor: there are various assumptions in trace processor that if see an event like sched_switch, we are seeing all the events of that type. Having inconsistencies can lead to undefined behaviour.

[LalitMaganti]

To be clear, having this functionality in Perfetto is a good idea and I think something we would be supportive of. It's just quite a niche feature (over 7 years, I've only heard it requested 2 or 3 times) and not the priority for us.

I'll reopen this as if someone wants to contribute this, we'd be supportive of adding it (you can look at how the syscall filtering works for inspiration). But it's probably not high enough priority for us to look at ourselves.

[wangjiyang]

I would use some ebpf features to do such filtering. Eg, calling bpf helpers like bpf_trace_printk, and acts as what atrace done to take advantage of ftrace features to link with perfetto. I am still investigating and not sure if traced_probe already has this kind of raw ftrace buffers collected. Please share some information if you know. Based on this next step would be parsing this raw buffer and emit track information to trace processor. This approach would be compatible with current ftrace importer, but has some challege because that would need to create a new data source and it depends on ebpf.

[wangjiyang]

Seems this is what i needed. Is there any samples?

https://perfetto.dev/docs/reference/trace-config-proto#PerfEvents.Tracepoint

[LalitMaganti]

Eg, calling bpf helpers like bpf_trace_printk, and acts as what atrace done to take advantage of ftrace features to link with perfetto.

If you are doing this, you don't need to do any integration with perf etc. You just need to enable the ftrace events ftrace/print, ftrace/bprint and ftrace/bputs and the data should just show up in Perfetto.

https://perfetto.dev/docs/reference/trace-config-proto#PerfEvents.Tracepoint

This is totally irrelevant, this is about traced_perf stack sampling. You don't need this, just enable the events above in the normal ftrace data source.