-
Notifications
You must be signed in to change notification settings - Fork 341
reasons for toplev measuring issues
Some nodes do not count on toplev. Here are some common reasons:
-
Your toplev is too old.
-
Your kernel is too old. Please see toplev kernel support
-
the event truly does not happen in your workload (that is not uncommon)
perf fails when running toplev. toplev makes heavy use of the PMU and is quite stress full for perf, which can result in it triggering perf problems.
The most common cause is when perf runs out of file descriptors. Either become root and increase ulimit -n .... to a larger value (you can su back to your original user then) or edit /etc/security/limits.conf and increase nofile there. In -a mode perf needs a file descriptor for each event on each logical CPU. A simple workaround is to dial back the levels or disable metrics.
This error can also happen when the Linux kernel does not support some event generated by toplev (for example a trace event). Pass -vv to perf (-- -v in toplev) to find out which event failed.
When using more than Level 1 or enabled metrics the kernel has to use counter multiplexing to measure all events. This causes the kernel to switch the events every 1ms - 10ms (depending on the timer tick, normally 2.5ms or 1ms) which causes multiplexing errors.
The CPU can normally only measure 4 counters at a time, or 8 if Hyper Threading is disabled. Multiplexing assumes the workload is "steady state" (does mostly the same) and runs reasonably long. The worst case for multiplexing is a workload which runs lots of different kinds of code in intervals less than the multiplex interval.
toplev tries hard to fit events from the same formula into the same group (a group is a unit of events that runs at the same time), but with some of the deeper levels that is not always possible.
Some workarounds for multiplexing issues:
-
Generally lower levels have less multiplexing. Disabling metrics can also lower multiplexing.
-
Let the program run longer.
-
Make the program more predictible, for example by minimizing any short periodic actions.
-
Measuring the initialization phase can often lead to problems because it tends to be short and very different from the rest of the program. Consider skipping it with --initial-delay.
-
You can play around with the setting in /sys/devices/cpu/perf_event_mux_interval_ms (note set it to 0 again when you're finished). WARNING: This will impact the power consumption and Turbo behavior of the system and should only be done if absolutely needed. I would be interested in feedback if different multiplexing intervals fix issues.
-
If your workload is very reproducible it's possible to disable multiplexing with --no-multiplex. This will rerun the workload as many times as needed to collect the necessary data.
When "-v" or "--verbose" is used, toplev does not use the TopDown thresholding mechanism. Many TopDown nodes are only meaningful when their parent nodes crossed their thresholds. So try disabling verbose mode. Without verbose toplev will also automatically hide out of range values.