You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We tried to implement a system for per-Node and whole-Model performance profiling by measuring local (i.e. only the operation itself) and total (i.e. the whole operation with all its parents) execution times of Node objects.
Related code:
elektronn2.utils.plotting.plot_exectimes() (visualises execution times of Nodes by plotting them into a file)
elektronn2.utils.d3viz.formatting.visualise_model() (No relevant code yet, but this function should eventually be able to optionally convey profiling results directly in the computation graph)
There are unsolved issues with this profiling code: While the measurements for some small example models often look good, for some nodes in larger models we get negative results for local execution times (which is obviously wrong, because that would mean the respective operation does some time travel magic and finishes even before starting the measurement).
Our first thought was that this has something to do with theano's automatic graph optimization, but even when disabling optimizations, we encountered this problem.
We should re-evaluate this issue after the port to the new backend (#4), because it could be a specific to the old backend.
If we can't resolve these issues, the relevant code (listed above) should be removed, otherwise, the functionality should be properly documented.
The text was updated successfully, but these errors were encountered:
We tried to implement a system for per-Node and whole-Model performance profiling by measuring local (i.e. only the operation itself) and total (i.e. the whole operation with all its parents) execution times of Node objects.
Related code:
elektronn2.neuromancer.node_basic.Node.measure_exectime()
(main implementation)elektronn2.neuromancer.node_basic.Node.last_exec_time
elektronn2.neuromancer.node_basic.Node.local_exec_time
elektronn2.neuromancer.node_basic.Node.total_exec_time
elektronn2.neuromancer.model.Model.measure_exectimes()
elektronn2.utils.plotting.plot_exectimes()
(visualises execution times of Nodes by plotting them into a file)elektronn2.utils.d3viz.formatting.visualise_model()
(No relevant code yet, but this function should eventually be able to optionally convey profiling results directly in the computation graph)There are unsolved issues with this profiling code: While the measurements for some small example models often look good, for some nodes in larger models we get negative results for local execution times (which is obviously wrong, because that would mean the respective operation does some time travel magic and finishes even before starting the measurement).
Our first thought was that this has something to do with theano's automatic graph optimization, but even when disabling optimizations, we encountered this problem.
We should re-evaluate this issue after the port to the new backend (#4), because it could be a specific to the old backend.
If we can't resolve these issues, the relevant code (listed above) should be removed, otherwise, the functionality should be properly documented.
The text was updated successfully, but these errors were encountered: