Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Profiling for Nodes and Models #5

Open
mdraw opened this issue May 19, 2017 · 0 comments
Open

Profiling for Nodes and Models #5

mdraw opened this issue May 19, 2017 · 0 comments

Comments

@mdraw
Copy link
Member

mdraw commented May 19, 2017

We tried to implement a system for per-Node and whole-Model performance profiling by measuring local (i.e. only the operation itself) and total (i.e. the whole operation with all its parents) execution times of Node objects.
Related code:

  • elektronn2.neuromancer.node_basic.Node.measure_exectime() (main implementation)
  • elektronn2.neuromancer.node_basic.Node.last_exec_time
  • elektronn2.neuromancer.node_basic.Node.local_exec_time
  • elektronn2.neuromancer.node_basic.Node.total_exec_time
  • elektronn2.neuromancer.model.Model.measure_exectimes()
  • elektronn2.utils.plotting.plot_exectimes() (visualises execution times of Nodes by plotting them into a file)
  • elektronn2.utils.d3viz.formatting.visualise_model() (No relevant code yet, but this function should eventually be able to optionally convey profiling results directly in the computation graph)

There are unsolved issues with this profiling code: While the measurements for some small example models often look good, for some nodes in larger models we get negative results for local execution times (which is obviously wrong, because that would mean the respective operation does some time travel magic and finishes even before starting the measurement).
Our first thought was that this has something to do with theano's automatic graph optimization, but even when disabling optimizations, we encountered this problem.
We should re-evaluate this issue after the port to the new backend (#4), because it could be a specific to the old backend.
If we can't resolve these issues, the relevant code (listed above) should be removed, otherwise, the functionality should be properly documented.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant