Skip to content

Commit

Permalink
Clarify another problem.
Browse files Browse the repository at this point in the history
  • Loading branch information
@itamarst
itamarst committed Nov 19, 2020
1 parent 16e2e03 commit 88153bc
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,11 @@ If you start a pool of threads that are not Python threads, the Python code that

Therefore, in order to ensure correct memory tracking, Fil disables thread pools in BLAS (used by NumPy), BLOSC (used e.g. by Zarr), OpenMP, and `numexpr`.
They are all set to use 1 thread, so calls should run in the calling Python thread and everything should be tracked correctly.
The downside is that this can reduce performance in some cases, since you're doing computation with one CPU instead of many.

This has some costs:

1. This can reduce performance in some cases, since you're doing computation with one CPU instead of many.
2. Insofar as these libraries allocate memory proportional to number of threads, the measured memory usage might be wrong.

Fil does this for the whole program when using `fil-profile run`.
When using the Jupyter kernel, anything run with the `%%filprofile` magic will have thread pools disabled, but other code should run normally.
Expand Down

0 comments on commit 88153bc

Please sign in to comment.