Creating runfiles tree is very slow - because of hermetic Python #14424
apivovarov
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello Everyone
Couple month ago one commit was added to XLA main
Use hermetic Python in TSL and XLA
90a69e9
The issue is:
pypi_lit
andpython_x86_64-unknown-linux-gnu
folders. Total 161,343 additional filesCommand I used:
bazel test //xla/mlir_hlo/tests/...
Before / After stats:
Size of bazel-out/k8-opt/bin/xla/mlir_hlo/tests/Dialect/mhlo
before: 40M
after: 1.8G !!!
Creating runfiles tree
before: 1 second
after: 110 seconds !!!
Content of typical
xxx.mlir.test.runfiles
folder:Before (size 476K):
MANIFEST init.py llvm-project xla
After (size 22M):
MANIFEST init.py llvm-project
pypi_lit python_x86_64-unknown-linux-gnu xla
MANIFEST number of lines
before: 101 lines
after: 4,815 lines
bazel test //xla/mlir_hlo/tests/...
creates 63xxx.mlir.test.runfiles
folders containing the same copy ofpypi_lit
andpython_x86_64-unknown-linux-gnu
subfolderspython_x86_64-unknown-linux-gnu
folder contains 2490 filespypi_lit
contains 71 filesTotal 161,343 additional files !!! (for 63 tests)
Is it possible to give particular options/flags to bazel to use old fast approach?
Beta Was this translation helpful? Give feedback.
All reactions