You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
With the introduction of haskell_module and narrowed dependencies we only depend on interface files of dependencies and do not include object files in build action inputs (unless required for Template Haskell) (#1623, #1654, #1663). However, interface files are generated by the module build action along side the modules object file. Meaning, following build actions will have to wait for both, the generation of the interface file they depend on, and the generation of the corresponding object file, even if they don't depend on it, before they can be executed.
The Rust compiler recently gained support for "pipelined" compilation. I.e. compilation of downstream modules can already be started when only the required metadata files of their dependencies have been generated. Such compilation does not need to wait for the generation of object files that it does not depend on. This can improve build parallelism and reduce the critical path.
rules_rustis considering adding support for this feature, and an implementation for Buck v2 exists according to this comment.
And according to this StackOverflow discussion it is possible to generate interface files alone with GHC and skip code-generation.
Describe the solution you'd like
Bazel has no direct support for a concept like pipelined compilation. An approximation can be achieved by defining two separate build actions: one to generate the interface file, and another to generate the object file. Downstream actions depending only on the interface file can start once the interface file generation of its dependencies completed and will not have to wait for the action performing code-generation to complete.
That said, to my knowledge, GHC cannot resume compilation based on a previously stored state. So, the action performing code-generation will have to repeat the work performed during interface file generation. The announcement of the Rust feature includes a paragraph about when it may still be beneficial to perform pipelined compilation despite such duplication of effort under the heading "When is pipelined compilation faster?".
Describe alternatives you've considered
The persistent worker could be modified such that it can generate the interface file first on a dedicated interface file generation action, but, maintain state until later an object file generation action continues the work based on that maintained state without having to duplicate work. However, this likely introduces a lot of additional complexity into the already complex problem of creating a persistent worker.
Additional context
I benchmarked interface file generation vs. full compilation on a simple example module. This example is probably not representative of most real-world code. So, further benchmarking would be needed to determine if such a feature would really be beneficial. The purpose here was just to get a first, rough gauge on how interface file generation and code generation relate to each other in terms of build time.
The example module was generated using the following script:
#!/usr/bin/env bash
cat >SimpleExample.hs <<EOF
module SimpleExample where
EOF
for ((i = 0; i < 100; i++)); do
cat >>SimpleExample.hs <<EOF
a$i :: Int -> Int
a$i x = a$((i - 1)) x + 1
EOF
done
The benchmark results are as follows:
(without optimization)
$ hyperfine -p "rm -f *.hi *.o" "ghc SimpleExample.hs" "ghc -fno-code -fwrite-interface SimpleExample.hs"
Benchmark 1: ghc SimpleExample.hs
Time (mean ± σ): 204.7 ms ± 18.0 ms [User: 164.5 ms, System: 36.0 ms]
Range (min … max): 174.4 ms … 231.9 ms 14 runs
Benchmark 2: ghc -fno-code -fwrite-interface SimpleExample.hs
Time (mean ± σ): 143.3 ms ± 21.4 ms [User: 108.8 ms, System: 31.1 ms]
Range (min … max): 93.5 ms … 172.6 ms 21 runs
Summary
'ghc -fno-code -fwrite-interface SimpleExample.hs' ran
1.43 ± 0.25 times faster than 'ghc SimpleExample.hs'
(with optimization)
$ hyperfine -p "rm -f *.hi *.o" "ghc -O SimpleExample.hs" "ghc -O -fno-code -fwrite-interface SimpleExample.hs"
Benchmark 1: ghc -O SimpleExample.hs
Time (mean ± σ): 231.1 ms ± 13.5 ms [User: 193.4 ms, System: 34.3 ms]
Range (min … max): 206.5 ms … 249.5 ms 12 runs
Benchmark 2: ghc -O -fno-code -fwrite-interface SimpleExample.hs
Time (mean ± σ): 140.6 ms ± 20.4 ms [User: 106.6 ms, System: 29.8 ms]
Range (min … max): 97.7 ms … 171.8 ms 29 runs
Summary
'ghc -O -fno-code -fwrite-interface SimpleExample.hs' ran
1.64 ± 0.26 times faster than 'ghc -O SimpleExample.hs'
I.e. without optimization interface file generation takes about 70% of the build time, and with optimization about 60%.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
With the introduction of
haskell_module
and narrowed dependencies we only depend on interface files of dependencies and do not include object files in build action inputs (unless required for Template Haskell) (#1623, #1654, #1663). However, interface files are generated by the module build action along side the modules object file. Meaning, following build actions will have to wait for both, the generation of the interface file they depend on, and the generation of the corresponding object file, even if they don't depend on it, before they can be executed.The Rust compiler recently gained support for "pipelined" compilation. I.e. compilation of downstream modules can already be started when only the required metadata files of their dependencies have been generated. Such compilation does not need to wait for the generation of object files that it does not depend on. This can improve build parallelism and reduce the critical path.
rules_rust
is considering adding support for this feature, and an implementation for Buck v2 exists according to this comment.And according to this StackOverflow discussion it is possible to generate interface files alone with GHC and skip code-generation.
Describe the solution you'd like
Bazel has no direct support for a concept like pipelined compilation. An approximation can be achieved by defining two separate build actions: one to generate the interface file, and another to generate the object file. Downstream actions depending only on the interface file can start once the interface file generation of its dependencies completed and will not have to wait for the action performing code-generation to complete.
That said, to my knowledge, GHC cannot resume compilation based on a previously stored state. So, the action performing code-generation will have to repeat the work performed during interface file generation. The announcement of the Rust feature includes a paragraph about when it may still be beneficial to perform pipelined compilation despite such duplication of effort under the heading "When is pipelined compilation faster?".
Describe alternatives you've considered
The persistent worker could be modified such that it can generate the interface file first on a dedicated interface file generation action, but, maintain state until later an object file generation action continues the work based on that maintained state without having to duplicate work. However, this likely introduces a lot of additional complexity into the already complex problem of creating a persistent worker.
Additional context
I benchmarked interface file generation vs. full compilation on a simple example module. This example is probably not representative of most real-world code. So, further benchmarking would be needed to determine if such a feature would really be beneficial. The purpose here was just to get a first, rough gauge on how interface file generation and code generation relate to each other in terms of build time.
The example module was generated using the following script:
The benchmark results are as follows:
(without optimization)
(with optimization)
I.e. without optimization interface file generation takes about 70% of the build time, and with optimization about 60%.
The text was updated successfully, but these errors were encountered: