CI: Consistent Caching #36
Comments
|
Another issue is that caches on main should be accessible to PRs but are not currently... (they do a full build of everything which is slow). I think we need to ensure that all caches are present on every commit to main. At the moment, by the look of the actions panel, not all caches are on main? I imagine fixing cache to be consistent naming would fix this anyways. https://stackoverflow.com/questions/69365200/github-actions-how-to-cache-dependencies-between-workflow-runs-of-different-bra |
|
This might require some more work (particularly with setting up the right compilers), but one option would be https://github.com/marketplace/actions/sccache-action The idea behind sccache is it hashes files (and all their includes) when they are built, and then later if you build the same thing, it just copies back the result. The advantage is you can just build everything ever time, and it should take very little time, and your cache can't get "out of sync", because if you change any files, they now won't be cached. I often do this on my own computer, to great success (it speeds up building minion from 3 minutes the first time, to 1.5 seconds the second time). I haven't tried using it on github actions I will admit. |
|
If you want to investigate this, at the moment minion doesn't support options to set the c++ compiler used, but I'm happy to add one. |
|
I think what we are already doing is fine - I just cache the entire of the minion build directory on github, then minion uses ccache internally. This makes rebuilds really fast. The problem is that each action uses a slightly different cache key that is calculated in different ways. Things would work better if there was one cache for the whole project, to be used across all actions. This would also have maintainability advantages when more solvers were added, as less files would need to be changed. I am actually forcing clean builds of minion at the moment on minion update - I imagine this is a holdover from the days where my build.rs were a bit broken and didn't rebuild stuff properly. (I also remember something not rebuilding properly minion side that you fixed?). I fixed this locally a while back, so I could probably remove this from the CI. |
|
Todo on completion: |
|
Also, for CI generally:
|
To clarify on this, Rust didn't used to regenerate the bindings if I had changed Minion. |
|
From Conjure's caching I think as long as you use the right cache-keys they can share the cache. The only problem is if we are generating a lot of stuff to be cached they can be kicked out of the cache (github gives us 10G). But this much storage should be enough for the most common tasks. |
|
I believe Minion should now correctly rebuild -- but let me know if you ever notice it misbehaving (either building too much, or too little) |
|
|
Alot of the CI jobs use their own caching mechanism that is hard coded to look at minion, chuffed, minion and chuffed, etc...
I am thinking of a shell script that gets the hash of all submodules from git dynamically, and hashes these, producing a hash that represents all submodule dependencies.
Using this, all caches can then be converted to the form:
This would make CI easier to update if we get new submodules, and allow more cache to be shared between jobs.
Also the paths to cache could be defined in one place (a script or text file in the repo).
(https://stackoverflow.com/questions/12641469/list-submodules-in-a-git-repository)
The text was updated successfully, but these errors were encountered: