Performance Upgrade

[patrickcleeve2]

Planned Performance Upgrades
The follow performance bottlenecks have been identified as future areas of development.

GPU Acceleration

• The main simulation wave propagation and fourier transforms have now been vectorised and is just matrix multiplication, which should be a target for GPU acceleration.

Cached Results

• When running a parameter sweep, especially with multiple stages, a large number of identical stages are re-run. Therefore there is an opportunity to increase performance by using cached results.
• The cache would re-use the results of blocks that are identical to previously run simulations (might be limited to the same run).
• Therefore, we need to develop a caching system to identify identical simulation stages, and use their result instead to reduce redundant simulations.
  

Distributed / HPC mode

• Currently the parameter sweep runs sequentially on a single machine, but each simulation is independent. Therefore there is an opportunity to incorporated parallel, or distributed operations, including incorporating HPC cluster scheduling such as https://jobqueue.dask.org/en/latest/
• This should provide a directly linear speed up to simulation speed since each simulation is completely independent.
  

Remove Matplotlib Plots

• Based on profiling, creating and saving these matplotlib plots is creating a significant (~40+%) overhead which is currently the bottleneck for single simulation performance.
• These plots were required before switching to persistent zarr arrays as we were unable to store the simulation data efficiently. However, they are no longer required as the majority of the plots required can be recreated from the configuration file, or the simulation zarr.
• Therefore they should be removed, and if the debugging plots are required (freq, delta, propagation, etc) we should write these to zarr, and generate the plots only when the user requires.