Benchmark solvers

Summary

This repository contains code for benchmarking the performance of exact algorithm solvers for conservation planning problems (see Mittelmann 2021 for general purpose benchmarks). Specifically, these benchmarks examine the performance of the Gurobi and IBM CPLEX commercial solvers. They also examine the CBC (COIN-OR branch and cut), HiGHS, and SYMPHONY (via the Rsymphony and lpsymphony R packages) open source solvers. The benchmarks are performed using a realistic conservation planning dataset, involving 72 species and costs derived from land valuation data (see Schuster et al. 2020). To investigate how well different solvers scale with different sized datasets, these benchmarks examine different versions of the dataset wherein planning units are resampled to different resolutions. The results of benchmark analysis are reported in the Benchmark vignette of the prioritizr R package.

Usage

After downloading (cloning) this repository, you can rerun the analysis on your own computer using the system command make clean all. Please ensure that the MODE variable in the Makefile is set to release to obtain correct results. Project maintainers can then use the make export command to upload the results files (results/results.rda and results/solutions.tif) to the Results storage release.

Repository structure

This repository is organized as follows:

• data:
  • raw: Raw data used to run the analysis (see Data storage release for large files).
  • intermediate: Intermediate data generated during analysis.
  • final: Final data created after completing the analysis.
• code:
  • parameters: Settings for running the analysis (TOML format).
  • R: Code used to run the analysis.
• results:
  • results.rda: Rdata file containing the results.
  • solutions.tif GeoTIFF files containing solutions form benchmark runs.
• renv: R package management.

Note that the files in the results directory are created after completing benchmark analysis. They are not present in the online repository. If you wish to access the results completed on our systems, please see files attached the Results storage release.

Software required

• Software
  • GNU make (version 4.1+)
  • GDAL (version 2.2+)
  • Gurobi (academic licenses are available for no cost)
  • IBM CPLEX (academic licenses are available for no cost)
  • R (version 4.3.1+)
• System packages (Ubuntu operating system):
  • gdal-bin
  • libjq-dev
  • libprotobuf-dev
  • protobuf-compiler
  • libudunits2-dev
  • libgdal-dev
  • libgeos-dev
  • libproj-dev
  • libv8-dev
  • libxml2-dev
  • libssl-dev
  • libgit2-dev
  • coinor-libcbc-dev
  • coinor-libclp-dev
  • coinor-libsymphony-dev

References

Schuster R, Hanson JO, Strimas-Mackey M and Bennett JR (2020) Exact integer linear programming solvers outperform simulated annealing for solving conservation planning problems. PeerJ, 8: e9258.

Mittelmann H (2021) Benchmarks for optimization software. http://plato.asu.edu/bench.html [accessed February 2021].