Check SCS performance on DUALC1 #19
Comments
|
Here's a fresh run of DUALC1 with my code using the same settings as you I believe (1e-4 for both eps_abs and eps_rel, all other settings just the defaults): I haven't tried the others from that list, but they are likely to be similar. |
|
When I run it from your code (the accuracies are 1e-7 here, but that should be ok): That's quite strange behaviour, and I would guess somehow the problem is not getting passed in correctly. For one, the problem dimensions between your run and mine are quite different. My guess is that come constraints that could / should be box constraints are being set as linear constraints, and that this problem has very high (ie, irrelevant) box constraints (like 1e10 or something) which is killing performance. |
|
(Stashing my notes on DUALC1 here, feel free to skip this message.) Checking the problem DUALC1 from its QPS file directly: The problem has this structure:
|
You're right 🚀 Not the box constraints which are easy in this problem, but the infinities in I've filtered out +infinities from $ python ./benchmark.py check_problem maros_meszaros_dense DUALC1
[2022-11-11 19:14:17,874] [info] Loading existing results from ./results/maros_meszaros_dense.csv (results.py:59)
[2022-11-11 19:14:18,147] [info] Check out `problem` for the DUALC1 problem (benchmark.py:170)
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.0.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: x, t = problem.solve(solver="scs", eps_abs=1e-4, eps_rel=1e-4, verbose=True)
------------------------------------------------------------------
SCS v3.2.2 - Splitting Conic Solver
(c) Brendan O'Donoghue, Stanford University, 2012
------------------------------------------------------------------
problem: variables n: 9, constraints m: 225
cones: z: primal zero / dual free vars: 1
l: linear vars: 214
b: box cone vars: 10
settings: eps_abs: 1.0e-04, eps_rel: 1.0e-04, eps_infeas: 1.0e-07
alpha: 1.50, scale: 1.00e-01, adaptive_scale: 1
max_iters: 100000, normalize: 1, rho_x: 1.00e-06
acceleration_lookback: 10, acceleration_interval: 10
lin-sys: sparse-direct-amd-qdldl
nnz(A): 1944, nnz(P): 45
------------------------------------------------------------------
iter | pri res | dua res | gap | obj | scale | time (s)
------------------------------------------------------------------
0| 1.07e+03 3.16e+06 6.01e+05 -6.42e+06 1.00e-01 9.85e-04
175| 8.89e-07 1.46e-02 3.31e-01 6.16e+03 1.00e-01 2.38e-03
------------------------------------------------------------------
status: solved
timings: total: 2.39e-03s = setup: 8.76e-04s + solve: 1.51e-03s
lin-sys: 9.76e-04s, cones: 1.10e-04s, accel: 1.00e-04s
------------------------------------------------------------------
objective = 6155.077695
------------------------------------------------------------------The only remaining difference with your output is the use of the linear cone for one-sided inequality constraints. |
|
Ok great, that probably explains most of the discrepancy between your results and mine, hopefully they align more after this! Other solvers may well have been affected by this issue too so probably worth re-running any of them that were. What to put in the box cone vs put in the linear constraint is a actually kind of an interesting open question. Overall, if the bounds are likely to be very loose, then I think that the box cone is a better choice for exactly the reason you ran into. SCS accepts infinite entries in the box cone, since it's just a projection that does nothing, so feel free to try using that if you want. Though it looks as though the performance of your run and my run are essentially identical despite using different box cones, this might not be true for other problems. |
|
Updated results on the Maros-Meszaros test set after the fix (diff, report). SCS's success rate (returns "optimal" and passes tolerance checks) is now 73% with default settings. Overall our results are much more aligned 👍 There is still some difference: in the CSV file you shared, 130 / 138 problems (94 %) have the "optimal" return status, while on my machine it's about 121 / 138 problems (88 %). The difference is on exactly 9 problems: CONT-300, HUES-MOD, HUESTIS, QETAMACR, QFFFFF80, QGFRDXPN, QPILOTNO, QSHELL and YAO. Those problems have the "optimal" status in your results, but don't solve on my machine. (Other than this, the problems that are unsolved both in your results and in mine are: BOYD2, OWELL20, QGROW15, QGROW22, QGROW7, STADAT1, STADAT2 and STADAT3.) I'm not too surprised that we see some difference though, as we don't have exactly the same setup. For instance it could be due to:
I will re-run the benchmark with the other solvers as well 👌 |
|
Yes it looks like some of those are either over the 1000 second timeout, or have actually failed to solve when I re-run them under the latest version. Some of the others should be fine though, some examples below. Possibly there is still a discrepancy in the conversion. |
|
Intriguing! I will check those three. |
stephane-caron
changed the title
Moved to its own issue: #41 |
stephane-caron
changed the title
From bodono/scs-python#63 (comment):
The text was updated successfully, but these errors were encountered: