Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid non-deterministic QPUnitTests failure by setting polish to true in QPUnitTests #172

Merged
merged 1 commit into from
Nov 6, 2024
Merged

Avoid non-deterministic QPUnitTests failure by setting polish to true in QPUnitTests #172

merged 1 commit into from
Nov 6, 2024

Conversation

traversaro
Copy link
Member

@traversaro traversaro commented Nov 5, 2024 •
edited
Loading

This was a bit hidden by #161, but I recently noticed that while most time the test QPUnitTests the output was:

traversaro@IITBMP014LW012:~/robotology-superbuild/buildapt/src/OsqpEigen$ ctest -VV -R QPUnitTest
UpdateCTestConfiguration  from :/home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen/DartConfiguration.tcl
Parse Config file:/home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen/DartConfiguration.tcl
Parse Config file:/home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen/DartConfiguration.tcl
Test project /home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 2
    Start 2: QPUnitTests

2: Test command: /home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen/bin/QPUnitTests
2: Working Directory: /home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen/tests
2: Test timeout computed to be: 1500
2: Randomness seeded to: 3883189244
2: -----------------------------------------------------------------
2:            OSQP v0.6.3  -  Operator Splitting QP Solver
2:               (c) Bartolomeo Stellato,  Goran Banjac
2:         University of Oxford  -  Stanford University 2021
2: -----------------------------------------------------------------
2: problem:  variables n = 2, constraints m = 0
2:           nnz(P) + nnz(A) = 3
2: settings: linear system solver = qdldl,
2:           eps_abs = 1.0e-03, eps_rel = 1.0e-03,
2:           eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
2:           rho = 1.00e-01 (adaptive),
2:           sigma = 1.00e-06, alpha = 1.00, max_iter = 4000
2:           check_termination: on (interval 25),
2:           scaling: on, scaled_termination: off
2:           warm start: on, polish: off, time_limit: off
2:
2: iter   objective    pri res    dua res    rho        time
2:    1  -1.6875e+00   0.00e+00   3.75e-06   1.00e-01   3.98e-05s
2:   25  -1.6875e+00   0.00e+00   3.33e-16   1.00e-01   4.64e-05s
2:
2: status:               solved
2: number of iterations: 25
2: optimal objective:    -1.6875
2: run time:             4.95e-05s
2: optimal rho estimate: 1.00e-06
2:
2: [OsqpEigen::Data::setHessianMatrix] Please set the number of variables before add the hessian matrix.
2: -----------------------------------------------------------------
2:            OSQP v0.6.3  -  Operator Splitting QP Solver
2:               (c) Bartolomeo Stellato,  Goran Banjac
2:         University of Oxford  -  Stanford University 2021
2: -----------------------------------------------------------------
2: problem:  variables n = 2, constraints m = 3
2:           nnz(P) + nnz(A) = 7
2: settings: linear system solver = qdldl,
2:           eps_abs = 1.0e-03, eps_rel = 1.0e-03,
2:           eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
2:           rho = 1.00e-01 (adaptive),
2:           sigma = 1.00e-06, alpha = 1.00, max_iter = 4000
2:           check_termination: on (interval 25),
2:           scaling: on, scaled_termination: off
2:           warm start: on, polish: off, time_limit: off
2:
2: iter   objective    pri res    dua res    rho        time
2:    1  -4.9384e-03   1.00e+00   2.00e+02   1.00e-01   1.37e-05s
2:   50   1.8800e+00   1.91e-07   7.50e-07   1.38e+00   2.69e-05s
2:
2: status:               solved
2: number of iterations: 50
2: optimal objective:    1.8800
2: run time:             2.93e-05s
2: optimal rho estimate: 1.36e+00
2:
2: expectedSolution: 0.3
2: 0.7
2: solver.getSolution(): 0.3
2: 0.7
2: tolerance: 0.0001
2: expectedSolution-solver.getSolution(): -1.91114e-07
2:  1.90972e-07
2: ===============================================================================
2: All tests passed (14 assertions in 2 test cases)
2:
1/1 Test #2: QPUnitTests ......................   Passed    0.00 sec

in some rare case, the test was failing as the solution found was much less accurate:

    Test #2: QPUnitTests ......................***Failed    0.00 sec
Randomness seeded to: 3198543723
-----------------------------------------------------------------
           OSQP v0.6.3  -  Operator Splitting QP Solver
              (c) Bartolomeo Stellato,  Goran Banjac
        University of Oxford  -  Stanford University 2021
-----------------------------------------------------------------
problem:  variables n = 2, constraints m = 0
          nnz(P) + nnz(A) = 3
settings: linear system solver = qdldl,
          eps_abs = 1.0e-03, eps_rel = 1.0e-03,
          eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
          rho = 1.00e-01 (adaptive),
          sigma = 1.00e-06, alpha = 1.00, max_iter = 4000
          check_termination: on (interval 25),
          scaling: on, scaled_termination: off
          warm start: on, polish: off, time_limit: off

iter   objective    pri res    dua res    rho        time
   1  -1.6875e+00   0.00e+00   3.75e-06   1.00e-01   3.82e-05s
  25  -1.6875e+00   0.00e+00   3.33e-16   1.00e-01   4.52e-05s

status:               solved
number of iterations: 25
optimal objective:    -1.6875
run time:             4.83e-05s
optimal rho estimate: 1.00e-06

[OsqpEigen::Data::setHessianMatrix] Please set the number of variables before add the hessian matrix.
-----------------------------------------------------------------
           OSQP v0.6.3  -  Operator Splitting QP Solver
              (c) Bartolomeo Stellato,  Goran Banjac
        University of Oxford  -  Stanford University 2021
-----------------------------------------------------------------
problem:  variables n = 2, constraints m = 3
          nnz(P) + nnz(A) = 7
settings: linear system solver = qdldl,
          eps_abs = 1.0e-03, eps_rel = 1.0e-03,
          eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
          rho = 1.00e-01 (adaptive),
          sigma = 1.00e-06, alpha = 1.00, max_iter = 4000
          check_termination: on (interval 25),
          scaling: on, scaled_termination: off
          warm start: on, polish: off, time_limit: off

iter   objective    pri res    dua res    rho        time
   1  -4.9384e-03   1.00e+00   2.00e+02   1.00e-01   1.02e-04s
  50   1.8798e+00   1.23e-03   2.46e-05   1.00e-01   1.16e-04s

status:               solved
number of iterations: 50
optimal objective:    1.8798
run time:             1.19e-04s
optimal rho estimate: 1.38e+00

expectedSolution: 0.3
0.7
solver.getSolution(): 0.298771
0.701228
tolerance: 0.0001
expectedSolution-solver.getSolution(): 0.00122892
 -0.001228

QPUnitTests is a Catch2 v3.4.0 host application.
Run with -? for options

-------------------------------------------------------------------------------
QPProblem
-------------------------------------------------------------------------------
/home/traversaro/robotology-superbuild/src/OsqpEigen/tests/QPTest.cpp:46
...............................................................................

/home/traversaro/robotology-superbuild/src/OsqpEigen/tests/QPTest.cpp:99: FAILED:
  REQUIRE( solver.getSolution().isApprox(expectedSolution, tolerance) )
with expansion:
  false

===============================================================================
test cases:  2 |  1 passed | 1 failed
assertions: 14 | 13 passed | 1 failed



0% tests passed, 1 tests failed out of 1

Total Test time (real) =   2.51 sec

The following tests FAILED:
          2 - QPUnitTests (Failed)

To remove the non-determinism, I also tried to set solver.settings()->setAdaptiveRho(false); to remove the non-determinism (see https://github.com/google/osqp-cpp?tab=readme-ov-file#faq), but this only made the failure deterministic, with output:

traversaro@IITBMP014LW012:~/robotology-superbuild/buildapt/src/OsqpEigen$ ctest --output-on-failure -R QPUnitTests --repeat until-fail:100000
Test project /home/traversaro/robotology-superbuild/buildapt/src/OsqpEigen
    Start 2: QPUnitTests
    Test #2: QPUnitTests ......................***Failed    0.00 sec
Randomness seeded to: 3354483813
-----------------------------------------------------------------
           OSQP v0.6.3  -  Operator Splitting QP Solver
              (c) Bartolomeo Stellato,  Goran Banjac
        University of Oxford  -  Stanford University 2021
-----------------------------------------------------------------
problem:  variables n = 2, constraints m = 0
          nnz(P) + nnz(A) = 3
settings: linear system solver = qdldl,
          eps_abs = 1.0e-03, eps_rel = 1.0e-03,
          eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
          rho = 1.00e-01 (adaptive),
          sigma = 1.00e-06, alpha = 1.00, max_iter = 4000
          check_termination: on (interval 25),
          scaling: on, scaled_termination: off
          warm start: on, polish: off, time_limit: off

iter   objective    pri res    dua res    rho        time
   1  -1.6875e+00   0.00e+00   3.75e-06   1.00e-01   6.05e-05s
  25  -1.6875e+00   0.00e+00   3.33e-16   1.00e-01   6.79e-05s

status:               solved
number of iterations: 25
optimal objective:    -1.6875
run time:             7.12e-05s
optimal rho estimate: 1.00e-06

[OsqpEigen::Data::setHessianMatrix] Please set the number of variables before add the hessian matrix.
-----------------------------------------------------------------
           OSQP v0.6.3  -  Operator Splitting QP Solver
              (c) Bartolomeo Stellato,  Goran Banjac
        University of Oxford  -  Stanford University 2021
-----------------------------------------------------------------
problem:  variables n = 2, constraints m = 3
          nnz(P) + nnz(A) = 7
settings: linear system solver = qdldl,
          eps_abs = 1.0e-03, eps_rel = 1.0e-03,
          eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
          rho = 1.00e-01 ,
          sigma = 1.00e-06, alpha = 1.00, max_iter = 4000
          check_termination: on (interval 25),
          scaling: on, scaled_termination: off
          warm start: on, polish: off, time_limit: off

iter   objective    pri res    dua res    rho        time
   1  -4.9384e-03   1.00e+00   2.00e+02   1.00e-01   1.52e-05s
  50   1.8798e+00   1.23e-03   2.46e-05   1.00e-01   2.71e-05s

status:               solved
number of iterations: 50
optimal objective:    1.8798
run time:             2.96e-05s
optimal rho estimate: 1.38e+00

expectedSolution: 0.3
0.7
solver.getSolution(): 0.298771
0.701228
tolerance: 0.0001
expectedSolution-solver.getSolution(): 0.00122892
 -0.001228


QPUnitTests is a Catch2 v3.4.0 host application.
Run with -? for options

-------------------------------------------------------------------------------
QPProblem
-------------------------------------------------------------------------------
/home/traversaro/robotology-superbuild/src/OsqpEigen/tests/QPTest.cpp:46
...............................................................................

/home/traversaro/robotology-superbuild/src/OsqpEigen/tests/QPTest.cpp:99: FAILED:
  REQUIRE( solver.getSolution().isApprox(expectedSolution, tolerance) )
with expansion:
  false

===============================================================================
test cases:  2 |  1 passed | 1 failed
assertions: 14 | 13 passed | 1 failed



0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.00 sec

The following tests FAILED:
          2 - QPUnitTests (Failed)
Errors while running CTest

After playing a bit with the tolerances, I noticed that setting the polish option to true ensure that the solution is always provided with high accuracy, so in this PR I enabled the option to avoid the risk of non-deterministic failure of QPUnitTests.

@traversaro
Copy link
Member Author

Please ignore the failing tests that are due to #161 and will be fixed in #171 .

@GiulioRomualdi GiulioRomualdi merged commit e535f6f into master Nov 6, 2024
22 of 25 checks passed
@GiulioRomualdi
Copy link
Member

Thank you!!! You fixed a really annoying bug in the test!

@traversaro
Copy link
Member Author

Please ignore the failing tests that are due to #161 and will be fixed in #171 .

That really scared me when working on https://github.com/ami-iit/qpsolvers-eigen, as for some reason there the issue was much more frequent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@GiulioRomualdi GiulioRomualdi Awaiting requested review from GiulioRomualdi

@S-Dafarra S-Dafarra Awaiting requested review from S-Dafarra

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@traversaro @GiulioRomualdi