Develop a common proof format and export proofs #458

gcarpio21 · 2025-03-19T12:29:15Z

Objective

This PR aims to expose proofs produced by supported solvers through the API, primarily via a getProof() method.

Key Changes

Introduced ProofNode, ProofDag, and ProofRule as the main components for handling proofs.
Implemented proof retrieval for the Z3 solver.
Explored a standardized format for proofs across different solvers.

Proof Format Conversion

To unify proof representation, this PR investigates converting solver-specific proofs into a common format, with RESOLUTE (used by SMTInterpol) as a candidate.

The proposed approach models proof rules in CNF and transforms them into resolution steps.
The Z3ToResoluteProofConverter provides an example of this transformation.

Task List

Next Steps

More flexibility while handling proofs could be provided through the ProofDag interface.

…getProof() in BasicProverEnvironment.java

…aviour of getProof to return unsupported operation.

…p_a_common_proof_format_and_export_proofs

…ts from SMTInterpol to a ResolutionProofDAG. Not working. Class SmtInterpolProofNode should help transform from the internal SMTInterpol proof representation to ResolutionProofDAG. Not working. Added some tests to evaluate proofs given back by SMTInterpol and the ResolutionProofDAG objects that should be returned.

Option requireProofs must be set to true through Configuration. Added a class for testing the proof generation of Z3. SMTInterpol: some documentation.

…roofs

Various changes to the ProofDAG interface and all classes inheriting it.

…roofs

…o multiple resolution steps.

…esolution steps.

…n now be converted to resolution proof rules.

- Renamed some classes for clarity - Z3: Made proof traversal non-recursive for efficiency.Added tests for proof-related functionality - CVC5: Implemented necessary classes for proof production - Improved documentation - Added test for retrieving proofs in ProverEnvironmentTest.java

…entation for when no formula is available.

… from Mathsat5

…object.

src/org/sosy_lab/java_smt/solvers/z3/Z3NonRecursiveProofProcessor.java

baierd

Some general things that i noticed (some may be mentioned in change requests, but apply generally):

Tests should be generalized and moved into the test package if they are not using native API. You can still have tests for a single solver in the test package by using the assume API and only allowing a single one. The name of the solver should be mentioned in the test name then.
Please add documentation to the new classes and interfaces. Especially mention which ones are used internally and which ones are public/usable by users and the purpose/background of the classes/interfaces.
You can have classes within classes, which is especially helpful to structure smaller components, e.g. nodes of a graph, within the context you are using them in. That makes the code much easier to read and structures it better.
Please avoid pushing print-outs or commented out code. If you are working on a feature thats not yet working, thats fine. Document it and don't use the component. Please add some documentation to the commented out code. Print-outs should never be pushed. Try using the logger or exceptions. In tests please use assertion instead of printouts. (It is OK to expect the current static output of a solver. If the test fails in the future, we devs are notified about changes that we might need to consider.)
Native API tests that just test that proofs work are good and can be added for all solvers supporting proofs.

baierd · 2025-03-21T17:20:26Z

src/org/sosy_lab/java_smt/ResProofRule.java

+ *
+ * @author Gabriel Carpio
+ */
+public class ResProofRule {


You could add sources (publication(s), reference sites etc.), the solvers that support this natively, as well as the usage/logic.

I added some documentation and will wait for further information on the format form Mathsat5 as it seems to be very similar.

As discussed, we will add this as far as possible and wait for more details from the MathSAT5 devs.

baierd · 2025-03-21T17:37:31Z

src/org/sosy_lab/java_smt/ResolutionProofDag.java

+public class ResolutionProofDag extends AbstractProofDag {
+
+  /*
+  public ResolutionProofDAG() {


Is this class being worked on? If yes, in what form?

Please try to not push commented out code if possible. Also, please document this if it is supposed to be used in the future.

Made some adjustments to this.

baierd · 2025-03-21T17:40:31Z

src/org/sosy_lab/java_smt/api/BasicProverEnvironment.java

+    try {
+      throw new UnsupportedOperationException("Proof generation isn't enabled.");
+    } catch (UnsupportedOperationException e) {
+      System.out.println(e.getMessage());


You can check if the prover is closed first, making sure that the prover is available.

It does not make much sense to throw an exception and catch it immediately again ;D
And please don't push print-outs. Try using the logger, exceptions if you want to notify the user.

Also, "Proof generation is not available for the chosen solver" is a better message for the exception type.

To check the state I will move this method to AbstractProver. There, there also a method checkGenerateProofs that mimics the behavior of the other check methods, I understand this should throw an Exception already. Aside from this, there is still the issue that, for example, for Z3 proof generation must be set as an Option in the configuration, so the solver should retrieve this information from the context in this case.

Since you have to start proof generation in the proofer anyways, why not just getProof() that then does the checks?

src/org/sosy_lab/java_smt/api/proofs/ProofDag.java

baierd · 2025-03-21T17:47:31Z

src/org/sosy_lab/java_smt/api/proofs/ProofFactory.java

+import org.sosy_lab.java_smt.ResProofRule.ResAxiom;
+
+public class ProofFactory {
+  public static ProofNode createSourceNode(ResAxiom rule, Formula formula) {


Building a proof should not be exposed to the user. Is ProofFactory available for users?

I'm currently trying to utilize this class to eliminate classes like Mathsat5ProofProcessor and Z3ToResoluteProofConverter. However, I'm facing an issue: ideally, the user shouldn't have direct access to these methods and should instead retrieve proofs via getProof from a ProverEnvironment instance. The challenge is that making the class non-public complicates its use within the solvers' packages.

In Java the following works: you expose public API in an interface. This is then implemented by a abstract class that has a second method for each public method that is not public (e.g. public ... getChildren() and protected ... getChildren0()) that is always called by the final default implementation of the public method. The non-public method throws a exception per default and is overridden by the solvers. Other methods needed to manipulate or build the proof are also part of the abstract class, but never public.

baierd · 2025-03-21T18:26:19Z

src/org/sosy_lab/java_smt/solvers/smtinterpol/SmtInterpolProofsTest.java

+
+
+      //String proofStr = proof.toString();
+      System.out.println(invokeGetProofMode(smtinterpol).toString());


Please use assertions instead of printouts.

baierd · 2025-03-21T18:26:39Z

src/org/sosy_lab/java_smt/solvers/z3/Z3ProofsTest.java

+    ShutdownManager shutdown = ShutdownManager.create();
+
+    // Create new context with SMTInterpol
+    context = Z3SolverContext.create(


Why not use the SolverContextFactory?

baierd · 2025-03-21T18:28:00Z

src/org/sosy_lab/java_smt/solvers/cvc5/CVC5ProofsTest.java

+  }
+
+  private static Solver createEnvironment() throws CVC5ApiException {
+    Solver newSolver = new Solver();


This tests nothing currently. Also, native proof tests are good, but should be part of the CVC5 native test-class.

baierd · 2025-03-21T18:28:50Z

src/org/sosy_lab/java_smt/solvers/mathsat5/Mathsat5NativeApiTest.java

@@ -79,6 +85,7 @@ public void createEnvironment() {
    long cfg = msat_create_config();

    msat_set_option_checked(cfg, "model_generation", "true");
+    msat_set_option_checked(cfg, "proof_generation", "true");


Why enable proofs for all tests and then build a new, dedicated solver environment for the proof test?

src/org/sosy_lab/java_smt/solvers/z3/Z3ToResoluteProofConverter.java

baierd · 2025-03-21T18:40:57Z

@leventeBajczi could you spare a look at this PR and maybe give us some feedback on the current proof interface?

…r by the name of the rule

…bproof>''. The signatures of 'addEdge' and 'addChild' changed accordingly

…gh 'checkGenerateProofs()', whether the solver is closed and whether the last evaluation was UNSAT

…sserting true and false, and when previously having evaluated once for multiple assertions and retrieving a proof, then asserting different formulas, evaluating and requesting a proof again.

…ly in the last changes to it.

…ed through 'checkGenerateProofs'

…ational and UF proof, test proof for only asserting false/true, test getting 2 different proofs with same prover instance.

…roofs

…and transformed into JavaSMT proofs. There is no apparent way to retrieve a formula just from a String.

…ronmentTest`

…cked warning remains in `OpenSMTProof` but is suppressed

…le_proofs.c`.

…eck that some proof is returned.

…sat5 proofs

… can't produce BV proofs with it enabled

…ception is thrown when requestion a proof for asserting `false`. Took MathSAT5 out of `proofOfFalseTest` test.

…d_export_proofs' into develop_a_common_proof_format_and_export_proofs

…ilable and throw a corresponding Exception.

gcarpio21 added 16 commits February 3, 2025 19:26

Added multiple classes for supporting proof generation. Added method …

9f449e7

…getProof() in BasicProverEnvironment.java

Added GENERATE_PROOFS to ProverOptions

b2bc9eb

Added implementation of getProof for SMTInterpol. Changed default beh…

766e665

…aviour of getProof to return unsupported operation.

Merge remote-tracking branch 'refs/remotes/origin/master' into develo…

9c07f08

…p_a_common_proof_format_and_export_proofs

Z3: context now able to be created with proof generation.

955c1af

Option requireProofs must be set to true through Configuration. Added a class for testing the proof generation of Z3. SMTInterpol: some documentation.

Merge branch 'master' into develop_a_common_proof_format_and_export_p…

375937d

…roofs

Z3: Proof ast given by Z3 is now parsed to DAG made out of Z3ProofNode.

af4f670

Various changes to the ProofDAG interface and all classes inheriting it.

Z3: Corrections to the parsing of the proof ast.

c2139ad

Various changes to the ProofDAG interface and all classes inheriting it.

Mathsat5: The Native API now includes the methods for proof processing

e05451c

Merge branch 'master' into develop_a_common_proof_format_and_export_p…

e69d1e0

…roofs

Z3: The transitivity proof step from Z3 is now able to be converted t…

a6a3efd

…o multiple resolution steps.

Z3: Small correction when converting the transitivity proof rule to r…

2015b46

…esolution steps.

Z3: True, Asserted, Goal, MP, Reflexivity and Symmetry proof rules ca…

a4bbe62

…n now be converted to resolution proof rules.

Renamed some classes

e7940f1

gcarpio21 requested a review from baierd March 19, 2025 12:29

gcarpio21 self-assigned this Mar 20, 2025

gcarpio21 added 7 commits March 20, 2025 18:22

Renaming some classes for clarity

f988d6a

Method getFormula() in the ProofRule interface has default implem…

fcdf44f

…entation for when no formula is available.

Added Mathsat5ProofRule class

1d9ab98

Added Mathsat5ProofProcessor class for processing the proofs straight…

2fec458

… from Mathsat5

Added Mathsat5ProofNode class for storing the proofs in JavaSMT

7c20761

Mathsat5: proofs are enabled by setting ProverOptions.GENERATE_PROOFS

3d89810

Mathsat5: method getProof() now can be called to get a ProofNode …

d88ecb2

…object.

gcarpio21 commented Mar 20, 2025

View reviewed changes

src/org/sosy_lab/java_smt/solvers/z3/Z3NonRecursiveProofProcessor.java Outdated Show resolved Hide resolved

baierd requested changes Mar 21, 2025

View reviewed changes

gcarpio21 added 2 commits March 21, 2025 19:43

Checkstyle

e87f469

Class ProofRuleRegistry now allows to get a proof rule from any solve…

018d857

…r by the name of the rule

gcarpio21 added 20 commits May 15, 2025 18:11

Made necessary changes to have proof production always enabled in Z3.

6bbb92d

Renamed 'ProofNode' into 'Subproof'

81780c6

Proofs: Proof edges is now a map from 'Subproof' -> 'LinkedHashSet<Su…

e751e16

…bproof>''. The signatures of 'addEdge' and 'addChild' changed accordingly

SmtInterpol: Added extra proof rules.

640c6f7

'ResProofRule': check for null removed. It should be checked elsewhere.

303713b

CVC5: 'getProof' now checks whether proof generation is enabled throu…

84f1a16

…gh 'checkGenerateProofs()', whether the solver is closed and whether the last evaluation was UNSAT

Proofs: Test cases for solvers' outputs for retrieving a proof when a…

86c01cd

…sserting true and false, and when previously having evaluated once for multiple assertions and retrieving a proof, then asserting different formulas, evaluating and requesting a proof again.

Changed incorrect cast in 'AbstractProof'

24991fe

'AbstractProof' method 'proofAsString' did not print formulas correct…

331f7ac

…ly in the last changes to it.

'AbstractProof' method 'proofAsString' did not print formulas correct…

48534bd

…ly in the last changes to it.

'TOTAL_INT' rule in 'ResProofRule' had wrong name previously.

dd460aa

MathSAT5: method 'getProof' now checks that proof generation is enabl…

a687265

…ed through 'checkGenerateProofs'

Added multiple tests for proofs: test boolean proof, integer proof, r…

0bbaa73

…ational and UF proof, test proof for only asserting false/true, test getting 2 different proofs with same prover instance.

Checkstyle

0739661

Merge branch 'master' into develop_a_common_proof_format_and_export_p…

16c6c9b

…roofs

Some (not working) code added so that proofs from OpenSMT are parsed …

bcad354

…and transformed into JavaSMT proofs. There is no apparent way to retrieve a formula just from a String.

No longer logging whole proof in 'LoggingBasicProverEnvironment'

e86e7e2

Added bit vector and array tests for retrieving proofs in `ProverEnvi…

7aa247c

…ronmentTest`

Multiple changes regarding the proof interface to fix warnings. Unche…

cec06cf

…cked warning remains in `OpenSMTProof` but is suppressed

MathSAT5: added test in Mathsat5NativeApiTest taken from `api_examp…

c8009f0

…le_proofs.c`.

gcarpio21 mentioned this pull request May 26, 2025

Formula can not be retrieved from a String using OpenSMT #487

Open

BaierD and others added 9 commits May 26, 2025 16:46

Don't check primitive for null in Mathsat5 native tests and simply ch…

25f67f7

…eck that some proof is returned.

Add require methods for theories used in proof tests

4200112

Add an assertion as a hint for a necessary check when traversing Math…

b6a786c

…sat5 proofs

Disable eager BV theory option for proof generation in Mathsat, as it…

27e4bb3

… can't produce BV proofs with it enabled

Add more require methods for theories used in proof tests

14e4991

MathSAT5: added test in Mathsat5NativeApiTest to confirm that an ex…

08f8038

…ception is thrown when requestion a proof for asserting `false`. Took MathSAT5 out of `proofOfFalseTest` test.

Merge remote-tracking branch 'origin/develop_a_common_proof_format_an…

d056b4e

…d_export_proofs' into develop_a_common_proof_format_and_export_proofs

MathSAT5: now correctly handles the case when no proof manager is ava…

6171224

…ilable and throw a corresponding Exception.

Checkstyle

78f891c



		//String proofStr = proof.toString();
		System.out.println(invokeGetProofMode(smtinterpol).toString());

Develop a common proof format and export proofs #458

Are you sure you want to change the base?

Develop a common proof format and export proofs #458

Uh oh!

Conversation

gcarpio21 commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Objective

Key Changes

Proof Format Conversion

Task List

Next Steps

Uh oh!

Uh oh!

baierd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gcarpio21 Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

baierd commented Mar 21, 2025

Uh oh!

Uh oh!

gcarpio21 commented Mar 19, 2025 •

edited

Loading

gcarpio21 Mar 25, 2025 •

edited

Loading