Kex Integration (#431)

* minimal attempt to run kex from testspark with harcoded settings

Needs debugging and testing
* load kex-runner jar from github (build.gradle.kts toplevel)
* setup code for kex properties
* KexErrorManager based on LLMErrorManager
* KexProcessManager based on EvoSuiteProcessManager
* Basic UI element (button for running kex)
* kex works only for the class codeType (todo funciton and line if possible)
* read resource files kex.policy and modules.info

* generate Report and series of simplifications for MVP

use a provided kex path for now instead of downloading jar and adding dependency
use ProcessBuilder (jdk) instead of OSProcessHandler (IJ sdk)
use kex.py instead of building java command directly
generate Report objects by reading generated java classes

* download kex as github release if it doesn't exist

* delete commented code

Deleted stuff:
running Kex through through OSProcessHandler (IJ sdk)
running Kex with kex.py
downloading kex from github in build.gradle.kts

* use javaparser to merge @before and @after generated methods

* compiling tests successfully by adding helper method to TestGenData.otherInfo

* fold anything but @test annotated code, but unfortunately only on UI updates

* remove python dependency by running kex jar directly

* use  project's java and add version check

* empty implementation of settings classes for kex

* allow setting kex arguments from kex settings page

* kex test generation for 'method' code type

arguably overkill solution of modify PsiMethodWrapper class
added parameter names, types and return tyepes explicitly
updated implementation for java and kotlin

* fold all but @test methods regardless of declaration order

* remove settings for unsupported kex features

* folding works even if junit isn't in classpath

* use the the subprocess manager from IJ framework

* empty kex path download to .cache and LOCALAPPDATA in windows

* fix lint. remove wildcard imports

* from IJ api, provide correct build directory to kex

For multimodule projects using TestSpark the correct module's
build directory path is passed based on the location of code for which
tests are generated

* set maxTests displayed and minimizeTestSuite options

* fix code fold to only happen once at the start not every ui update

* error handling for errors in kex process manager

also fixed a bug with the way options were passed to kex subprocess
made them a list of strings instead of a single big space separated string

* fix lint

* provide signatures with FQNs to kex for java

This also includes a simple string based mapping to jvm types through type erasure

* refactor. extract functons, add comments, remove redundancies

* fix lint

* every option in kex cmd is preceeded by --option

* use kotlin.time.Duration instead of Int

* refactor error handling code, check for non-zero exit code

* make generated code manipulation more robust

It no longer depends on the order of the methods generated

* only check if the correct verison of kex exists

* make download url a property and version a user setting

* support jdk version 8. no --add-opens

* bump up kex version to 0.0.10

* allow kex test generation for classes not in a package

* null safety while folding helper code

* Merge branch 'development' into edwin1729/improvement/kex-integration

* disallow kex and line code type being chosen simultaneously

* update description tab and README.md with kex

* fix lint

* fix run coverage by making method name same as class name

This is expected by TestSpark. Relevant file TestProcess.kt:createXmlFromJacoco:109

* undo mistaken removal of addLanguageTextFieldListener before code folding

* Add empty LLM response handling (#342)

Show a warning if LLM returns a response that cannot be parsed into a test case (e.g., an explanation of this test case rather than a modification).

* Restrict TestSpark action to suitable code types and fix generation for a line (#344)

* fix update function

* create availableForGeneration

* ktlint

* feat: add javadoc for `JavaPsiHelper.availableForGeneration`

* feat: check for nullness of a PSI file in `TestSparkAction.update`

* feat: update javadocs in `PsiComponents.kt`

* feat: check for a class or method/func in `KotlinPsiHelper.availableForGeneration`

* feat: add TODO to `ToolUtils` about a potential bug

The bug is reflected in the issue #375.

* feat: make `PsiHelper.getSurroundingLineNumber` return 1-based line numbers

Before, the `KotlinPsiHelper` returned a 0-based line number which caused an issue with line-based test generation.
The generated prompt contained a line above the selected one.

* feat: implement line-based test generation with CUT as a context

When there is no surrounding method about the selected line,
we use the CUT as a context for this line. The CUT must always be present.
Otherwise, the generation action should have been disabled for this line.

* refactor: apply ktlint

* feat: add `See` in TODO

* feat: add TODO and surround $NAME in backticks in `linePrompt` template

* feat: collect class constructor signatures in `PsiClassWrapper`

* feat: remove backticks from `linePrompt`

* feat: fill line-based test generation with additional context

The line-based test generation that has a method as a context
of the line now also accepts constructors of the containing class.

* refactor: use `firstOrNull` for `cut` extraction

* refactor: apply ktlint

* fix: add required parameter to `ClassRepresentation` in tests

* publish: core module version `4.0.0`

The major version increased due to the change of the public API of `PromptGenerator.generatePromptForLine` method.

---------

Co-authored-by: Vladislav Artiukhov <[email protected]>

* Data class for execution results

* Minor refactoring

* apply ktlint

* fix getJavaVersion

* add KexSettingsService to plugin.xml

* fix DefaultKexSettingsState

* fix KexSettingsState

* fix Bundles

* Force junit4 for EvoSuite tests

* JUnit version forcing

* fix getExceptionData

* fix tools

* fix managers

* fix TestCasePanelBuilder.kt

* fix TestProcessor.kt

* remove unused import

* fix ktlint

* fix ktlint

* remove TestSparkStarter.kt

* Add support for language applicability checks in Tool

* Add support for language applicability checks in Llm

* Add support for language applicability checks in Kex

* Add support for language applicability checks in EvoSuite

* Filter test generation buttons by language compatibility

* Refactor imports for SupportedLanguage in EvoSuite and Llm

* Set progress indicator text during Kex test generation

---------

Co-authored-by: Edwin Fernando <[email protected]>
Co-authored-by: Edwin Fernando <[email protected]>
Co-authored-by: Edwin Fernando <[email protected]>
Co-authored-by: Vladislav Artiukhov <[email protected]>
Co-authored-by: Iurii Zaitsev <[email protected]>
Co-authored-by: Hello-zoka <[email protected]>

README.md

@@ -42,6 +42,15 @@ TestSpark currently supports two test generation strategies:
 <li>Generate tests for Java classes, methods, and single lines.</li>
 </ul>
 
+<h4>Symbolic execution-based test generation</h4>
+<p>For this type of test generation, TestSpark uses <a href="https://github.com/vorpal-research/kex">Kex</a>, supporting symbolic execution for Java Byte Code. </p>
+<ul>
+<li>Supports up to Java 8 and upwards.</li>
+<li>Powered by SMT solvers, it supports really high coverages given larger time frames.</li>
+<li>Generated test cases are however not very readable (there are plans to automatically refactor with the help of LLMs).</li>
+<li>Generates tests for Java classes and methods.</li>
+</ul>
+
 <p>Initially implemented by <a href="https://www.ciselab.nl">CISELab</a> at <a href="https://se.ewi.tudelft.nl">SERG @ TU Delft</a>, TestSpark is currently developed and maintained by <a href="https://lp.jetbrains.com/research/ictl/">ICTL at JetBrains Research</a>.</p>
 
 ## <span style="color:crimson; font-size:150%; font-weight:bold"> DISCLAIMER </span>

build.gradle.kts

@@ -170,6 +170,9 @@ dependencies {
     // https://gitlab.com/mvysny/konsume-xml
     implementation("com.gitlab.mvysny.konsume-xml:konsume-xml:1.0")
 
+    // for merging generated kex tests into a single file
+    implementation("com.github.javaparser:javaparser-core:3.26.1")
+
     // From the jetbrains repository
     testImplementation("com.intellij.remoterobot:remote-robot:0.11.13")
     testImplementation("com.intellij.remoterobot:remote-fixtures:0.11.13")

core/src/main/kotlin/org/jetbrains/research/testspark/core/data/TestGenerationData.kt

@@ -16,6 +16,7 @@ data class TestGenerationData(
     var importsCode: MutableSet<String> = mutableSetOf(),
     var packageName: String = "",
     var runWith: String = "",
+    // Modifications to this code in the tool-window editor are forgotten when apply to test suite
     var otherInfo: String = "",
 
     // changing parameters with a large prompt

gradle.properties

@@ -7,6 +7,7 @@ pluginName = TestSpark
 pluginVersion = 0.3.1
 
 evosuiteVersion = 1.0.5
+kexVersion = 0.0.8
 
 # See https://plugins.jetbrains.com/docs/intellij/build-number-ranges.html
 # for insight into build numbers and IntelliJ Platform versions.

java/src/main/kotlin/org/jetbrains/research/testspark/java/JavaPsiMethodWrapper.kt

@@ -37,6 +37,15 @@ class JavaPsiMethodWrapper(private val psiMethod: PsiMethod) : PsiMethodWrapper
     override val signature: String
         get() = buildSignature(psiMethod)
 
+    override val parameterNames: List<String>
+        get() = psiMethod.parameterList.parameters.map { it.name }
+
+    override val parameterTypes: List<String>
+        get() = psiMethod.parameterList.parameters.map { it.type.canonicalText }
+
+    override val returnType: String
+        get() = psiMethod.returnType?.canonicalText ?: "void"
+
     val parameterList = psiMethod.parameterList
 
     val isConstructor: Boolean = psiMethod.isConstructor

kotlin/src/main/kotlin/org/jetbrains/research/testspark/kotlin/KotlinPsiMethodWrapper.kt

@@ -37,6 +37,15 @@ class KotlinPsiMethodWrapper(val psiFunction: KtFunction) : PsiMethodWrapper {
     override val signature: String
         get() = buildSignature(psiFunction)
 
+    override val parameterNames: List<String>
+        get() = psiFunction.valueParameters.map { it.name ?: "" }
+
+    override val parameterTypes: List<String>
+        get() = psiFunction.valueParameters.map { it.typeReference?.text ?: "Any" }
+
+    override val returnType: String
+        get() = psiFunction.typeReference?.text ?: "Unit"
+
     val parameterList = psiFunction.valueParameterList
 
     val isPrimaryConstructor: Boolean = psiFunction is KtPrimaryConstructor

langwrappers/src/main/kotlin/org/jetbrains/research/testspark/langwrappers/PsiComponents.kt

@@ -25,6 +25,9 @@ interface PsiMethodWrapper {
     val name: String
     val methodDescriptor: String
     val signature: String
+    val parameterNames: List<String>
+    val parameterTypes: List<String>
+    val returnType: String
     val text: String?
     val containingClass: PsiClassWrapper?
     val containingFile: PsiFile?

src/main/kotlin/org/jetbrains/research/testspark/actions/TestSparkAction.kt

@@ -28,21 +28,26 @@ import org.jetbrains.research.testspark.settings.evosuite.EvoSuiteSettingsState
 import org.jetbrains.research.testspark.settings.llm.LLMSettingsState
 import org.jetbrains.research.testspark.tools.TestsExecutionResultManager
 import org.jetbrains.research.testspark.tools.evosuite.EvoSuite
+import org.jetbrains.research.testspark.tools.kex.Kex
 import org.jetbrains.research.testspark.tools.llm.Llm
 import org.jetbrains.research.testspark.tools.template.Tool
 import java.awt.BorderLayout
 import java.awt.CardLayout
+import java.awt.Component
 import java.awt.Dimension
 import java.awt.Font
 import java.awt.Toolkit
 import java.awt.event.WindowAdapter
 import java.awt.event.WindowEvent
+import javax.swing.Box
+import javax.swing.BoxLayout
 import javax.swing.ButtonGroup
 import javax.swing.JButton
 import javax.swing.JFrame
 import javax.swing.JLabel
 import javax.swing.JPanel
 import javax.swing.JRadioButton
+import javax.swing.SwingConstants
 
 /**
  * Represents an action to be performed in the TestSpark plugin.
@@ -110,7 +115,9 @@ class TestSparkAction : AnAction() {
 
         private val llmButton = JRadioButton("<html><b>${Llm().name}</b></html>")
         private val evoSuiteButton = JRadioButton("<html><b>${EvoSuite().name}</b></html>")
+        private val kexButton = JRadioButton("<html><b>${Kex().name}</b></html>")
         private val testGeneratorButtonGroup = ButtonGroup()
+        private val kexForLineCodeTypeErrMsg = JLabel() // The error displayed when if kex and line code type are chosen
 
         private val psiHelper: PsiHelper
             get() {
@@ -201,13 +208,17 @@ class TestSparkAction : AnAction() {
             panelTitle.add(JLabel(TestSparkIcons.pluginIcon))
             panelTitle.add(textTitle)
 
-            testGeneratorButtonGroup.add(llmButton)
-            testGeneratorButtonGroup.add(evoSuiteButton)
+            if (Llm().appliedForLanguage(psiHelper.language)) testGeneratorButtonGroup.add(llmButton)
+            if (EvoSuite().appliedForLanguage(psiHelper.language)) testGeneratorButtonGroup.add(evoSuiteButton)
+            if (Kex().appliedForLanguage(psiHelper.language)) testGeneratorButtonGroup.add(kexButton)
 
             val testGeneratorPanel = JPanel()
             testGeneratorPanel.add(JLabel("Select the test generator:"))
-            testGeneratorPanel.add(llmButton)
-            testGeneratorPanel.add(evoSuiteButton)
+            for (button in testGeneratorButtonGroup.elements) testGeneratorPanel.add(button)
+            if (testGeneratorButtonGroup.elements.toList().size == 1) {
+                // A single button is selected by default
+                testGeneratorButtonGroup.elements.toList()[0].isSelected = true
+            }
 
             for ((codeType, codeTypeName) in codeTypes) {
                 val button = JRadioButton(codeTypeName)
@@ -236,8 +247,15 @@ class TestSparkAction : AnAction() {
                 .panel
 
             val nextButtonPanel = JPanel()
+            nextButtonPanel.layout = BoxLayout(nextButtonPanel, BoxLayout.Y_AXIS)
             nextButton.isEnabled = false
+            nextButton.alignmentX = Component.CENTER_ALIGNMENT
+            kexForLineCodeTypeErrMsg.alignmentX = Component.CENTER_ALIGNMENT
+            kexForLineCodeTypeErrMsg.horizontalAlignment = SwingConstants.CENTER
+            nextButtonPanel.add(kexForLineCodeTypeErrMsg)
+            nextButtonPanel.add(Box.createVerticalStrut(10)) // Add some space between label and button
             nextButtonPanel.add(nextButton)
+            updateNextButton()
 
             val cardPanel = JPanel(BorderLayout())
             cardPanel.add(panelTitle, BorderLayout.NORTH)
@@ -267,6 +285,10 @@ class TestSparkAction : AnAction() {
                 updateNextButton()
             }
 
+            kexButton.addActionListener {
+                updateNextButton()
+            }
+
             for ((_, button) in codeTypeButtons) {
                 button.addActionListener {
                     llmSetupPanelFactory.setPromptEditorType(button.text)
@@ -286,6 +308,8 @@ class TestSparkAction : AnAction() {
                     cardLayout.next(panel)
                     cardLayout.next(panel)
                     pack()
+                } else if (kexButton.isSelected) {
+                    startKexGeneration()
                 } else if (evoSuiteButton.isSelected && !evoSuiteSettingsState.evosuiteSetupCheckBoxSelected) {
                     startEvoSuiteGeneration()
                 } else {
@@ -387,6 +411,7 @@ class TestSparkAction : AnAction() {
             dispose()
         }
 
+        private fun startKexGeneration() = startUnitTestGenerationTool(tool = Kex())
         private fun startEvoSuiteGeneration() = startUnitTestGenerationTool(tool = EvoSuite())
         private fun startLLMGeneration() = startUnitTestGenerationTool(tool = Llm())
 
@@ -398,12 +423,23 @@ class TestSparkAction : AnAction() {
          * This method should be called whenever the mentioned above buttons are clicked.
          */
         private fun updateNextButton() {
-            val isTestGeneratorButtonGroupSelected = llmButton.isSelected || evoSuiteButton.isSelected
+            val isTestGeneratorButtonGroupSelected = llmButton.isSelected || evoSuiteButton.isSelected || kexButton.isSelected
             val isCodeTypeButtonGroupSelected = codeTypeButtons.any { it.second.isSelected }
-            nextButton.isEnabled = isTestGeneratorButtonGroupSelected && isCodeTypeButtonGroupSelected
+            val kexForCodeLineType =
+                kexButton.isSelected && codeTypeButtons.any { (codeType, button) -> codeType == CodeType.LINE && button.isSelected }
+            if (kexForCodeLineType) {
+                kexForLineCodeTypeErrMsg.text =
+                    "<html><b><font color='orange'>* Kex cannot generate tests for a single line. Please change your selection</font></b></html>"
+            } else {
+                kexForLineCodeTypeErrMsg.text = ""
+            }
+
+            nextButton.isEnabled =
+                isTestGeneratorButtonGroupSelected && isCodeTypeButtonGroupSelected && !kexForCodeLineType
 
             if ((llmButton.isSelected && !llmSettingsState.llmSetupCheckBoxSelected && !llmSettingsState.provideTestSamplesCheckBoxSelected) ||
-                (evoSuiteButton.isSelected && !evoSuiteSettingsState.evosuiteSetupCheckBoxSelected)
+                (evoSuiteButton.isSelected && !evoSuiteSettingsState.evosuiteSetupCheckBoxSelected) ||
+                kexButton.isSelected
             ) {
                 nextButton.text = PluginLabelsBundle.get("ok")
             } else {

src/main/kotlin/org/jetbrains/research/testspark/bundles/evosuite/EvoSuiteDefaultsBundle.kt

@@ -14,4 +14,8 @@ object EvoSuiteDefaultsBundle : DynamicBundle(EvoSuiteBundlePaths.defaults) {
      */
     @Nls
     fun get(@PropertyKey(resourceBundle = EvoSuiteBundlePaths.defaults) key: String): String = getMessage(key)
+        // In Intellij Platform version 2, the DynamicBundle returns the whole path and the value at the end in plugin verification.
+        // Each is separated by "|" (e.g., "|b|properties.llm.LLMDefaults|k|maxLLMRequest|3")
+        // if we do not split them here, the process will throw java.lang.NumberFormatException
+        .split("|").last()
 }

src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexBundlePaths.kt

@@ -0,0 +1,8 @@
+package org.jetbrains.research.testspark.bundles.kex
+
+object KexBundlePaths {
+    const val defaults: String = "properties.kex.KexDefaults"
+    const val messages: String = "properties.kex.KexMessages"
+    const val labels: String = "properties.kex.KexLabels"
+    const val settings: String = "properties.kex.KexSettings"
+}

src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexDefaultsBundle.kt

@@ -0,0 +1,21 @@
+package org.jetbrains.research.testspark.bundles.kex
+
+import com.intellij.DynamicBundle
+import org.jetbrains.annotations.Nls
+import org.jetbrains.annotations.PropertyKey
+
+/**
+ * Loads the `resources` directory.
+ */
+object KexDefaultsBundle : DynamicBundle(KexBundlePaths.defaults) {
+
+    /**
+     * Gets the requested default value.
+     */
+    @Nls
+    fun get(@PropertyKey(resourceBundle = KexBundlePaths.defaults) key: String): String = getMessage(key)
+        // In Intellij Platform version 2, the DynamicBundle returns the whole path and the value at the end in plugin verification.
+        // Each is separated by "|" (e.g., "|b|properties.llm.LLMDefaults|k|maxLLMRequest|3")
+        // if we do not split them here, the process will throw java.lang.NumberFormatException
+        .split("|").last()
+}

src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexLabelsBundle.kt

@@ -0,0 +1,17 @@
+package org.jetbrains.research.testspark.bundles.kex
+
+import com.intellij.DynamicBundle
+import org.jetbrains.annotations.Nls
+import org.jetbrains.annotations.PropertyKey
+
+/**
+ * Loads the `resources` directory.
+ */
+object KexLabelsBundle : DynamicBundle(KexBundlePaths.labels) {
+
+    /**
+     * Gets the requested default value.
+     */
+    @Nls
+    fun get(@PropertyKey(resourceBundle = KexBundlePaths.labels) key: String): String = getMessage(key)
+}

src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexMessagesBundle.kt

@@ -0,0 +1,17 @@
+package org.jetbrains.research.testspark.bundles.kex
+
+import com.intellij.DynamicBundle
+import org.jetbrains.annotations.Nls
+import org.jetbrains.annotations.PropertyKey
+
+/**
+ * Loads the `resources` directory.
+ */
+object KexMessagesBundle : DynamicBundle(KexBundlePaths.messages) {
+
+    /**
+     * Gets the requested default value.
+     */
+    @Nls
+    fun get(@PropertyKey(resourceBundle = KexBundlePaths.messages) key: String): String = getMessage(key)
+}

src/main/kotlin/org/jetbrains/research/testspark/bundles/kex/KexSettingsBundle.kt

@@ -0,0 +1,17 @@
+package org.jetbrains.research.testspark.bundles.kex
+
+import com.intellij.DynamicBundle
+import org.jetbrains.annotations.Nls
+import org.jetbrains.annotations.PropertyKey
+
+/**
+ * Loads the `resources` directory.
+ */
+object KexSettingsBundle : DynamicBundle(KexBundlePaths.settings) {
+
+    /**
+     * Gets the requested default value.
+     */
+    @Nls
+    fun get(@PropertyKey(resourceBundle = KexBundlePaths.settings) key: String): String = getMessage(key)
+}

src/main/kotlin/org/jetbrains/research/testspark/data/kex/KexMode.kt

@@ -0,0 +1,5 @@
+package org.jetbrains.research.testspark.data.kex
+
+enum class KexMode {
+    Symbolic, Concolic,
+}