Integrate taintbench

README.md

@@ -136,30 +136,110 @@ To have detailed information about each group of tests run, [see here.](new-metr
 | JSVFA 2.0  |   0.91    |  0.76  |  0.83   |
 |   Basic    |   0.93    |  0.97  |  0.95   |
 
-### Taintbench: (WIP) 
+### TAINTBENCH:
 
-[Taintbench](https://github.com/TaintBench/TaintBench/releases/download/TaintBenchSuite/TaintBench.zip) contains a set o Android Apks that are old malware apps.
-We have created a file `taintbench.properties` in `src/test/resources` to set the configurations.
+[TAINTBENCH](https://taintbench.github.io/) is a benchmark that contains a [set of old malware Android Apks](https://github.com/TaintBench/TaintBench/releases/download/TaintBenchSuite/TaintBench.zip) , 
+and it is introduced by [Paper: TaintBench: Automatic real-world malware benchmarking of Android taint analyses](https://doi.org/10.1007/s10664-021-10013-5), 
+which in its result section presents six experiments to answer one of its RQ: *How effective are taint analysis tools on TaintBench compared to DroidBench*,
+where FLOWDROID and AMANDROID as the chosen tools.
 
-> failed: ?, passed: 1, ignored: ? of 39 test (?%)
+In the next sections, we will focus in **Experiment 2** and **Experiment 3** and use our tool (JSVFA) to reproduce them. 
+After that, we will compare the already computed results for FLOWDROID to our results.
 
-- [Roidsec]
-- [ ]
-- [ ]
+- The result for each APK tested using JSVFA are presented in a table that contains the following information.
+  - **Expected:** The amount of taint flows presented by TAINTBENCH
+  - **Actual:** The amount of taint flows detected by JSVFA
+  - **Status:** If the test PASS OR FAIL
+  - **TP:** True Positive
+  - **FP:** False Positive
+  - **Precision:** TP/(TP + FP)
+  - **Recall:** TP/P
+  - **F-score:** (2 x Precision x Recall)/(Precision + Recall)
+- We have created a file `taintbench.properties` in `src/test/resources` to set the configurations
 
+**Disclaimer**: Although TAINTBENCH contains 203 expected [taint flows](https://taintbench.github.io/taintbenchSuite/),
+we have decided to use only 186 expected cases because the mentioned paper, uses as a reference, works with those amounts.
+
+#### EXPERIMENT I
+
+This case emulates **Experiment 2 - TB2** that states:
+
+>All tools are configured with sources and sinks defined in benchmark suite.
+
+The mentioned sources and sinks can be found in [TB_SourcesAndSinks](https://github.com/TaintBench/TaintBench/blob/master/TB_SourcesAndSinks.txt), 
+and we have stored them in `src/test/scala/br/unb/cic/android/TaintBenchSpec.scala`.
+
+As a result, we got `37 failed and 2 passed of 39 tests` and comparing to FLOWDROID we detected the same amount of `TP(41)`
+but less amount of `FP(9)` and about metrics, we got a better `precision(0.82)`, the same `recall(0.22)` value
+and a slightly better `F-score(0.35)`.
+
+- JSVFA metrics, to have detailed information about each group of tests run, [see here.](taintbench-experiment-I.md)
+
+| Expected | Actual | TP | FP | Precision | Recall | F-score  |
+|:--------:|:------:|:--:|:--:|:---------:|:------:|:--------:|
+|   186    |   50   | 41 | 9  |   0.82    |  0.22  |   0.35   |
+
+- FLOWDROID metrics from [Paper](https://doi.org/10.1007/s10664-021-10013-5)
+
+| Expected | Actual | TP | FP | Precision | Recall | F-score  |
+|:--------:|:------:|:--:|:--:|:---------:|:------:|:--------:|
+|   186    |   55   | 41 | 14 |   0.75    |  0.22  |   0.34   |
+
+##### Observation
+- From the 37 failing tests, 28 of them reported zero flows.
+
+
+#### EXPERIMENT II
+
+This case emulates **Experiment 3 - TB3** that configures:
+
+>For each benchmark app, a list of sources and sinks defined in this app is used to 
+configure all tools. Each tool analyzes each benchmark app with the associated list 
+of sources and sinks 
+
+The mentioned lists can be found in https://taintbench.github.io/taintbenchSuite/, and we have stored them by individual
+files in `src/test/scala/br/unb/cic/android/specs`.
+
+As a result, we got `38 failed and 1 passed of 39 test` and comparing to FLOWDROID we detect a several better amount of `TP(135)`
+but also a several amount of `FP(318)` and about metric, we got a significant less `precision(0.30)` due to the high amount of FP; however,
+a good `recall(0.73)` value and a better `F-score(0.42)`.
+
+- JSVFA metrics, to have detailed information about each group of tests run, [see here.](taintbench-experiment-II.md)
+
+
+| Expected | Actual | TP  | FP  | Precision | Recall | F-score |
+|:--------:|:------:|:---:|:---:|:---------:|:------:|:-------:|
+|   186    |  453   | 135 | 318 |   0.30    |  0.73  |  0.42   |
+
+
+- FLOWDROID metrics from Paper https://doi.org/10.1007/s10664-021-10013-5
+
+| Expected | Actual | TP | FP | Precision | Recall | F-score |
+|:--------:|:------:|:--:|:--:|:---------:|:------:|:-------:|
+|   186    |   57   | 43 | 14 |   0.75    |  0.23  |  0.35   |
+
+
+##### Observation
+- We got a big amount of FP.
 
 ## Tasks
 ### WIP
+- [ ] Create Git Action flow.
+
+### TO-DO
 - [ ] Finish integration of Taintbench.
 - [ ] Check if each test in Securibench has the right expected values.
 - [ ] Add set up project documentation.
 - [ ] Integrate Securibench as a submodule.
+- [ ] Compute metrics for Securibench results.
 - [ ] Fix bugs for Securibench in folders
   - [ ] Datastructure
   - [ ] Factory
   - [ ] Session
   - [ ] Strong Update
   - [ ] Aliasing
 
+### DONE
+- [X] Integration of Taintbench.
 
 [//]: # (## License)

src/main/scala/br/unb/cic/soot/svfa/configuration/AndroidSootConfiguration.scala

@@ -12,6 +12,7 @@ trait AndroidSootConfiguration extends SootConfiguration {
 
   def platform(): String
   override def configureSoot(): Unit = {
+    // Set up InfoflowAndroidConfiguration
     val config = new InfoflowAndroidConfiguration
     config.setCallgraphAlgorithm(InfoflowConfiguration.CallgraphAlgorithm.CHA)
     config.setCodeEliminationMode(
@@ -24,8 +25,8 @@ trait AndroidSootConfiguration extends SootConfiguration {
     config.setIgnoreFlowsInSystemPackages(true)
     config.setExcludeSootLibraryClasses(true)
 
+    // Create the data flow analyzer
     val flowDroid = new SetupApplication(config)
-
     flowDroid.constructCallgraph()
   }
 

src/main/scala/br/unb/cic/soot/svfa/configuration/AndroidSootConfiguration2.scala

@@ -0,0 +1,134 @@
+package br.unb.cic.soot.svfa.configuration
+
+import soot._
+import soot.jimple.infoflow.InfoflowConfiguration
+import soot.jimple.infoflow.android.config.SootConfigForAndroid
+import soot.jimple.infoflow.android.{InfoflowAndroidConfiguration, SetupApplication}
+import soot.options.Options
+
+import java.io.File
+import java.util.Collections
+
+trait AndroidSootConfiguration2 extends SootConfiguration {
+
+  def apk: String
+
+  def platform(): String
+
+  def callGraph(): CG = SPARK
+
+  def applicationClassPath(): List[String] = List("target/scala-2.12/test-classes", System.getProperty("user.home")+"/.m2/repository/javax/servlet/javax.servlet-api/3.0.1/javax.servlet-api-3.0.1.jar")
+
+  def sootClassPath(): String = ""
+
+  override def configureSoot() {
+    // Do InfoflowAndroidConfiguration
+    val config = new InfoflowAndroidConfiguration
+    config.setCallgraphAlgorithm(InfoflowConfiguration.CallgraphAlgorithm.SPARK)
+    config.setCodeEliminationMode(
+      InfoflowConfiguration.CodeEliminationMode.NoCodeElimination
+    )
+    config.getAnalysisFileConfig.setAndroidPlatformDir(platform)
+    config.getAnalysisFileConfig.setTargetAPKFile(apk)
+
+    // Do Soot Configuration
+    G.reset()
+    soot.options.Options.v().set_allow_phantom_refs(true);
+    soot.options.Options.v().set_output_format(soot.options.Options.output_format_none);
+//    soot.options.Options.v().set_process_dir(Collections.singletonList(apk));
+//    soot.options.Options.v().set_force_android_jar(platform());
+    soot.options.Options.v().set_src_prec(soot.options.Options.src_prec_apk_class_jimple);
+    soot.options.Options.v().set_keep_line_number(false);
+    soot.options.Options.v().set_keep_offset(false);
+    soot.options.Options.v().set_throw_analysis(soot.options.Options.throw_analysis_dalvik);
+    soot.options.Options.v().set_process_multiple_dex(true);
+    soot.options.Options.v().set_whole_program(true);
+
+    // Soot config for Android
+    val sootConfig = new SootConfigForAndroid();
+    sootConfig.setSootOptions(soot.options.Options.v(), config);
+
+    // Create the data flow analyzer
+    val analyzer = new SetupApplication(config);
+    analyzer.setSootConfig(sootConfig);
+    analyzer.constructCallgraph();
+  }
+  def configureSoot2() {
+//    val config = new InfoflowAndroidConfiguration
+//    config.setCallgraphAlgorithm(InfoflowConfiguration.CallgraphAlgorithm.SPARK)
+//    config.setCodeEliminationMode(
+//      InfoflowConfiguration.CodeEliminationMode.NoCodeElimination
+//    )
+//    config.getAnalysisFileConfig.setAndroidPlatformDir(platform)
+//    config.getAnalysisFileConfig.setTargetAPKFile(apk)
+//
+//    val flowDroid = new SetupApplication(config)
+//
+//    flowDroid.constructCallgraph()
+
+//    val androidJar = config.getAnalysisFileConfig.getAndroidPlatformDir
+//    val apkFileLocation = config.getAnalysisFileConfig.getTargetAPKFile
+
+    // Clean up any old Soot instance we may have
+//    val config = new InfoflowAndroidConfiguration
+    G.reset
+
+    Options.v.set_no_bodies_for_excluded(true)
+    Options.v.set_allow_phantom_refs(true)
+    Options.v.set_output_format(Options.output_format_none)
+    Options.v.set_whole_program(true)
+    Options.v.set_process_dir(Collections.singletonList(apk))
+    Options.v.set_force_android_jar(platform())
+    Options.v.set_src_prec(Options.src_prec_apk_class_jimple)
+    Options.v.set_keep_offset(false)
+    Options.v.set_keep_line_number(true)
+    Options.v.set_throw_analysis(Options.throw_analysis_dalvik)
+    Options.v.set_process_multiple_dex(true)
+    Options.v.set_ignore_resolution_errors(true)
+
+    // Set soot phase option if original names should be used
+    Options.v.setPhaseOption("jb", "use-original-names:true")
+
+    // Set the Soot configuration options. Note that this will needs to be
+    // done before we compute the classpath.
+
+    Options
+      .v()
+      .set_soot_classpath(
+        sootClassPath() + File.pathSeparator + pathToJCE() + File.pathSeparator + pathToRT()
+      )
+    Main.v.autoSetOptions()
+    configureCallGraphPhase()
+
+    // Load whatever we need
+    Scene.v.loadNecessaryClasses()
+
+    // Make sure that we have valid Jimple bodies
+    PackManager.v.getPack("wjpp").apply()
+
+    // Patch the callgraph to support additional edges. We do this now,
+    // because during callback discovery, the context-insensitive callgraph
+    // algorithm would flood us with invalid edges.
+//    val configuration = new InfoflowAndroidConfiguration();
+//    configuration.setSootIntegrationMode(SootIntegrationMode.UseExistingInstance);
+//    val flowDroid = new SetupApplication(configuration)
+//    flowDroid.constructCallgraph()
+  }
+
+
+  def configureCallGraphPhase() {
+    callGraph() match {
+      case CHA => Options.v().setPhaseOption("cg.cha", "on")
+      case SPARK => {
+        Options.v().setPhaseOption("cg.spark", "on")
+//        Options.v().setPhaseOption("cg.spark", "cs-demand:true")
+//        Options.v().setPhaseOption("cg.spark", "string-constants:true")
+        Options.v.setPhaseOption("cg.spark", "string-constants:true")
+      }
+      case SPARK_LIBRARY => {
+        Options.v().setPhaseOption("cg.spark", "on")
+        Options.v().setPhaseOption("cg", "library:any-subtype")
+      }
+    }
+  }
+}