Skip to content

ASSERT-KTH/pankti-experiments

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

pankti-experiments

Scripts, generated tests, and data by pankti

Project Selection Criteria

Regarding the experimental targets, the following criteria is applied:

  • The project should be a real-world product which has a certain number of end-users instead of a prototype/toy example
  • The project should contain a certain number of GitHub stars and commits, which indicate its popularity and complexity
  • The project can be deployed with the research lab's calculation resource, with a relatively realistic workload for experiments

There are 3 projects selected for the experiments. The following table shows the descriptive metrics of each project.

project Version LOC Classes App. Methods Test Methods TS Method Cov. TS Line Cov. WL Method Cov. WL Line Cov.
jitsi-jicofo stable-4857 Sum: 28,318
Java: 25,165
146 1,350 63 49.4%
667/1,350
46.7%
3,537/7,571
48.9%
660/1,350
46.2%
3,500/7,571
PDFBox 2.0.21 Sum: 728,812
Java: 161,976
1,278 11,042 1,675 54.8%
6,049/11,042
53.5%
34,653/64,787
21.6%
2,390/11,042
21.0%
13,630/64,787
Broadleaf 6.1.4-GA Sum: 618,465
Java: 211,030
684 6,173 197 23.9%
1,478/6,173
23.7%
5,846/24,667
23.1%
1,424/6,173
19.2%
4,735/24,667

The Commands for Descriptive Metrics Measurement

lines of code

Command cloc (Count Lines of Code) is used to calculated LOC.

cloc .

Classes, TS Method Cov., and TS Line Cov.

Add the following xml code into the pom file of the target project within <project></project>.

<build>
  <plugins>
    <plugin>
      <groupId>org.jacoco</groupId>
      <artifactId>jacoco-maven-plugin</artifactId>
      <version>0.8.5</version>
      <executions>
        <execution>
          <id>default-prepare-agent</id>
          <goals>
            <goal>prepare-agent</goal>
          </goals>
        </execution>
        <execution>
          <id>default-report</id>
          <goals>
            <goal>report</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

Then run mvn test and mvn jacoco:report. The JaCoCo report will be generated at ./target/site/jacoco. If the project contains multiple modules, the plugin can be declared in the root pom file. Run mvn jacoco:report-aggregate after mvn test to get an aggregated report.

JaCoCo for Maven Multi-Module Builds

If a project contains multiple sub-modules, some extra work might be needed in order to generate an aggregated JaCoCo report. (https://github.com/jacoco/jacoco/wiki/MavenMultiModule)

  • Create a dedicated module in your project for generation of the report. This module should depend on all or some other modules in the project. Here is an example for PDFBox.
  • Run mvn jacoco:report-aggregate after mvn test to get an aggregated report.
  • If your project already defines VM arguments for test execution, be sure that they will include property defined by JaCoCo. (https://www.jacoco.org/jacoco/trunk/doc/prepare-agent-mojo.html)
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <argLine>@{argLine} -your -extra -arguments</argLine>
  </configuration>
</plugin>
Another Workaround for Aggregated Report Generation
  • Simply run mvn test first, after which a set of jacoco.exec files should be generated in each sub-module's folder
  • Use jacococli.jar to merge all the execution data files, e.g.,
java -jar jacococli.jar merge sub-module-a/jacoco.exec sub-module-b/jacoco.exec --destfile jacoco-merged.exec
  • Use jacococli.jar to generate an aggregated report manually
java -jar jacococli.jar report ./jacoco-merged.exec \
--classfiles ./sub-module-a/target/classes \
--classfiles ./sub-module-b/target/classes \
--sourcefiles ./sub-module-a/src/main/java \
--sourcefiles ./sub-module-b/src/main/java \
--html ./jacoco-report

This is very straightforward, though a bit tedious.

WL (workload) Method Cov. and WL Line Cov.

Download JaCoCo agent jar from here: https://www.jacoco.org/jacoco/

Add the following options when starting the target application:

-javaagent:<path-to-jacoco>/lib/jacocoagent.jar

Then feed the application with the workload, the coverage information will be dumped into file jacoco.exec when the JVM terminates. Use the following command to generate an html report:

java -jar <path-to-jacoco>/lib/jacococli.jar report ./jacoco.exec --classfiles <path-to-application>/target/classes/com --sourcefiles <path-to-application>/src/main/java --html ./jacoco-report

Note that --classfiles and --sourcefiles can be specified multiple times if you want to analyze several specific sub-modules in the project.


Target method invocations and object profile counts for Jicofo, PDFBox, and BroadLeaf