Update readme and tool option descriptions (#132)

README.md

@@ -24,6 +24,7 @@ The JSON files in our benchmark of breaking dependency updates have the followin
 {
     "url": "<github pr url>",
     "project": "<github_project>",
+    "projectOrganisation": "<github_project_organisation>",
     "breakingCommit": "<sha>",
     "prAuthor": "{human|bot}",
     "preCommitAuthor": "{human|bot}",
@@ -38,7 +39,8 @@ The JSON files in our benchmark of breaking dependency updates have the followin
       "githubCompareLink": "<the github comparison link for the previous and breaking tag releases of the updated dependency if it exists>",
       "mavenSourceLinkPre": "<maven source jar link for the previous release of the updated dependency if it exists>",
       "mavenSourceLinkBreaking": "<maven source jar link for the breaking release of the updated dependency if it exists>",
-      "updatedFileType": "{pom|jar}"
+      "updatedFileType": "{pom|jar}",
+      "dependencySection" : "{dependencies|dependencyManagement|buildPlugins|buildPluginManagement|profileBuildPlugins}"
   },
     "preCommitReproductionCommand": "<the command to compile and run tests without the breaking update commit>",
     "breakingUpdateReproductionCommand": "<the command to compile and run tests with the breaking update commit>",
@@ -62,18 +64,15 @@ The data gathering workflow is as follows:
     * We use Maven version 3.8.6
     * We run OpenJDK
     * As a starting point, we use Java 11
-  * The reproduction can result in 5 different successful outcomes:
-    * The project build fails _after_ the dependency is updated due to unresolved dependencies, but not before.
-      This is a successful reproduction corresponding to the label "DEPENDENCY_RESOLUTION_FAILURE".
-    * The project build fails _after_ the dependency is updated due to maven enforcer plugin errors, but not before.
-      This is a successful reproduction corresponding to the label "MAVEN_ENFORCER_FAILURE".
+  * The reproduction can result in different successful outcomes based on the Maven goal where the failure happens. For example,
     * The compilation step fails _after_ the dependency is updated, but not before.
       This is a successful reproduction corresponding to the label "COMPILATION_FAILURE".
     * The test step fails _after_ the dependency is updated, but not before.
       This is a successful reproduction corresponding to the label "TEST_FAILURE".
-    * The project build fails _after_ the dependency is updated due to an unknown error which cannot be categorized
-      into above other failure types.
-      This is a successful reproduction corresponding to the label "UNKNOWN_FAILURE".
+    * The project build fails _after_ the dependency is updated due to unresolved dependencies, but not before.
+      This is a successful reproduction corresponding to the label "DEPENDENCY_RESOLUTION_FAILURE".
+    * The project build fails _after_ the dependency is updated when executing the maven enforcer plugin, but not before.
+      This is a successful reproduction corresponding to the label "MAVEN_ENFORCER_FAILURE".
 * Stage 4 : Build two Docker images for each successfully reproduced breaking update, 
             and isolate all environment / network requests by downloading them.
             After stage 4, by running the preCommitReproductionCommand, and the breakingUpdateReproductionCommand, 
@@ -100,12 +99,16 @@ java -jar target/BreakingUpdateReproducer.jar --help
 ```
 
 ## Stats
-As of Sep 4 2023:
-  * The benchmark consists of 629 reproducible breaking updates from 162 unique projects.
-    - Of these breaking updates, 263 (41.81%) fail compilation with the updated dependency.
-    - 212 (33.70%) fail tests with the updated dependency.
-    - 5 (0.79%) have dependency resolution failures with the updated dependency.
-    - 69 (10.97%) fail after updating the dependency due to maven enforcer failures.
-    - 3 (0.48%) fail due to unknown failures after updating the dependency.
-  * Overall, reproduction has been attempted for 5392 breaking updates, and 4763 (88.33%) could not be locally reproduced.
+As of Sep 6 2023:
+  * The benchmark consists of 628 reproducible breaking updates from 161 unique projects.
+    - Of these breaking updates, 263 (41.88%) fail compilation with the updated dependency.
+    - 213 (33.92%) fail tests with the updated dependency.
+    - 6 (0.96%) have dependency resolution failures with the updated dependency.
+    - 69 (10.99%) fail after updating the dependency due to maven enforcer failures.
+    - 54 (8.60%) fail due to Jenkins failures after updating the dependency.
+    - 14 (2.23%) fail due to dependency locks.
+    - 4 (0.64%) fail due to JAXB failures.
+    - 1 (0.16%) fail due to SCM plugin failures during the execution of the goal checkout.
+    - 4 (0.64%) fail due to Checkstyle failures after updating the dependency.
+  * Overall, reproduction has been attempted for 5391 breaking updates, and 4763 (88.35%) could not be locally reproduced.
   * For 0 potential breaking updates, reproduction has not been attempted yet.

data/benchmark/18102f7a87e7ab8308a86208d47696c060110d0c.json

@@ -13,14 +13,14 @@
     "newVersion" : "2.12.6.1",
     "dependencyScope" : "compile",
     "versionUpdateType" : "other",
+    "dependencySection" : "dependencyManagement",
     "githubCompareLink" : "https://github.com/fasterxml/jackson-databind/compare/jackson-databind-2.10.0...jackson-databind-2.12.6.1",
     "mavenSourceLinkPre" : "https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.10.0/jackson-databind-2.10.0-sources.jar",
     "mavenSourceLinkBreaking" : "https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.12.6.1/jackson-databind-2.12.6.1-sources.jar",
-    "updatedFileType" : "JAR",
-    "dependencySection" : "dependencyManagement"
+    "updatedFileType" : "JAR"
   },
   "preCommitReproductionCommand" : "docker run ghcr.io/chains-project/breaking-updates:18102f7a87e7ab8308a86208d47696c060110d0c-pre",
   "breakingUpdateReproductionCommand" : "docker run ghcr.io/chains-project/breaking-updates:18102f7a87e7ab8308a86208d47696c060110d0c-breaking",
   "javaVersionUsedForReproduction" : "11",
-  "failureCategory" : "UNKNOWN_FAILURE"
+  "failureCategory" : "TEST_FAILURE"
 }

data/benchmark/196375c2019dd6b324c102ed31ad34520e48ac01.json

@@ -22,5 +22,5 @@
   "preCommitReproductionCommand" : "docker run ghcr.io/chains-project/breaking-updates:196375c2019dd6b324c102ed31ad34520e48ac01-pre",
   "breakingUpdateReproductionCommand" : "docker run ghcr.io/chains-project/breaking-updates:196375c2019dd6b324c102ed31ad34520e48ac01-breaking",
   "javaVersionUsedForReproduction" : "11",
-  "failureCategory" : "UNKNOWN_FAILURE"
+  "failureCategory" : "DEPENDENCY_RESOLUTION_FAILURE"
 }

image_metadata.json

@@ -3486,10 +3486,10 @@
     "prevImageProjectFolderSize" : "884"
   },
   "18102f7a87e7ab8308a86208d47696c060110d0c" : {
-    "postImageM2FolderSize" : "39016",
-    "prevImageM2FolderSize" : "215284",
-    "postImageProjectFolderSize" : "94124",
-    "prevImageProjectFolderSize" : "189360"
+    "postImageM2FolderSize" : "164060",
+    "prevImageM2FolderSize" : "215276",
+    "postImageProjectFolderSize" : "197748",
+    "prevImageProjectFolderSize" : "189316"
   },
   "4b9fdf1135b704e050cf785d6ea353d24f4c64bb" : {
     "postImageM2FolderSize" : "93948",

scripts/calculate_stats.sh

@@ -12,7 +12,11 @@ num_compilation_failure=$(grep -ro "\"COMPILATION_FAILURE\"" data/benchmark | wc
 num_test_failure=$(grep -ro "\"TEST_FAILURE\"" data/benchmark | wc -l)
 num_dependency_resolution_failure=$(grep -ro "\"DEPENDENCY_RESOLUTION_FAILURE\"" data/benchmark | wc -l)
 num_mvn_enforcer_failure=$(grep -ro "\"MAVEN_ENFORCER_FAILURE\"" data/benchmark | wc -l)
-num_unknown_failure=$(grep -ro "\"UNKNOWN_FAILURE\"" data/benchmark | wc -l)
+num_jenkins_failure=$(grep -ro "\"JENKINS_PLUGIN_FAILURE\"" data/benchmark | wc -l)
+num_dependency_lock_failure=$(grep -ro "\"DEPENDENCY_LOCK_FAILURE\"" data/benchmark | wc -l)
+num_jaxb_failure=$(grep -ro "\"JAXB_FAILURE\"" data/benchmark | wc -l)
+num_scm_failure=$(grep -ro "\"SCM_CHECKOUT_FAILURE\"" data/benchmark | wc -l)
+num_checkstyle_failure=$(grep -ro "\"CHECKSTYLE_FAILURE\"" data/benchmark | wc -l)
 num_unreproducible=$(find data/unsuccessful-reproductions -iname "*.json" | wc -l)
 num_reproduced=$(find data/benchmark -iname "*.json" | wc -l)
 num_attempted=$(("$num_reproduced" + "$num_unreproducible"))
@@ -27,7 +31,11 @@ As of $(LC_TIME=en_US.UTF-8 date +"%b %-d %Y"):
     - $num_test_failure ($(get_fraction_as_percentage "$num_test_failure" "$num_reproduced")) fail tests with the updated dependency.
     - $num_dependency_resolution_failure ($(get_fraction_as_percentage "$num_dependency_resolution_failure" "$num_reproduced")) have dependency resolution failures with the updated dependency.
     - $num_mvn_enforcer_failure ($(get_fraction_as_percentage "$num_mvn_enforcer_failure" "$num_reproduced")) fail after updating the dependency due to maven enforcer failures.
-    - $num_unknown_failure ($(get_fraction_as_percentage "$num_unknown_failure" "$num_reproduced")) fail due to unknown failures after updating the dependency.
+    - $num_jenkins_failure ($(get_fraction_as_percentage "$num_jenkins_failure" "$num_reproduced")) fail due to Jenkins failures after updating the dependency.
+    - $num_dependency_lock_failure ($(get_fraction_as_percentage "$num_dependency_lock_failure" "$num_reproduced")) fail due to dependency locks.
+    - $num_jaxb_failure ($(get_fraction_as_percentage "$num_jaxb_failure" "$num_reproduced")) fail due to JAXB failures.
+    - $num_scm_failure ($(get_fraction_as_percentage "$num_scm_failure" "$num_reproduced")) fail due to SCM plugin failures during the execution of the goal checkout.
+    - $num_checkstyle_failure ($(get_fraction_as_percentage "$num_checkstyle_failure" "$num_reproduced")) fail due to Checkstyle failures after updating the dependency.
   * Overall, reproduction has been attempted for $num_attempted breaking updates, and $num_unreproducible ($(get_fraction_as_percentage "$num_unreproducible" "$num_attempted")) could not be locally reproduced.
   * For $num_not_attempted potential breaking updates, reproduction has not been attempted yet."
 export STATS

src/main/java/reproducer/Main.java

@@ -105,7 +105,8 @@ private static class Reproduce implements Runnable {
                 names = {"-ud", "--user-data-dir"},
                 paramLabel = "USER-DATA-DIR",
                 description = "The directory where the user data in Chrome is saved. This is required to keep an active " +
-                        "web session with GitHub."
+                        "web session with GitHub, when downloading the workflow log files. If it is not necessary to download" + 
+                        "workflow logs, this option can be ignored."
         )
         String userDataDir;