Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade all dependencies to the latest versions #230

Merged
merged 12 commits into from
Oct 23, 2023
Merged

Upgrade all dependencies to the latest versions #230

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
18db155
Unify style
Mingun Aug 25, 2021
ef035e8
Upgrade FastParse to the latest version (1.0.0->2.3.3)
Mingun Nov 23, 2020
92373c0
Upgrade ScalaTest to the latest version (3.2.12->3.2.15) with sbt-sca…
Mingun Dec 5, 2020
7b806de
Depend only on used ScalaTest modules:
Mingun Dec 5, 2020
d07272b
Upgrade sbt-native-packager plugin (1.3.12->1.9.16)
Mingun Dec 5, 2020
0cc4fb3
Use sbt-crossproject instead of deprecated crossProjectFromBuilder as…
Mingun Dec 5, 2020
6b03028
Upgrade scopt (3.6.0->4.1.0) and scala-js plugin (0.6.33->1.13.1)
Mingun Dec 5, 2020
6ba2dc9
Upgrade SnakeYAML (1.28->2.0)
Mingun Nov 20, 2020
086d687
project/plugins.sbt: add missing final newline
generalmimon Oct 21, 2023
cfc37f5
Adapt `snakeyaml.Yaml` constructor invocation to SnakeYAML 2.0
generalmimon Oct 21, 2023
c28a56c
Expressions: fix unrecognized backslash in string literal
generalmimon Oct 22, 2023
0fe8b69
Adapt `buildNpmJsFile` task to the new Scala.js 1.x output
generalmimon Oct 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 9 additions & 11 deletions build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ lazy val root = project.in(file(".")).
publishLocal := {}
)

lazy val compiler = crossProject.in(file(".")).
lazy val compiler = crossProject(JSPlatform, JVMPlatform).
in(file(".")).
enablePlugins(JavaAppPackaging).
settings(
organization := "io.kaitai",
Expand Down Expand Up @@ -70,17 +71,19 @@ lazy val compiler = crossProject.in(file(".")).
Compile / sourceGenerators += generateVersionTask.taskValue, // update automatically on every rebuild

libraryDependencies ++= Seq(
"com.github.scopt" %%% "scopt" % "3.6.0",
"com.lihaoyi" %%% "fastparse" % "1.0.0",
"org.yaml" % "snakeyaml" % "1.28"
"com.github.scopt" %%% "scopt" % "4.1.0",
"com.lihaoyi" %%% "fastparse" % "2.3.3",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noting that the format of FastParse error messages has changed, so when inserted to our own error message, we often end up with things like expected Expected .... See the comparison of sbt test output before and after this PR:

@@ -220,7 +220,7 @@ tr
 [info]         error: unable to access 'bar' in expr_field_unknown context (SimpleMatchers.scala:34)
 [info] - expr_inst_value_broken *** FAILED ***
 [info]   ..\tests\formats_err\expr_inst_value_broken.ksy: /instances/foo:
-[info]         error: parsing expression '1 *' failed on 1:3, expected "or" | CharsWhile(Set( , n)) | "\\\n" | End
+[info]         error: parsing expression '1 *' failed on 1:3, expected Expected end-of-input:1:3, found "*"
 [info]    did not equal expr_inst_value_broken.ksy: /instances/foo:
 [info]         error: parsing expression '1 *' failed on 1:3, expected "or" | CharsWhile(Set( , n)) | "\\\n" | End (SimpleMatchers.scala:34)
 [info] - expr_size_bad_id *** FAILED ***
@@ -258,7 +258,7 @@ tr
 [info]         error: unable to find type 'bar', searching from expr_type_cast_unknown (SimpleMatchers.scala:34)
 [info] - expr_unbalanced *** FAILED ***
 [info]   ..\tests\formats_err\expr_unbalanced.ksy: /seq/0:
-[info]         error: parsing expression '(1 + 5' failed on 1:1, expected "not" ~ !(namePart) ~ not_test | comparison
+[info]         error: parsing expression '(1 + 5' failed on 1:1, expected Expected (kw | comparison):1:1, found "(1 + 5"
 [info]    did not equal expr_unbalanced.ksy: /seq/0:
 [info]         error: parsing expression '(1 + 5' failed on 1:1, expected "not" ~ !(namePart) ~ not_test | comparison (SimpleMatchers.scala:34)
 [info] - expr_unknown_method1 *** FAILED ***
@@ -273,7 +273,7 @@ tr
 [info]         error: don't know how to call method 'frobnicate' of object type 'CalcBytesType' (SimpleMatchers.scala:34)
 [info] - expr_wrong_and *** FAILED ***
 [info]   ..\tests\formats_err\expr_wrong_and.ksy: /instances/both:
-[info]         error: parsing expression 'foo == 1 && bar == 2' failed on 1:10, expected "or" | CharsWhile(Set( , n)) | "\\\n" | End, did you mean 'and'?
+[info]         error: parsing expression 'foo == 1 && bar == 2' failed on 1:10, expected Expected end-of-input:1:10, found "&& bar == ", did you mean 'and'?
 [info]    did not equal expr_wrong_and.ksy: /instances/both:
 [info]         error: parsing expression 'foo == 1 && bar == 2' failed on 1:10, expected "or" | CharsWhile(Set( , n)) | "\\\n" | End, did you mean 'and'? (SimpleMatchers.scala:34)
 [info] - instance_name_bad *** FAILED ***
@@ -504,7 +504,7 @@ tr
 [info]         error: expected map, got foo (class java.lang.String) (SimpleMatchers.scala:34)
 [info] - switch_cases_malformed_quoting *** FAILED ***
 [info]   ..\tests\formats_err\switch_cases_malformed_quoting.ksy: /seq/1/cases/^AHEM:
-[info]         error: parsing expression '^AHEM' failed on 1:1, expected "not" ~ !(namePart) ~ not_test | comparison
+[info]         error: parsing expression '^AHEM' failed on 1:1, expected Expected (kw | comparison):1:1, found "^AHEM"
 [info]    did not equal switch_cases_malformed_quoting.ksy: /seq/1/cases/^AHEM:
 [info]         error: parsing expression '^AHEM' failed on 1:1, expected "not" ~ !(namePart) ~ not_test | comparison (SimpleMatchers.scala:34)
 [info] - switch_cases_malformed_quoting2 *** FAILED ***
@@ -534,7 +534,7 @@ tr
 [info]         error: missing mandatory argument `switch-on` (SimpleMatchers.scala:34)
 [info] - switch_on_malformed *** FAILED ***
 [info]   ..\tests\formats_err\switch_on_malformed.ksy: /seq/0:
-[info]         error: parsing expression '42/' failed on 1:3, expected "or" | CharsWhile(Set( , n)) | "\\\n" | End
+[info]         error: parsing expression '42/' failed on 1:3, expected Expected end-of-input:1:3, found "/"
 [info]    did not equal switch_on_malformed.ksy: /seq/0/switch-on:
 [info]         error: parsing expression '42/' failed on 1:3, expected "or" | CharsWhile(Set( , n)) | "\\\n" | End (SimpleMatchers.scala:34)
 [info] - top_empty *** FAILED ***

It's not that important, but we can think about how to improve this in the future.

"org.yaml" % "snakeyaml" % "2.0"
)
).
jvmSettings(
name := NAME,

Compile / mainClass := Some("io.kaitai.struct.JavaMain"),
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "3.2.12" % "test"
"org.scalatest" %% "scalatest-funspec" % "3.2.15" % "test",
"org.scalatest" %% "scalatest-funsuite" % "3.2.15" % "test",
"org.scalatest" %% "scalatest-shouldmatchers" % "3.2.15" % "test",
),

Test / testOptions += Tests.Argument(TestFrameworks.ScalaTest, "-u", "target/test_out"),
Expand Down Expand Up @@ -228,13 +231,8 @@ lazy val buildNpmJsFileTask = Def.task {
| }
|}(typeof self !== 'undefined' ? self : this, function () {
|
|var exports = {};
|var __ScalaJSEnv = { exportsNamespace: exports };
|
|$compiledFileContents
|
|return exports.MainJs;
|
|return MainJs;
|}));
|""".stripMargin

Expand Down
2 changes: 1 addition & 1 deletion js/src/main/scala/io/kaitai/struct/format/JavaScriptKSYParser.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ object JavaScriptKSYParser {
val yamlScala = yamlJavascriptToScala(yaml)
val firstSpec = ClassSpec.fromYaml(yamlScala, None)
val specs = new JavaScriptClassSpecs(importer, firstSpec)
Main.importAndPrecompile(specs, config).map{ problems =>
Main.importAndPrecompile(specs, config).map { problems =>
// throw the first (if any) severe (not a warning) problem as an exception
problems.find(p => p.severity != ProblemSeverity.Warning) match {
case Some(problem) => throw problem.toException
Expand Down
2 changes: 1 addition & 1 deletion jvm/src/main/scala/io/kaitai/struct/JavaMain.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ object JavaMain {

def parseCommandLine(args: Array[String]): Option[CLIConfig] = {
val parser = new scopt.OptionParser[CLIConfig](Version.name) {
override def showUsageOnError = true
override def showUsageOnError = Some(true)

head(Version.name, Version.version)

Expand Down
11 changes: 2 additions & 9 deletions jvm/src/main/scala/io/kaitai/struct/formats/JavaKSYParser.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,8 @@ import io.kaitai.struct.JavaMain.CLIConfig
import io.kaitai.struct.format.{ClassSpec, ClassSpecs}
import io.kaitai.struct.problems.{CompilationProblem, CompilationProblemException, ProblemCoords, ProblemSeverity, YAMLParserError}
import io.kaitai.struct.{Log, Main}
import org.yaml.snakeyaml.constructor.SafeConstructor
import org.yaml.snakeyaml.error.MarkedYAMLException
import org.yaml.snakeyaml.representer.Representer
import org.yaml.snakeyaml.{DumperOptions, LoaderOptions, Yaml}
import org.yaml.snakeyaml.{LoaderOptions, Yaml}

import scala.collection.JavaConverters._
import scala.concurrent.Await
Expand Down Expand Up @@ -71,12 +69,7 @@ object JavaKSYParser {
def getYamlLoader: Yaml = {
val loaderOptions = new LoaderOptions
loaderOptions.setAllowDuplicateKeys(false)
new Yaml(
new SafeConstructor,
new Representer,
new DumperOptions,
loaderOptions
)
new Yaml(loaderOptions)
}

def readerToYaml(reader: Reader): Any = {
Expand Down
5 changes: 3 additions & 2 deletions project/plugins.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
logLevel := Level.Warn

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.3.12")
addSbtPlugin("org.scala-js" % "sbt-scalajs" % "0.6.33")
addSbtPlugin("com.github.sbt" % "sbt-native-packager" % "1.9.16")
Copy link
Member

@generalmimon generalmimon Oct 23, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this update induces the following change (among others, which are just uninteresting refactorings) in the Bash script launcher in bin/kaitai-struct-compiler of generated artifacts:

--- 1/kaitai-struct-compiler-0.11-SNAPSHOT20231021.183147.746d773b/bin/kaitai-struct-compiler
+++ 2/kaitai-struct-compiler-0.11-SNAPSHOT20231023.110424.c6c6db7b/bin/kaitai-struct-compiler
   ...
@@ -267,7 +283,7 @@ loadConfigFile() {
 }

 # Now check to see if it's a good enough version
-# TODO - Check to see if we have a configured default java version, otherwise use 1.6
+# TODO - Check to see if we have a configured default java version, otherwise use 1.8
 java_version_check() {
   readonly java_version=$("$java_cmd" -version 2>&1 | awk -F '"' '/version/ {print $2}')
   if [[ "$java_version" == "" ]]; then
@@ -281,10 +297,10 @@ java_version_check() {
     if [[ "$major" -eq "1" ]]; then
      local major=$(echo "$java_version" | cut -d'.' -f2)
     fi
-    if [[ "$major" -lt "6" ]]; then
+    if [[ "$major" -lt "8" ]]; then
       echo
       echo The java installation you have is not up to date
-      echo $app_name requires at least version 1.6+, you have
+      echo $app_name requires at least version 1.8+, you have
       echo version $java_version
       echo
       echo Please go to http://www.java.com/getjava/ and download

So the version check has been updated to only accept Java 8+, as opposed to Java 6 as before. This change apparently comes from version https://github.com/sbt/sbt-native-packager/releases/tag/v1.9.12:

Merged pull requests:

At least in our case, it turned out that this only improves the error message, since the produced artifacts were compiled for at least Java 8 already.

If you try to run the Universal .zip build on Java 7, e.g. via Docker:

PS C:\temp\Mingun-upgrade-dependencies> docker run -it -v ${PWD}:/share azul/zulu-openjdk:7
root@699f1294788e:/# java -version
openjdk version "1.7.0_352"
OpenJDK Runtime Environment (Zulu 7.56.0.11-CA-linux64) (build 1.7.0_352-b01)
OpenJDK 64-Bit Server VM (Zulu 7.56.0.11-CA-linux64) (build 24.352-b01, mixed mode)

  1. If the universal .zip was built with a compiler version before this PR (https://github.com/kaitai-io/kaitai_struct_compiler/tree/746d773beb553c75f254a2a409a02fb4150b5b36), you'll get this:

    root@699f1294788e:/# /share/kaitai-struct-compiler-0.11-SNAPSHOT20231021.183147.746d773b/bin/kaitai-struct-compiler --version
Exception in thread "main" java.lang.UnsupportedClassVersionError: io/kaitai/struct/JavaMain : Unsupported major.minor version 52.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:808)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:448)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:65)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:349)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:348)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:430)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:329)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:363)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:588)

  2. If the universal .zip was built with a compiler version after this PR, it will produce a human-readable message:

    root@699f1294788e:/# /share/kaitai-struct-compiler-0.11-SNAPSHOT20231023.110424.c6c6db7b/bin/kaitai-struct-compiler --version

The java installation you have is not up to date
requires at least version 1.8+, you have
version 1.7.0_352

Please go to http://www.java.com/getjava/ and download
a valid Java Runtime and install before running .

So, this is purely a slight improvement, not dropping existing support for anything.

addSbtPlugin("org.scala-js" % "sbt-scalajs" % "1.13.1")
Copy link
Member

@generalmimon generalmimon Oct 22, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This causes a few changes in the generated js/npm/ folder after the sbt fastOptJS buildNpmJsFile buildNpmPackage invocation:

  1. The trick with providing an exportsNamespace that we later use to retrieve the MainJs object will no longer work, because the __ScalaJSEnv variable handling has been removed from the generated Scala.js code - see diff:

    @@ -9,24510 +9,37385 @@
 }(typeof self !== 'undefined' ? self : this, function () {

 var exports = {};
 var __ScalaJSEnv = { exportsNamespace: exports };

+let MainJs;
 (function(){
 'use strict';
-/* Scala.js runtime support
- * Copyright 2013 LAMP/EPFL
- * Author: Sébastien Doeraene
- */
-
-/* ---------------------------------- *
- * The top-level Scala.js environment *
- * ---------------------------------- */
-
-
-
-
-
-// Get the environment info
-var $env = (typeof __ScalaJSEnv === "object" && __ScalaJSEnv) ? __ScalaJSEnv : {};
-
-// Global scope
-var $g =
-  (typeof $env["global"] === "object" && $env["global"])
-    ? $env["global"]
-    : ((typeof global === "object" && global && global["Object"] === Object) ? global : this);
-$env["global"] = $g;
-
-
-
-
-// Where to send exports
-
-
-
-var $e =
-  (typeof $env["exportsNamespace"] === "object" && $env["exportsNamespace"])
-    ? $env["exportsNamespace"] : $g;
-
-$env["exportsNamespace"] = $e;
    @@ -97082,7 +108681,9 @@ var $d_scm_ArrayBuffer = new $TypeData().initClass({
   Ljava_io_Serializable: 1
 });
 $c_scm_ArrayBuffer.prototype.$classData = $d_scm_ArrayBuffer;
-$e.MainJs = $m_Lio_kaitai_struct_MainJs$();
+$L0 = new $c_RTLong(0, 0);
+$d_J.zero = $L0;
+MainJs = $m_Lio_kaitai_struct_MainJs$();
 }).call(this);
 //# sourceMappingURL=kaitai-struct-compiler-js-fastopt.js.map


 return exports.MainJs;

 }));

    (Notice our return exports.MainJs; statement at the end, which ends up evaluating to undefined, because the var exports = {}; variable hasn't changed since we initialized it.)

    So we need to adapt our building script:

    kaitai_struct_compiler/build.sbt

    Lines 229 to 238 in 746d773

    |}(typeof self !== 'undefined' ? self : this, function () {
    |
    |var exports = {};
    |var __ScalaJSEnv = { exportsNamespace: exports };
    |
    |$compiledFileContents
    |
    |return exports.MainJs;
    |
    |}));

  2. The $linkingInfo object properties have changed from "assumingES6": false to "assumingES6": true + "esVersion": 6, indicating that it's been compiled for ECMAScript 6. This is probably fine, ES6 comes from 2015 and thus it's very well supported in pretty much all modern browsers and environments (see https://compat-table.github.io/compat-table/es6/). Just wanted to point out this change, even though I don't think there's anything wrong with it.

    Nowadays, this change might only be a problem for special environments like Duktape, but I haven't seen any project that would be running the JS build of kaitai-struct-compiler via Duktape (the closest thing I'm aware of is https://github.com/taviso/kiewtai, but it uses Duktape only to run pre-generated JavaScript parsers and the KS runtime lib for JS). So that's really a niche concern, and there are tools like Babel that can transform ES6 code to be ES5 compatible if someone needs it.

     var $linkingInfo = {
-  "envInfo": $env,
-  "assumingES6": false,
-  "semantics": {
-    // ...
-    "productionMode": false
-  },
-  "linkerVersion": "0.6.33",
-  "globalThis": this
+  "esVersion": 6,
+  "assumingES6": true,
+  "productionMode": false,
+  "linkerVersion": "1.13.1",
+  "fileLevelThis": this
 };

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, both of these changes are mentioned in the release post:

  1. https://www.scala-js.org/news/2020/02/25/announcing-scalajs-1.0.0/#java-system-properties-are-not-loaded-from-__scalajsenv-nor-sbts-javaoptions-anymore
  2. https://www.scala-js.org/news/2020/02/25/announcing-scalajs-1.0.0/#scalajs-emits-ecmascript-2015-code-by-default

addSbtPlugin("org.portable-scala" % "sbt-scalajs-crossproject" % "1.0.0")
Loading