Upgrade all dependencies to the latest versions

[Mingun]

Fixes kaitai-io/kaitai_struct#683

[generalmimon]

The commit d10b795 has been apparently pulled here by accident, it belongs to #222.

[Mingun]

No, this branch is intentionally based on the #222 branch.

Not checked yet that project is in a buildable state after each commit after rebase, so converted to draft.

[Mingun]

AS time goes it becomes more and more difficult to monitor the relevance of this PR. If you are not ready to take it all at once, then you can cherry-pick individual commits. The two scala-test commits are independent from the others, SnakeYaml and FastParse upgrades are also isolated and easy-testable.

The most significant upgrade is the scala-js because I have no ideas how to test it -- building kaitai-struct from sources still a complicated and not-well documented task and I have no time to learn it again. At least I tried to collect all useful links about changes in the each component -- see the corresponding commit messages.

Upgrading sbt-related infrastructure should be done in the commit order, because each step depending on the previous.

[Mingun]

Just note, that scala-js up to 1.10 has a security vulnerability and it is recommended to upgrade ASAP:
https://www.scala-js.org/news/2022/04/04/announcing-scalajs-1.10.0/

[generalmimon]

Just noting that the format of FastParse error messages has changed, so when inserted to our own error message, we often end up with things like expected Expected .... See the comparison of sbt test output before and after this PR:

@@ -220,7 +220,7 @@ tr
 [info]         error: unable to access 'bar' in expr_field_unknown context (SimpleMatchers.scala:34)
 [info] - expr_inst_value_broken *** FAILED ***
 [info]   ..\tests\formats_err\expr_inst_value_broken.ksy: /instances/foo:
-[info]         error: parsing expression '1 *' failed on 1:3, expected "or" | CharsWhile(Set( , n)) | "\\\n" | End
+[info]         error: parsing expression '1 *' failed on 1:3, expected Expected end-of-input:1:3, found "*"
 [info]    did not equal expr_inst_value_broken.ksy: /instances/foo:
 [info]         error: parsing expression '1 *' failed on 1:3, expected "or" | CharsWhile(Set( , n)) | "\\\n" | End (SimpleMatchers.scala:34)
 [info] - expr_size_bad_id *** FAILED ***
@@ -258,7 +258,7 @@ tr
 [info]         error: unable to find type 'bar', searching from expr_type_cast_unknown (SimpleMatchers.scala:34)
 [info] - expr_unbalanced *** FAILED ***
 [info]   ..\tests\formats_err\expr_unbalanced.ksy: /seq/0:
-[info]         error: parsing expression '(1 + 5' failed on 1:1, expected "not" ~ !(namePart) ~ not_test | comparison
+[info]         error: parsing expression '(1 + 5' failed on 1:1, expected Expected (kw | comparison):1:1, found "(1 + 5"
 [info]    did not equal expr_unbalanced.ksy: /seq/0:
 [info]         error: parsing expression '(1 + 5' failed on 1:1, expected "not" ~ !(namePart) ~ not_test | comparison (SimpleMatchers.scala:34)
 [info] - expr_unknown_method1 *** FAILED ***
@@ -273,7 +273,7 @@ tr
 [info]         error: don't know how to call method 'frobnicate' of object type 'CalcBytesType' (SimpleMatchers.scala:34)
 [info] - expr_wrong_and *** FAILED ***
 [info]   ..\tests\formats_err\expr_wrong_and.ksy: /instances/both:
-[info]         error: parsing expression 'foo == 1 && bar == 2' failed on 1:10, expected "or" | CharsWhile(Set( , n)) | "\\\n" | End, did you mean 'and'?
+[info]         error: parsing expression 'foo == 1 && bar == 2' failed on 1:10, expected Expected end-of-input:1:10, found "&& bar == ", did you mean 'and'?
 [info]    did not equal expr_wrong_and.ksy: /instances/both:
 [info]         error: parsing expression 'foo == 1 && bar == 2' failed on 1:10, expected "or" | CharsWhile(Set( , n)) | "\\\n" | End, did you mean 'and'? (SimpleMatchers.scala:34)
 [info] - instance_name_bad *** FAILED ***
@@ -504,7 +504,7 @@ tr
 [info]         error: expected map, got foo (class java.lang.String) (SimpleMatchers.scala:34)
 [info] - switch_cases_malformed_quoting *** FAILED ***
 [info]   ..\tests\formats_err\switch_cases_malformed_quoting.ksy: /seq/1/cases/^AHEM:
-[info]         error: parsing expression '^AHEM' failed on 1:1, expected "not" ~ !(namePart) ~ not_test | comparison
+[info]         error: parsing expression '^AHEM' failed on 1:1, expected Expected (kw | comparison):1:1, found "^AHEM"
 [info]    did not equal switch_cases_malformed_quoting.ksy: /seq/1/cases/^AHEM:
 [info]         error: parsing expression '^AHEM' failed on 1:1, expected "not" ~ !(namePart) ~ not_test | comparison (SimpleMatchers.scala:34)
 [info] - switch_cases_malformed_quoting2 *** FAILED ***
@@ -534,7 +534,7 @@ tr
 [info]         error: missing mandatory argument `switch-on` (SimpleMatchers.scala:34)
 [info] - switch_on_malformed *** FAILED ***
 [info]   ..\tests\formats_err\switch_on_malformed.ksy: /seq/0:
-[info]         error: parsing expression '42/' failed on 1:3, expected "or" | CharsWhile(Set( , n)) | "\\\n" | End
+[info]         error: parsing expression '42/' failed on 1:3, expected Expected end-of-input:1:3, found "/"
 [info]    did not equal switch_on_malformed.ksy: /seq/0/switch-on:
 [info]         error: parsing expression '42/' failed on 1:3, expected "or" | CharsWhile(Set( , n)) | "\\\n" | End (SimpleMatchers.scala:34)
 [info] - top_empty *** FAILED ***

It's not that important, but we can think about how to improve this in the future.

[generalmimon]

This causes a few changes in the generated js/npm/ folder after the sbt fastOptJS buildNpmJsFile buildNpmPackage invocation:

1. The trick with providing an exportsNamespace that we later use to retrieve the MainJs object will no longer work, because the __ScalaJSEnv variable handling has been removed from the generated Scala.js code - see diff:

   @@ -9,24510 +9,37385 @@
    }(typeof self !== 'undefined' ? self : this, function () {
   
    var exports = {};
    var __ScalaJSEnv = { exportsNamespace: exports };
   
   +let MainJs;
    (function(){
    'use strict';
   -/* Scala.js runtime support
   - * Copyright 2013 LAMP/EPFL
   - * Author: Sébastien Doeraene
   - */
   -
   -/* ---------------------------------- *
   - * The top-level Scala.js environment *
   - * ---------------------------------- */
   -
   -
   -
   -
   -
   -// Get the environment info
   -var $env = (typeof __ScalaJSEnv === "object" && __ScalaJSEnv) ? __ScalaJSEnv : {};
   -
   -// Global scope
   -var $g =
   -  (typeof $env["global"] === "object" && $env["global"])
   -    ? $env["global"]
   -    : ((typeof global === "object" && global && global["Object"] === Object) ? global : this);
   -$env["global"] = $g;
   -
   -
   -
   -
   -// Where to send exports
   -
   -
   -
   -var $e =
   -  (typeof $env["exportsNamespace"] === "object" && $env["exportsNamespace"])
   -    ? $env["exportsNamespace"] : $g;
   -
   -$env["exportsNamespace"] = $e;

   @@ -97082,7 +108681,9 @@ var $d_scm_ArrayBuffer = new $TypeData().initClass({
      Ljava_io_Serializable: 1
    });
    $c_scm_ArrayBuffer.prototype.$classData = $d_scm_ArrayBuffer;
   -$e.MainJs = $m_Lio_kaitai_struct_MainJs$();
   +$L0 = new $c_RTLong(0, 0);
   +$d_J.zero = $L0;
   +MainJs = $m_Lio_kaitai_struct_MainJs$();
    }).call(this);
    //# sourceMappingURL=kaitai-struct-compiler-js-fastopt.js.map
   
   
    return exports.MainJs;
   
    }));

   (Notice our return exports.MainJs; statement at the end, which ends up evaluating to undefined, because the var exports = {}; variable hasn't changed since we initialized it.)

   So we need to adapt our building script:

   kaitai_struct_compiler/build.sbt

   Lines 229 to 238 in 746d773

   	|}(typeof self !== 'undefined' ? self : this, function () {
   	|
   	|var exports = {};
   	|var __ScalaJSEnv = { exportsNamespace: exports };
   	|
   	|$compiledFileContents
   	|
   	|return exports.MainJs;
   	|
   	|}));

2. The $linkingInfo object properties have changed from "assumingES6": false to "assumingES6": true + "esVersion": 6, indicating that it's been compiled for ECMAScript 6. This is probably fine, ES6 comes from 2015 and thus it's very well supported in pretty much all modern browsers and environments (see https://compat-table.github.io/compat-table/es6/). Just wanted to point out this change, even though I don't think there's anything wrong with it.

   Nowadays, this change might only be a problem for special environments like Duktape, but I haven't seen any project that would be running the JS build of kaitai-struct-compiler via Duktape (the closest thing I'm aware of is https://github.com/taviso/kiewtai, but it uses Duktape only to run pre-generated JavaScript parsers and the KS runtime lib for JS). So that's really a niche concern, and there are tools like Babel that can transform ES6 code to be ES5 compatible if someone needs it.

    var $linkingInfo = {
   -  "envInfo": $env,
   -  "assumingES6": false,
   -  "semantics": {
   -    // ...
   -    "productionMode": false
   -  },
   -  "linkerVersion": "0.6.33",
   -  "globalThis": this
   +  "esVersion": 6,
   +  "assumingES6": true,
   +  "productionMode": false,
   +  "linkerVersion": "1.13.1",
   +  "fileLevelThis": this
    };

[generalmimon]

BTW, both of these changes are mentioned in the release post:

1. https://www.scala-js.org/news/2020/02/25/announcing-scalajs-1.0.0/#java-system-properties-are-not-loaded-from-__scalajsenv-nor-sbts-javaoptions-anymore
2. https://www.scala-js.org/news/2020/02/25/announcing-scalajs-1.0.0/#scalajs-emits-ecmascript-2015-code-by-default

[generalmimon]

Note: this update induces the following change (among others, which are just uninteresting refactorings) in the Bash script launcher in bin/kaitai-struct-compiler of generated artifacts:

--- 1/kaitai-struct-compiler-0.11-SNAPSHOT20231021.183147.746d773b/bin/kaitai-struct-compiler
+++ 2/kaitai-struct-compiler-0.11-SNAPSHOT20231023.110424.c6c6db7b/bin/kaitai-struct-compiler
   ...
@@ -267,7 +283,7 @@ loadConfigFile() {
 }

 # Now check to see if it's a good enough version
-# TODO - Check to see if we have a configured default java version, otherwise use 1.6
+# TODO - Check to see if we have a configured default java version, otherwise use 1.8
 java_version_check() {
   readonly java_version=$("$java_cmd" -version 2>&1 | awk -F '"' '/version/ {print $2}')
   if [[ "$java_version" == "" ]]; then
@@ -281,10 +297,10 @@ java_version_check() {
     if [[ "$major" -eq "1" ]]; then
      local major=$(echo "$java_version" | cut -d'.' -f2)
     fi
-    if [[ "$major" -lt "6" ]]; then
+    if [[ "$major" -lt "8" ]]; then
       echo
       echo The java installation you have is not up to date
-      echo $app_name requires at least version 1.6+, you have
+      echo $app_name requires at least version 1.8+, you have
       echo version $java_version
       echo
       echo Please go to http://www.java.com/getjava/ and download

So the version check has been updated to only accept Java 8+, as opposed to Java 6 as before. This change apparently comes from version https://github.com/sbt/sbt-native-packager/releases/tag/v1.9.12:

Merged pull requests:

• require java 8 require java 8 sbt/sbt-native-packager#1519

At least in our case, it turned out that this only improves the error message, since the produced artifacts were compiled for at least Java 8 already.

If you try to run the Universal .zip build on Java 7, e.g. via Docker:

PS C:\temp\Mingun-upgrade-dependencies> docker run -it -v ${PWD}:/share azul/zulu-openjdk:7
root@699f1294788e:/# java -version
openjdk version "1.7.0_352"
OpenJDK Runtime Environment (Zulu 7.56.0.11-CA-linux64) (build 1.7.0_352-b01)
OpenJDK 64-Bit Server VM (Zulu 7.56.0.11-CA-linux64) (build 24.352-b01, mixed mode)

1. If the universal .zip was built with a compiler version before this PR (https://github.com/kaitai-io/kaitai_struct_compiler/tree/746d773beb553c75f254a2a409a02fb4150b5b36), you'll get this:

   root@699f1294788e:/# /share/kaitai-struct-compiler-0.11-SNAPSHOT20231021.183147.746d773b/bin/kaitai-struct-compiler --version
   Exception in thread "main" java.lang.UnsupportedClassVersionError: io/kaitai/struct/JavaMain : Unsupported major.minor version 52.0
           at java.lang.ClassLoader.defineClass1(Native Method)
           at java.lang.ClassLoader.defineClass(ClassLoader.java:808)
           at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
           at java.net.URLClassLoader.defineClass(URLClassLoader.java:448)
           at java.net.URLClassLoader.access$100(URLClassLoader.java:65)
           at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
           at java.net.URLClassLoader$1.run(URLClassLoader.java:349)
           at java.security.AccessController.doPrivileged(Native Method)
           at java.net.URLClassLoader.findClass(URLClassLoader.java:348)
           at java.lang.ClassLoader.loadClass(ClassLoader.java:430)
           at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:329)
           at java.lang.ClassLoader.loadClass(ClassLoader.java:363)
           at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:588)
   

2. If the universal .zip was built with a compiler version after this PR, it will produce a human-readable message:

   root@699f1294788e:/# /share/kaitai-struct-compiler-0.11-SNAPSHOT20231023.110424.c6c6db7b/bin/kaitai-struct-compiler --version
   
   The java installation you have is not up to date
   requires at least version 1.8+, you have
   version 1.7.0_352
   
   Please go to http://www.java.com/getjava/ and download
   a valid Java Runtime and install before running .
   

---

So, this is purely a slight improvement, not dropping existing support for anything.

[generalmimon]

@Mingun Thanks a lot, merging in!

[Mingun]

Thanks you that you finally get a time to review and finish this. Glad to see any movements in the project!