Upgrade JDK toolchain/target version to 17 and add GraalJS #1207

gregschohn · 2024-12-20T04:15:27Z

Upgrade JDK toolchain and target version for most packages to 17 to get GraalJS+LiquidJS working via a transformer. After noticing extremely dissappointing results for liquid templated transforms (see below), I reworked the type mappings sanitization transform in pure javascript code. The results for recording 10K bulk request transformations went from

Old jinja transform

[INFO ] 2024-12-22 02:37:04,761762 [main] TypeMappingsSanitizationTransformerBulkTest - Total time=PT14.153184457S over 10000 runs

New javascript based transformation

[INFO ] 2024-12-22 02:42:58,180275 [main] TypeMappingsSanitizationTransformerBulkTest - Total time=PT0.904187796S over 10000 runs

That's a 14x improvement over the jinja code. Considering that we noticed that RFS performance went down by 20x, that would imply that 95% of the time was spent on transforms. Making that 14x faster could imply a ~10x performance increase (or only 2x worse than what it was without transformations). Notice that there's a lot less serialization back and forth with the javascript transform than with the template approach, but the code's also pretty sub-optimal, possibly like what I would expect to come in from a contributor - at least if they didn't have much experience with javascript.

There are two commits in this PR. The first one implements the TypeMappinsSanitization as an incomplete liquid transform. I stopped developing that because a rough test showed a huge performance difference between Liquid.js and a raw javascript call. The results on my mac were about 300x slower to use the templating solution than to just run JSON.stringify() (likely much worse than jinjava).

Below are some logs showing those differences for the liquid transforms.

/Library/Java/JavaVirtualMachines/amazon-corretto-17.jdk/Contents/Home/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCI ...
org.opensearch.migrations.transform.LiquidJsTransformerTest
[To redirect Truffle log output to a file use one of the following options:
* '--log.file=<path>' if the option is passed using a guest language launcher.
* '-Dpolyglot.log.file=<path>' if the option is passed using the host Java launcher.
* Configure logging using the polyglot embedding API.]
[engine] WARNING: The polyglot engine uses a fallback runtime that does not support runtime compilation to native code.
Execution without runtime compilation will negatively impact the guest application performance.
The following cause was found: Libgraal compilation is not available on this JVM. Alternatively, the org.graalvm.compiler:compiler module can be put on the --upgrade-module-path.
For more information see: https://www.graalvm.org/latest/reference-manual/embed-languages/#runtime-optimization-support.
To disable this warning use the '--engine.WarnInterpreterOnly=false' option or the '-Dpolyglot.engine.WarnInterpreterOnly=false' system property.
[To redirect Truffle log output to a file use one of the following options:
* '--log.file=<path>' if the option is passed using a guest language launcher.
* '-Dpolyglot.log.file=<path>' if the option is passed using the host Java launcher.
* Configure logging using the polyglot embedding API.]
[engine] WARNING: The polyglot engine uses a fallback runtime that does not support runtime compilation to native code.
Execution without runtime compilation will negatively impact the guest application performance.
The following cause was found: Libgraal compilation is not available on this JVM. Alternatively, the org.graalvm.compiler:compiler module can be put on the --upgrade-module-path.
For more information see: https://www.graalvm.org/latest/reference-manual/embed-languages/#runtime-optimization-support.
To disable this warning use the '--engine.WarnInterpreterOnly=false' option or the '-Dpolyglot.engine.WarnInterpreterOnly=false' system property.
[INFO ] 2024-12-20 04:11:56,654825 [main] LiquidJsTransformerTest - Run 0: PT4.052694709S
[INFO ] 2024-12-20 04:11:59,694596 [main] LiquidJsTransformerTest - Run 1: PT3.039395S
[INFO ] 2024-12-20 04:12:02,747087 [main] LiquidJsTransformerTest - Run 2: PT3.052563917S
[INFO ] 2024-12-20 04:12:05,823644 [main] LiquidJsTransformerTest - Run 3: PT3.076653792S
[INFO ] 2024-12-20 04:12:08,833241 [main] LiquidJsTransformerTest - Run 4: PT3.009687375S
[To redirect Truffle log output to a file use one of the following options:
* '--log.file=<path>' if the option is passed using a guest language launcher.
* '-Dpolyglot.log.file=<path>' if the option is passed using the host Java launcher.
* Configure logging using the polyglot embedding API.]
[engine] WARNING: The polyglot engine uses a fallback runtime that does not support runtime compilation to native code.
Execution without runtime compilation will negatively impact the guest application performance.
The following cause was found: Libgraal compilation is not available on this JVM. Alternatively, the org.graalvm.compiler:compiler module can be put on the --upgrade-module-path.
For more information see: https://www.graalvm.org/latest/reference-manual/embed-languages/#runtime-optimization-support.
To disable this warning use the '--engine.WarnInterpreterOnly=false' option or the '-Dpolyglot.engine.WarnInterpreterOnly=false' system property.
[INFO ] 2024-12-20 04:12:08,906295 [main] LiquidJsTransformerTest - Run 0: PT0.021998S
[INFO ] 2024-12-20 04:12:08,916581 [main] LiquidJsTransformerTest - Run 1: PT0.010198S
[INFO ] 2024-12-20 04:12:08,927812 [main] LiquidJsTransformerTest - Run 2: PT0.011137708S
[INFO ] 2024-12-20 04:12:08,938391 [main] LiquidJsTransformerTest - Run 3: PT0.010506958S
[INFO ] 2024-12-20 04:12:08,948759 [main] LiquidJsTransformerTest - Run 4: PT0.010288125S
[INFO ] 2024-12-20 04:12:08,959167 [main] LiquidJsTransformerTest - Run 5: PT0.010324625S

Description

Issues Resolved

Testing

Check List

New functionality includes testing
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…et GraalJS+LiquidJS working via a transformer. There's a rough test in place that shows the relative performance between Liquid.js and a raw javascript call. The results on my mac were about 300x slower to use the templating solution than to just run JSON.stringify(). /Library/Java/JavaVirtualMachines/amazon-corretto-17.jdk/Contents/Home/bin/java -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCI ... org.opensearch.migrations.transform.LiquidJsTransformerTest [To redirect Truffle log output to a file use one of the following options: * '--log.file=<path>' if the option is passed using a guest language launcher. * '-Dpolyglot.log.file=<path>' if the option is passed using the host Java launcher. * Configure logging using the polyglot embedding API.] [engine] WARNING: The polyglot engine uses a fallback runtime that does not support runtime compilation to native code. Execution without runtime compilation will negatively impact the guest application performance. The following cause was found: Libgraal compilation is not available on this JVM. Alternatively, the org.graalvm.compiler:compiler module can be put on the --upgrade-module-path. For more information see: https://www.graalvm.org/latest/reference-manual/embed-languages/#runtime-optimization-support. To disable this warning use the '--engine.WarnInterpreterOnly=false' option or the '-Dpolyglot.engine.WarnInterpreterOnly=false' system property. [To redirect Truffle log output to a file use one of the following options: * '--log.file=<path>' if the option is passed using a guest language launcher. * '-Dpolyglot.log.file=<path>' if the option is passed using the host Java launcher. * Configure logging using the polyglot embedding API.] [engine] WARNING: The polyglot engine uses a fallback runtime that does not support runtime compilation to native code. Execution without runtime compilation will negatively impact the guest application performance. The following cause was found: Libgraal compilation is not available on this JVM. Alternatively, the org.graalvm.compiler:compiler module can be put on the --upgrade-module-path. For more information see: https://www.graalvm.org/latest/reference-manual/embed-languages/#runtime-optimization-support. To disable this warning use the '--engine.WarnInterpreterOnly=false' option or the '-Dpolyglot.engine.WarnInterpreterOnly=false' system property. [INFO ] 2024-12-20 04:11:56,654825 [main] LiquidJsTransformerTest - Run 0: PT4.052694709S [INFO ] 2024-12-20 04:11:59,694596 [main] LiquidJsTransformerTest - Run 1: PT3.039395S [INFO ] 2024-12-20 04:12:02,747087 [main] LiquidJsTransformerTest - Run 2: PT3.052563917S [INFO ] 2024-12-20 04:12:05,823644 [main] LiquidJsTransformerTest - Run 3: PT3.076653792S [INFO ] 2024-12-20 04:12:08,833241 [main] LiquidJsTransformerTest - Run 4: PT3.009687375S [To redirect Truffle log output to a file use one of the following options: * '--log.file=<path>' if the option is passed using a guest language launcher. * '-Dpolyglot.log.file=<path>' if the option is passed using the host Java launcher. * Configure logging using the polyglot embedding API.] [engine] WARNING: The polyglot engine uses a fallback runtime that does not support runtime compilation to native code. Execution without runtime compilation will negatively impact the guest application performance. The following cause was found: Libgraal compilation is not available on this JVM. Alternatively, the org.graalvm.compiler:compiler module can be put on the --upgrade-module-path. For more information see: https://www.graalvm.org/latest/reference-manual/embed-languages/#runtime-optimization-support. To disable this warning use the '--engine.WarnInterpreterOnly=false' option or the '-Dpolyglot.engine.WarnInterpreterOnly=false' system property. [INFO ] 2024-12-20 04:12:08,906295 [main] LiquidJsTransformerTest - Run 0: PT0.021998S [INFO ] 2024-12-20 04:12:08,916581 [main] LiquidJsTransformerTest - Run 1: PT0.010198S [INFO ] 2024-12-20 04:12:08,927812 [main] LiquidJsTransformerTest - Run 2: PT0.011137708S [INFO ] 2024-12-20 04:12:08,938391 [main] LiquidJsTransformerTest - Run 3: PT0.010506958S [INFO ] 2024-12-20 04:12:08,948759 [main] LiquidJsTransformerTest - Run 4: PT0.010288125S [INFO ] 2024-12-20 04:12:08,959167 [main] LiquidJsTransformerTest - Run 5: PT0.010324625S Signed-off-by: Greg Schohn <[email protected]>

…transform invocation scripts and rewrite the type mappings sanitization transformer to be pure javascript as opposed to liquid (or jinja). I've tweaked a bulk request test time the accumulated time for 10K warm runs and the average time to transform is around 0.2ms (without '-ea'), which is a major improvement over liquid/jinja scripts. Signed-off-by: Greg Schohn <[email protected]>

testHelperFixtures/build.gradle

Signed-off-by: Andre Kurait <[email protected]>

.../jsonLiquidJSTransformer/src/main/java/org/opensearch/migrations/transform/ScriptRunner.java

Signed-off-by: Andre Kurait <[email protected]>

…ontract Signed-off-by: Andre Kurait <[email protected]>

Signed-off-by: Andre Kurait <[email protected]>

CreateSnapshot/build.gradle

TrafficCapture/build.gradle

...tionPlugins/jsonMessageTransformers/jsonLiquidJSTransformer/src/main/resources/liquid.min.js

...Transformer/src/test/java/org/opensearch/migrations/transform/JavascriptTransformerTest.java

...rmers/jsonTypeMappingsSanitizationTransformer/src/main/resources/js/typeMappingsSanitizer.js

...st/java/org/opensearch/migrations/transform/TypeMappingsSanitizationTransformerBulkTest.java

Signed-off-by: Andre Kurait <[email protected]>

… JsonJSTransformerProvider Signed-off-by: Andre Kurait <[email protected]>

Signed-off-by: Andre Kurait <[email protected]>

… Graalvm compatibility -s Signed-off-by: Andre Kurait <[email protected]>

Signed-off-by: Andre Kurait <[email protected]>

gregschohn

Great work moving this along Andre. Thanks! I'll keep an eye out for future changes.

...st/java/org/opensearch/migrations/transform/TypeMappingsSanitizationTransformerBulkTest.java

gregschohn · 2025-01-13T12:47:52Z

TrafficCapture/dockerSolution/README.md

+The tools in this directory can only be built if you have Java version 17
+installed.


This isn't true, right? Gradle download 17 if it wasn't present - so you really just need whatever the minimum version of java is to run ../../gradlew

Verified this is the case and updated. We need Java 8-17 (until bumping gradle up to 8.4 when we'll gain 21 support)

Performed more verification and found that spotless fails for Java 17 language usage when the host is Java 11. Junit and build downloads and runs as expected when the host is java 8 or 11

Fixed spotless temporarily, will address root cause with https://opensearch.atlassian.net/browse/MIGRATIONS-2344

TrafficCapture/trafficCaptureProxyServer/build.gradle

build.gradle

gregschohn · 2025-01-13T13:14:59Z

...c/test/java/org/opensearch/migrations/transform/TypeMappingsSanitizationCreateIndexTest.java

-            "          \"type\" : {\n" +
-            "            \"type\" : \"keyword\"\n" +
-            "          },\n" +


Why is this no longer expected? Shouldn't we be adding this (or have an option to)? I've seen this conversion in a number of examples.

This wasn't fully implemented. I'd want this as a feature that would also update the documents as well that could be disabled. This can be an extension in the future

Will be addressed with https://opensearch.atlassian.net/browse/MIGRATIONS-2267

gregschohn · 2025-01-13T13:16:41Z

...c/test/java/org/opensearch/migrations/transform/TypeMappingsSanitizationTransformerTest.java

+            "    }\n" +
+            "  }\n" +
+            "}";
+        var rewriter = new TypeMappingsSanitizationTransformer(null, null);


what source version do we default to?

Nothing defined. This would be expected to work if you were migrating from 2.x -> 2.x

...Plugins/jsonMessageTransformers/jsonTypeMappingsSanitizationTransformerProvider/build.gradle

...der/src/test/java/org/opensearch/migrations/replay/TypeMappingsSanitizationProviderTest.java

AndreKurait

Let's wait to merge this until we get license approval for GraalJS UPL License

…tings, and proper thread safe support for IJsonTransformers Signed-off-by: Andre Kurait <[email protected]>

Signed-off-by: Andre Kurait <[email protected]>

License approval granted

gregschohn added 2 commits December 19, 2024 23:14

chelma reviewed Jan 2, 2025

View reviewed changes

testHelperFixtures/build.gradle Outdated Show resolved Hide resolved

Merge branch 'main' into liquidJsTransforms

4816155

Signed-off-by: Andre Kurait <[email protected]>

gregschohn commented Jan 8, 2025

View reviewed changes

.../jsonLiquidJSTransformer/src/main/java/org/opensearch/migrations/transform/ScriptRunner.java Outdated Show resolved Hide resolved

AndreKurait added 3 commits January 9, 2025 12:07

Merge remote-tracking branch 'upstream/main' into liquidJsTransforms

7b4cecd

Signed-off-by: Andre Kurait <[email protected]>

Update javascript transformations to handle Object IJsonTransformer c…

ed2a0a8

…ontract Signed-off-by: Andre Kurait <[email protected]>

Remove LiquidJs due to performance slowdown over Javascript

7e12bac

Signed-off-by: Andre Kurait <[email protected]>

peternied reviewed Jan 9, 2025

View reviewed changes

AndreKurait added 2 commits January 9, 2025 13:10

Create JsonJSTransformerProvider for custom scripts

1fe744a

Signed-off-by: Andre Kurait <[email protected]>

Update TypeMappingSanitizationTransformerProvider to share logic with…

efc4df2

… JsonJSTransformerProvider Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the liquidJsTransforms branch from b6285b3 to b65d7c5 Compare January 9, 2025 21:32

Revert vscode settings

397d72e

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the liquidJsTransforms branch 2 times, most recently from 80b975c to 789e459 Compare January 10, 2025 05:20

Update project (except capture proxy and dependencies) to Java 17 for…

738cf2b

… Graalvm compatibility -s Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the liquidJsTransforms branch from 789e459 to 738cf2b Compare January 10, 2025 05:21

AndreKurait added 2 commits January 9, 2025 23:41

Enable error prone for JDK 17

4091797

Signed-off-by: Andre Kurait <[email protected]>

re-enable task tree

ae06216

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the liquidJsTransforms branch from 9f2f8b6 to ae06216 Compare January 10, 2025 05:47

Update readme and transformation type mapping parameter input

4515ed3

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the liquidJsTransforms branch from 681daa0 to 4515ed3 Compare January 10, 2025 14:30

AndreKurait added 2 commits January 10, 2025 16:05

Fix javascript typemapping sanitization

bc6e289

Signed-off-by: Andre Kurait <[email protected]>

Restrict polyglot host access

a1c3a6a

Signed-off-by: Andre Kurait <[email protected]>

gregschohn commented Jan 13, 2025

View reviewed changes

AndreKurait previously requested changes Jan 13, 2025

View reviewed changes

AndreKurait added 4 commits January 13, 2025 21:43

Refactor javascript transformers with graaljs community, security set…

7be2a63

…tings, and proper thread safe support for IJsonTransformers Signed-off-by: Andre Kurait <[email protected]>

Update Java Version docs

a3fd9d7

Signed-off-by: Andre Kurait <[email protected]>

Add comment on task tree

313d768

Signed-off-by: Andre Kurait <[email protected]>

Retain compatibility for spotless with jdk 11

9a48175

Signed-off-by: Andre Kurait <[email protected]>

Support arbritrary static bindings object

5b6fc2a

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the liquidJsTransforms branch from 14e856d to 5b6fc2a Compare January 14, 2025 19:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade JDK toolchain/target version to 17 and add GraalJS #1207

Upgrade JDK toolchain/target version to 17 and add GraalJS #1207

gregschohn commented Dec 20, 2024 •

edited by AndreKurait

Loading

gregschohn left a comment

gregschohn Jan 13, 2025

AndreKurait Jan 14, 2025

AndreKurait Jan 14, 2025

AndreKurait Jan 14, 2025

gregschohn Jan 13, 2025

AndreKurait Jan 14, 2025

AndreKurait Jan 15, 2025

gregschohn Jan 13, 2025

AndreKurait Jan 14, 2025

AndreKurait left a comment

		The tools in this directory can only be built if you have Java version 17
		installed.

Upgrade JDK toolchain/target version to 17 and add GraalJS #1207

Are you sure you want to change the base?

Upgrade JDK toolchain/target version to 17 and add GraalJS #1207

Conversation

gregschohn commented Dec 20, 2024 • edited by AndreKurait Loading

Description

Issues Resolved

Testing

Check List

gregschohn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndreKurait left a comment

Choose a reason for hiding this comment

gregschohn commented Dec 20, 2024 •

edited by AndreKurait

Loading