Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrading to Truffle libraries 24.2.0 #12500

Open
wants to merge 18 commits into
base: develop
Choose a base branch
from

Conversation

JaroslavTulach
Copy link
Member

@JaroslavTulach JaroslavTulach commented Mar 14, 2025

Pull Request Description

New version of Truffle libraries and Graal.js and GraalPython implementations is just around corner delivering needed API enhancements like threadAccessDeniedHandler. Let's get ready for the update and check what our CI thinks about it.

Important Notes

Necessary conditions for integration:

  • GraalVM 24.2.0 artifacts appear on Maven central
  • branching for March edition of Enso is done
  • CI is happy with minimal code changes - preferably just version bump

Once we upgrade to 24.2.0 APIs we:

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

  • The documentation has been updated, if necessary.
  • All code follows the
    Scala,
    Java,
  • Unit tests continue to work
  • Benchmark show no (huge) regressions

@JaroslavTulach JaroslavTulach added the CI: Keep up to date Automatically update this PR to the latest develop. label Mar 18, 2025
@JaroslavTulach
Copy link
Member Author

org.graalvm bits for version 24.2.0 are available on Maven central. Let's schedule another build.

@JaroslavTulach JaroslavTulach removed the CI: Keep up to date Automatically update this PR to the latest develop. label Mar 18, 2025
@JaroslavTulach JaroslavTulach added the CI: Clean build required CI runners will be cleaned before and after this PR is built. label Mar 18, 2025
@JaroslavTulach
Copy link
Member Author

JaroslavTulach commented Mar 19, 2025

There is bunch of Postgress failures related to missing setLength on java.lang.StringBuilder. The minimal reproducer is:

polyglot java import java.lang.StringBuilder as Java_String_Builder

main size=0 =
    sb = Java_String_Builder.new
    sb.append "Hello World"
    if size == 0 then sb.toString else
        sb.setLength size
        sb.toString

when invoke without any arguments, it prints Hello World - both in develop version as well as on this PR. However when invoked with argument 5, the execution fails:

sbt:enso> runEngineDistribution --run s.enso 5
Execution finished with an error: No_Such_Method.Error
        at <enso> s.main<arg-2>(s.enso:7:9-25)
        at <enso> s.main(s.enso:6-8)
No_Such_Method.Error

with current PR. In develop it properly prints Hello. Workaround is to call sb.delete instead as done in a7c7e97 - but why did this work in previous version!?

Reproducing with Graal.js

This is a simple reproducer which doesn't involve Enso, but uses Graal.js instead:

        var code = """
        (function(sb, len) {
            sb.setLength(len);
        });
        """;
        var ctx = Context.create("js");
        var fn = ctx.eval("js", code);
        var sb = new StringBuilder("Hello World");
        sb.setLength(10);
        fn.execute(sb, 5);
        System.out.println(sb);

when this code is executed, it fails on fn.execute(sb, 5):

Exception in thread "main" TypeError: invokeMember (setLength) on java.lang.StringBuilder failed due to: Unknown identifier: setLength
	at <js> :anonymous(Unnamed:2:25-41)
	at org.graalvm.polyglot.Value.execute(Value.java:930)
	at cz.xelfi.demo.nosetlength.NoSetLength.main(NoSetLength.java:19)

e.g. sb.setLength(10) in Java code is fine. sb.setLength(5) in JavaScript code is failing. setLength isn't listed in the members of sb!

@JaroslavTulach JaroslavTulach force-pushed the wip/jtulach/UpgradeToTruffle242 branch from a7c7e97 to 58a1550 Compare March 19, 2025 15:34
@JaroslavTulach JaroslavTulach added this to the 2025-Q2 Release milestone Mar 19, 2025
@JaroslavTulach
Copy link
Member Author

JaroslavTulach commented Mar 19, 2025

$ git diff
diff --git test/Base_Tests/src/Semantic/Python_Interop_Spec.enso test/Base_Tests/src/Semantic/Python_Interop_Spec.enso
index d96a1f0e79..25c6fc9bc7 100644
--- test/Base_Tests/src/Semantic/Python_Interop_Spec.enso
+++ test/Base_Tests/src/Semantic/Python_Interop_Spec.enso
@@ -26,7 +26,7 @@ type My_Type
 
     foreign python do_catch self = """
         try:
-            self.do_throw()
+            self.do_throw(self)
         except: # Polyglot exceptions can only be caught with except-all.
             return self.a

@enso-bot
Copy link

enso-bot bot commented Mar 20, 2025

Jaroslav Tulach reports a new STANDUP for yesterday (2025-03-19):

Progress: .

- fixing postgres test failure: https://github.com/enso-org/enso/pull/12500#issuecomment-2735320540
- brought up with GraalVM guys: https://graalvm.slack.com/archives/CNQSB2DHD/p1742363891452099
- investigating Python failures: https://github.com/enso-org/enso/pull/12500#issuecomment-2737563730
- remove duplicate `.so` reported: https://github.com/enso-org/enso/issues/12557
- legal review
- many meetings It should be finished by 2025-03-25.

@JaroslavTulach
Copy link
Member Author

JaroslavTulach commented Mar 20, 2025

GraalPy version 24.2.0

  • the first message to my object: isMemberReadable("do_throw")
  • then readMember("do_throw")
  • we return the Function - without any connection to self
  • we get isExecutable message
  • then execute message - which is "delayed", as self argument is missing

GraalPy version 24.0.0

  • the first message is isMemberInvocable("do_throw")
  • then invokeMember("do_throw")

Consistent readMember . execute

@JaroslavTulach
Copy link
Member Author

Now there is a remaining failure when running the IDE (CCing @farmaazon, just for case):

   pw:browser [pid=8044][out] Old: FrameVariableNames{names=[]} new FrameVariableNames{names=[]} +0ms
  pw:browser [pid=8044][out] 	at org.enso.runtime.compiler/org.enso.compiler.pass.analyse.FramePointerAnalysis$.updateMeta(FramePointerAnalysis.scala:327) +0ms
  pw:browser [pid=8044][out] 	at org.enso.runtime.compiler/org.enso.compiler.pass.analyse.FramePointerAnalysis$.updateSymbolNames(FramePointerAnalysis.scala:99) +0ms
  pw:browser [pid=8044][out] 	at org.enso.runtime.compiler/org.enso.compiler.pass.analyse.FramePointerAnalysis$.$anonfun$maybAttachFrameVariableNames$2(FramePointerAnalysis.scala:258) +0ms
  pw:browser [pid=8044][out] 	at org.enso.runtime.compiler/org.enso.compiler.pass.analyse.FramePointerAnalysis$.$anonfun$maybAttachFrameVariableNames$2$adapted(FramePointerAnalysis.scala:258) +0ms
  pw:browser [pid=8044][out] 	at [email protected]/scala.Option.foreach(Option.scala:437) +0ms
  pw:browser [pid=8044][out] 	at org.enso.runtime.compiler/org.enso.compiler.pass.analyse.FramePointerAnalysis$.maybAttachFrameVariableNames(FramePointerAnalysis.scala:258) +1ms
  pw:browser [pid=8044][out] 	at org.enso.runtime.compiler/org.enso.compiler.pass.analyse.FramePointerAnalysis$.$anonfun$processCallArguments$1(FramePointerAnalysis.scala:250) +0ms
  pw:browser [pid=8044][out] 	at org.enso.runtime.compiler/org.enso.compiler.pass.analyse.FramePointerAnalysis$.$anonfun$processCallArguments$1$adapted(FramePointerAnalysis.scala:245) +0ms
  pw:browser [pid=8044][out] 	at [email protected]/scala.collection.immutable.List.foreach(List.scala:334) +0ms
  pw:browser [pid=8044][out] 	at org.enso.runtime.compiler/org.enso.compiler.pass.analyse.FramePointerAnalysis$.processCallArguments(FramePointerAnalysis.scala:245) +

How can this be influenced by change of Truffle libraries!? This code is executed before any Truffle code runs! Isn't this some unrelated failure?

@enso-bot
Copy link

enso-bot bot commented Mar 21, 2025

Jaroslav Tulach reports a new STANDUP for yesterday (2025-03-20):

Progress: .

mergify bot pushed a commit that referenced this pull request Mar 21, 2025
- create real `VirtualFrame` and use it instead of `null` in the test
- explicitly request `org.graalvm.polyglot` module
- extracted from #12500 to keep the "switch PR" simple
- it is a good idea to do these robustness changes anyway
farmaazon pushed a commit that referenced this pull request Mar 21, 2025
- create real `VirtualFrame` and use it instead of `null` in the test
- explicitly request `org.graalvm.polyglot` module
- extracted from #12500 to keep the "switch PR" simple
- it is a good idea to do these robustness changes anyway
@enso-bot
Copy link

enso-bot bot commented Mar 22, 2025

Jaroslav Tulach reports a new STANDUP for yesterday (2025-03-21):

Progress: .

Discord
Discord is great for playing games and chilling with friends, or even building a worldwide community. Customize your own space to talk, play, and hang out.

@JaroslavTulach
Copy link
Member Author

JaroslavTulach commented Mar 22, 2025

When I execute the Enso GUI & engine, I see the engine stops at:

"job-pool-2" #100 [333859] daemon prio=5 os_prio=0 cpu=190,53ms elapsed=159,60s tid=0x00007880e801c360 nid=333859 waiting on condition  [0x0000788288ced000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
        - parking to wait for  <0x000000063301ae50> (a com.oracle.truffle.api.impl.ThreadLocalHandshake$Barrier)
        at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:221)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:754)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly([email protected]/AbstractQueuedSynchronizer.java:1099)
        at com.oracle.truffle.api.impl.ThreadLocalHandshake$Barrier.await(org.graalvm.truffle/ThreadLocalHandshake.java:273)
        at com.oracle.truffle.api.impl.ThreadLocalHandshake$Handshake.get(org.graalvm.truffle/ThreadLocalHandshake.java:500)
        at com.oracle.truffle.api.impl.ThreadLocalHandshake$Handshake.get(org.graalvm.truffle/ThreadLocalHandshake.java:295)
        at com.oracle.truffle.polyglot.PolyglotLanguageContext.lambda$ensureInitialized$1(org.graalvm.truffle/PolyglotLanguageContext.java:862)
        at com.oracle.truffle.polyglot.PolyglotLanguageContext$$Lambda/0x00007882b0bf0208.apply(org.graalvm.truffle/Unknown Source)
        at com.oracle.truffle.api.impl.ThreadLocalHandshake$TruffleSafepointImpl.setBlockedBoundary(org.graalvm.truffle/ThreadLocalHandshake.java:881)
        at com.oracle.truffle.api.impl.ThreadLocalHandshake$TruffleSafepointImpl.setBlocked(org.graalvm.truffle/ThreadLocalHandshake.java:852)
        at com.oracle.truffle.api.TruffleSafepoint.setBlockedThreadInterruptible(org.graalvm.truffle/TruffleSafepoint.java:338)
        at com.oracle.truffle.polyglot.PolyglotLanguageContext.ensureInitialized(org.graalvm.truffle/PolyglotLanguageContext.java:860)
        at com.oracle.truffle.polyglot.PolyglotContextImpl.getContextInitialized(org.graalvm.truffle/PolyglotContextImpl.java:833)
        at com.oracle.truffle.polyglot.EngineAccessor$EngineImpl.parseForLanguage(org.graalvm.truffle/EngineAccessor.java:284)
        at com.oracle.truffle.api.TruffleLanguage$Env.parseInternal(org.graalvm.truffle/TruffleLanguage.java:2559)
        at org.enso.interpreter.runtime.EnsoContext.parseInternal(org.enso.runtime/EnsoContext.java:980)
        at org.enso.interpreter.node.expression.builtin.text.RegexCompileNode.compile(org.enso.runtime/RegexCompileNode.java:67)
...
...
     at org.enso.interpreter.runtime.state.RunStateNode.doFresh(org.enso.runtime/RunStateNode.java:89)
        at org.enso.interpreter.runtime.state.RunStateNodeGen$Uncached.execute(org.enso.runtime/RunStateNodeGen.java:269)
        at org.enso.interpreter.service.ExecutionService.execute(org.enso.runtime.instrument.common/ExecutionService.java:211)
        at org.enso.interpreter.service.ExecutionService.execute(org.enso.runtime.instrument.common/ExecutionService.java:259)
        at org.enso.interpreter.instrument.job.ProgramExecutionSupport$.executeProgram(org.enso.runtime.instrument.common/ProgramExecutionSupport.scala:166)
  • The execution goes thru the first call to RunStateNode.execute
  • and stops somewhere in processing regular expressions.
    • the source for TRegex is Flavor=ECMAScript/[^/\\]*[/\\]|[^/\\]+/usg
  • waiting for a ThreadLocalHandshake$Barrier
  • the lock 0x000000063301ae50 doesn't seem to be held by any other thread

@enso-bot
Copy link

enso-bot bot commented Mar 24, 2025

Jaroslav Tulach reports a new STANDUP for the last Saturday (2025-03-22):

Progress: .

@JaroslavTulach
Copy link
Member Author

JaroslavTulach commented Mar 24, 2025

When I execute the Enso GUI & engine, I see the engine stops at:

The new Truffle must be confused by Enso threads that are started, but doing nothing. As far as I can guess, it tries to invoke a safepoint action, but our threads, that are not executing (just blocked on a lock) do not allow for doing that. The following patch workarounds the initialization problem:

$ git diff
diff --git engine/runtime-language-epb/src/main/java/org/enso/interpreter/epb/EpbContext.java engine/runtime-language-epb/src/main/java/org/enso/interpreter/epb/EpbContext.java
index c688e12645..8439bd05dd 100644
--- engine/runtime-language-epb/src/main/java/org/enso/interpreter/epb/EpbContext.java
+++ engine/runtime-language-epb/src/main/java/org/enso/interpreter/epb/EpbContext.java
@@ -48,6 +48,12 @@ final class EpbContext {
    * @param preInitializeLanguages comma separated list of languages to immediately initialize
    */
   public void initialize(String preInitializeLanguages) {
+    for (var entry : env.getInternalLanguages().entrySet()) {
+      var l = entry.getValue();
+      if ("regex".equals(l.getId())) {
+        env.initializeLanguage(l);
+      }
+    }
     if (!isInner) {
       if (innerContext == null) {
         innerContext =
diff --git engine/runtime/src/main/java/org/enso/interpreter/runtime/EnsoContext.java engine/runtime/src/main/java/org/enso/interpreter/runtime/EnsoContext.java
index cbdfa11bf8..c7dc3270f0 100644
--- engine/runtime/src/main/java/org/enso/interpreter/runtime/EnsoContext.java
+++ engine/runtime/src/main/java/org/enso/interpreter/runtime/EnsoContext.java
@@ -212,6 +212,7 @@ public final class EnsoContext {
         pkg -> packageRepository.registerMainProjectPackage(pkg.libraryName(), pkg));
 
     var preinit = environment.getOptions().get(RuntimeOptions.PREINITIALIZE_KEY);
+    preinit = "regex";
     if (preinit != null && preinit.length() > 0) {
       var epb = environment.getInternalLanguages().get("epb");
       @SuppressWarnings("unchecked")

by initializing the regex language sooner than the other threads are started. Let's see what CI thinks about such a change: 858f5ab

@enso-bot
Copy link

enso-bot bot commented Mar 25, 2025

Jaroslav Tulach reports a new STANDUP for yesterday (2025-03-24):

Progress: .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
-compiler CI: Clean build required CI runners will be cleaned before and after this PR is built.
Projects
Status: 🔧 Implementation
Development

Successfully merging this pull request may close these issues.

2 participants