Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

jdk24 criu Blocking operation is not allowed in CRIU single thread mode #21111

Open
pshipton opened this issue Feb 11, 2025 · 5 comments · Fixed by ibmruntimes/openj9-openjdk-jdk24#21 · May be fixed by ibmruntimes/openj9-openjdk-jdk24#27
Open

jdk24 criu Blocking operation is not allowed in CRIU single thread mode #21111

pshipton opened this issue Feb 11, 2025 · 5 comments · Fixed by ibmruntimes/openj9-openjdk-jdk24#21 · May be fixed by ibmruntimes/openj9-openjdk-jdk24#27
Assignees
@JasonFengJ9
Labels
comp:vm criu Used to track CRIU snapshot related work jdk24 test failure
Milestone
Java 24 (0.50)

Comments

@pshipton
Copy link
Member

pshipton commented Feb 11, 2025 •
edited
Loading

https://openj9-jenkins.osuosl.org/job/Test_openjdk24_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/13
cmdLineTester_criu_nonPortableRestore_Xtrace_tracepoint_3

23:20:17  Testing: Restore trace options test with -Xtrace before checkpoint - 2
23:20:17  Test start time: 2025/02/11 04:20:17 Coordinated Universal Time
23:20:17  Running command: bash /home/jenkins/workspace/Test_openjdk24_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuScript.sh /home/jenkins/workspace/Test_openjdk24_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk24_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/bin/java " -Xgcpolicy:optthruput  -Xtrace:print={j9jcl.219}" org.openj9.criu.OptionsFileTest TraceOptionsTest2 1
23:20:17  Time spent starting: 3 milliseconds
23:20:20  Time spent executing: 2468 milliseconds
23:20:20  Test result: FAILED
23:20:20  Output from test:
23:20:20   [OUT] start running script
23:20:20   [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
23:20:20   [OUT] export LD_BIND_NOT=on
23:20:20   [OUT] /home/jenkins/workspace/Test_openjdk24_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/jdkbinary/j2sdk-image/bin/java -XX:+EnableCRIUSupport  -Xgcpolicy:optthruput  -Xtrace:print={j9jcl.219} -cp /home/jenkins/workspace/Test_openjdk24_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criu.jar org.openj9.criu.OptionsFileTest TraceOptionsTest2 1
23:20:20   [OUT] 04:20:17.190*0x99200           j9jcl.219      > initializeKnownClasses
23:20:20   [OUT] Pre-checkpoint
23:20:20   [OUT] main: Tue Feb 11 04:20:17 UTC 2025, Performing CRIUSupport.checkpointJVM(), System.currentTimeMillis(): 1739247617454, System.nanoTime(): 13390170114362931
23:20:20   [OUT] JVMDUMP034I User requested Java dump using '/home/jenkins/workspace/Test_openjdk24_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/output_17392433569434/cmdLineTester_criu_nonPortableRestore_Xtrace_tracepoint_3/javacore.20250211.042017.1816772.0001.txt' through CRIUSingleThreadModeJVMCRIUException
23:20:20   [OUT] JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk24_j9_sanity.functional_ppc64le_linux_Nightly_testList_0/aqa-tests/TKG/output_17392433569434/cmdLineTester_criu_nonPortableRestore_Xtrace_tracepoint_3/javacore.20250211.042017.1816772.0001.txt
23:20:20   [OUT] Exception in thread "main" java.lang.BootstrapMethodError: java.lang.invoke.LambdaConversionException: Exception instantiating lambda object
23:20:20   [OUT] 	at java.base/java.security.Provider.<clinit>(Provider.java:1180)
23:20:20   [OUT] 	at java.base/openj9.internal.criu.SecurityProviders.lambda$registerResetCRIUState$0(SecurityProviders.java:51)
23:20:20   [OUT] 	at java.base/openj9.internal.criu.J9InternalCheckpointHookAPI$J9InternalCheckpointHook.runHook(J9InternalCheckpointHookAPI.java:143)
23:20:20   [OUT] 	at java.base/openj9.internal.criu.J9InternalCheckpointHookAPI.runHooks(J9InternalCheckpointHookAPI.java:98)
23:20:20   [OUT] 	at java.base/openj9.internal.criu.J9InternalCheckpointHookAPI.runPreCheckpointHooksSingleThread(J9InternalCheckpointHookAPI.java:107)
23:20:20   [OUT] 	at java.base/openj9.internal.criu.InternalCRIUSupport.checkpointJVMImpl(Native Method)
23:20:20   [OUT] 	at java.base/openj9.internal.criu.InternalCRIUSupport.checkpointJVM(InternalCRIUSupport.java:945)
23:20:20   [OUT] 	at openj9.criu/org.eclipse.openj9.criu.CRIUSupport.checkpointJVM(CRIUSupport.java:525)
23:20:20   [OUT] 	at org.openj9.criu.CRIUTestUtils.checkPointJVM(CRIUTestUtils.java:92)
23:20:20   [OUT] 	at org.openj9.criu.OptionsFileTest.traceOptionsTest2(OptionsFileTest.java:218)
23:20:20   [OUT] 	at org.openj9.criu.OptionsFileTest.main(OptionsFileTest.java:54)
23:20:20   [OUT] Caused by: java.lang.invoke.LambdaConversionException: Exception instantiating lambda object
23:20:20   [OUT] 	at java.base/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:236)
23:20:20   [OUT] 	at java.base/java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:354)
23:20:20   [OUT] 	at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:134)
23:20:20   [OUT] 	at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:316)
23:20:20   [OUT] 	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:281)
23:20:20   [OUT] 	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:271)
23:20:20   [OUT] 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:193)
23:20:20   [OUT] 	... 11 more
23:20:20   [OUT] Caused by: openj9.internal.criu.JVMCheckpointException: Blocking operation is not allowed in CRIU single thread mode.
23:20:20   [OUT] 	at java.base/jdk.internal.ref.PhantomCleanable.<init>(PhantomCleanable.java:77)
23:20:20   [OUT] 	at java.base/jdk.internal.ref.CleanerImpl$PhantomCleanableRef.<init>(CleanerImpl.java:164)
23:20:20   [OUT] 	at java.base/java.lang.ref.Cleaner.register(Cleaner.java:225)
23:20:20   [OUT] 	at java.base/java.lang.invoke.MethodHandleNatives$CallSiteContext.make(MethodHandleNatives.java:92)
23:20:20   [OUT] 	at java.base/java.lang.invoke.CallSite.<init>(CallSite.java:145)
23:20:20   [OUT] 	at java.base/java.lang.invoke.ConstantCallSite.<init>(ConstantCallSite.java:50)
23:20:20   [OUT] 	at java.base/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:229)
23:20:20   [OUT] 	... 17 more
23:20:20   [OUT] initiate restore
23:20:20   [OUT] Error (criu/protobuf.c:72): Unexpected EOF on (empty-image)
23:20:20   [OUT] Removed test output files
23:20:20   [OUT] finished script
23:20:20  >> Success condition was not found: [Output match: Killed]
23:20:20  >> Required condition was found: [Output match: Pre-checkpoint]
23:20:20  >> Success condition was not found: [Output match: terminateRemainingThreads]
23:20:20  >> Success condition was not found: [Output match: Post-checkpoint]
23:20:20  >> Success condition was not found: [Output match: TEST PASSED]
23:20:20  >> Failure condition was not found: [Output match: org.eclipse.openj9.criu.JVMRestoreException]
23:20:20  >> Failure condition was not found: [Output match: TEST FAILED]
23:20:20  >> Failure condition was not found: [Output match: CRIU is not enabled]
23:20:20  >> Failure condition was not found: [Output match: Operation not permitted]
23:20:20  >> Success condition was not found: [Output match: Thread pid mismatch]
23:20:20  >> Success condition was not found: [Output match: do not match expected]
23:20:20  >> Success condition was not found: [Output match: Unable to create a thread:]
23:20:20  >> Failure condition was not found: [Output match: Could not dump the JVM processes, err=-70]
23:20:20  >> Failure condition was found: [Output match: User requested Java dump using]
23:20:20  >> Failure condition was not found: [Output match: AOT load and compilation disabled post restore]

https://openj9-jenkins.osuosl.org/job/Test_openjdk24_j9_sanity.functional_s390x_linux_Nightly_testList_0/13
cmdLineTester_criu_nonPortableRestore_0

01:51:30  Testing: Restore dump options test with no dump options specified before checkpoint and -Xdump:dynamic required
01:51:30  Test start time: 2025/02/11 01:51:29 Eastern Standard Time
01:51:30  Running command: bash /home/jenkins/workspace/Test_openjdk24_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuScript.sh /home/jenkins/workspace/Test_openjdk24_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk24_j9_sanity.functional_s390x_linux_Nightly_testList_0/jdkbinary/j2sdk-image/bin/java " -Xjit -XX:+CRIURestoreNonPortableMode  -Xdump:dynamic" org.openj9.criu.OptionsFileTest dumpOptionsTestRequireDynamic 1 false false
01:51:30  Time spent starting: 18 milliseconds
01:51:35  Time spent executing: 4257 milliseconds
01:51:35  Test result: FAILED
01:51:35  Output from test:
01:51:35   [OUT] start running script
01:51:35   [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
01:51:35   [OUT] export LD_BIND_NOT=on
01:51:35   [OUT] /home/jenkins/workspace/Test_openjdk24_j9_sanity.functional_s390x_linux_Nightly_testList_0/jdkbinary/j2sdk-image/bin/java -XX:+EnableCRIUSupport  -Xjit -XX:+CRIURestoreNonPortableMode  -Xdump:dynamic -cp /home/jenkins/workspace/Test_openjdk24_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criu.jar org.openj9.criu.OptionsFileTest dumpOptionsTestRequireDynamic 1
01:51:35   [OUT] Pre-checkpoint
01:51:35   [OUT] main: Tue Feb 11 01:51:30 EST 2025, Performing CRIUSupport.checkpointJVM(), System.currentTimeMillis(): 1739256691226, System.nanoTime(): 1739256691222832224
01:51:35   [OUT] JVMDUMP034I User requested Java dump using '/home/jenkins/workspace/Test_openjdk24_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/output_17392557039159/cmdLineTester_criu_nonPortableRestore_0/javacore.20250211.015131.3832593.0001.txt' through CRIUSingleThreadModeJVMCRIUException
01:51:35   [OUT] JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk24_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/output_17392557039159/cmdLineTester_criu_nonPortableRestore_0/javacore.20250211.015131.3832593.0001.txt
01:51:35   [OUT] Exception in thread "main" java.lang.BootstrapMethodError: java.lang.invoke.LambdaConversionException: Exception instantiating lambda object
01:51:35   [OUT] 	at java.base/java.security.Provider.<clinit>(Provider.java:1180)
01:51:35   [OUT] 	at java.base/openj9.internal.criu.SecurityProviders.lambda$registerResetCRIUState$0(SecurityProviders.java:51)
01:51:35   [OUT] 	at java.base/openj9.internal.criu.J9InternalCheckpointHookAPI$J9InternalCheckpointHook.runHook(J9InternalCheckpointHookAPI.java:143)
01:51:35   [OUT] 	at java.base/openj9.internal.criu.J9InternalCheckpointHookAPI.runHooks(J9InternalCheckpointHookAPI.java:98)
01:51:35   [OUT] 	at java.base/openj9.internal.criu.J9InternalCheckpointHookAPI.runPreCheckpointHooksSingleThread(J9InternalCheckpointHookAPI.java:107)
01:51:35   [OUT] 	at java.base/openj9.internal.criu.InternalCRIUSupport.checkpointJVMImpl(Native Method)
01:51:35   [OUT] 	at java.base/openj9.internal.criu.InternalCRIUSupport.checkpointJVM(InternalCRIUSupport.java:945)
01:51:35   [OUT] 	at openj9.criu/org.eclipse.openj9.criu.CRIUSupport.checkpointJVM(CRIUSupport.java:525)
01:51:35   [OUT] 	at org.openj9.criu.CRIUTestUtils.checkPointJVM(CRIUTestUtils.java:92)
01:51:35   [OUT] 	at org.openj9.criu.OptionsFileTest.dumpOptionsTestRequireDynamic(OptionsFileTest.java:298)
01:51:35   [OUT] 	at org.openj9.criu.OptionsFileTest.main(OptionsFileTest.java:69)
01:51:35   [OUT] Caused by: java.lang.invoke.LambdaConversionException: Exception instantiating lambda object
01:51:35   [OUT] 	at java.base/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:236)
01:51:35   [OUT] 	at java.base/java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:354)
01:51:35   [OUT] 	at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:134)
01:51:35   [OUT] 	at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:316)
01:51:35   [OUT] 	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:281)
01:51:35   [OUT] 	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:271)
01:51:35   [OUT] 	at java.base/java.lang.invoke.MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:193)
01:51:35   [OUT] 	... 11 more
01:51:35   [OUT] Caused by: openj9.internal.criu.JVMCheckpointException: Blocking operation is not allowed in CRIU single thread mode.
01:51:35   [OUT] 	at java.base/jdk.internal.ref.PhantomCleanable.<init>(PhantomCleanable.java:77)
01:51:35   [OUT] 	at java.base/jdk.internal.ref.CleanerImpl$PhantomCleanableRef.<init>(CleanerImpl.java:164)
01:51:35   [OUT] 	at java.base/java.lang.ref.Cleaner.register(Cleaner.java:225)
01:51:35   [OUT] 	at java.base/java.lang.invoke.MethodHandleNatives$CallSiteContext.make(MethodHandleNatives.java:92)
01:51:35   [OUT] 	at java.base/java.lang.invoke.CallSite.<init>(CallSite.java:145)
01:51:35   [OUT] 	at java.base/java.lang.invoke.ConstantCallSite.<init>(ConstantCallSite.java:50)
01:51:35   [OUT] 	at java.base/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:229)
01:51:35   [OUT] 	... 17 more
01:51:35   [OUT] initiate restore
01:51:35   [OUT] Error (criu/protobuf.c:72): Unexpected EOF on (empty-image)
01:51:35   [OUT] Removed test output files
01:51:35   [OUT] finished script
01:51:35  >> Success condition was not found: [Output match: Killed]
01:51:35  >> Required condition was found: [Output match: Pre-checkpoint]
01:51:35  >> Success condition was not found: [Output match: Processing dump event "vmstop"]
01:51:35  >> Success condition was not found: [Output match: Processing dump event "throw", detail "java/lang/OutOfMemoryError"]
01:51:35  >> Success condition was not found: [Output match: JVM requested Java dump]
01:51:35  >> Success condition was not found: [Output match: Post-checkpoint]
01:51:35  >> Failure condition was not found: [Output match: org.eclipse.openj9.criu.JVMRestoreException]
01:51:35  >> Failure condition was not found: [Output match: Dump option unrecognized: -Xdump:nofailover]
01:51:35  >> Failure condition was not found: [Output match: CRIU is not enabled]
01:51:35  >> Failure condition was not found: [Output match: Operation not permitted]
01:51:35  >> Success condition was not found: [Output match: Thread pid mismatch]
01:51:35  >> Success condition was not found: [Output match: do not match expected]
01:51:35  >> Success condition was not found: [Output match: Unable to create a thread:]
01:51:35  >> Failure condition was not found: [Output match: Could not dump the JVM processes, err=-70]
01:51:35  >> Failure condition was found: [Output match: User requested Java dump using]
01:51:35  >> Failure condition was not found: [Output match: AOT load and compilation disabled post restore]
@pshipton pshipton added comp:vm criu Used to track CRIU snapshot related work jdk24 test failure labels Feb 11, 2025
@pshipton pshipton added this to the Java 24 (0.50) milestone Feb 11, 2025
@pshipton
Copy link
Member Author

@tajila @JasonFengJ9 fyi

@JasonFengJ9 JasonFengJ9 self-assigned this Feb 11, 2025
@JasonFengJ9
Copy link
Member

JasonFengJ9 commented Feb 11, 2025 •
edited
Loading

50x cmdLineTester_criu_nonPortableRestore_0 grinder - https://openj9-jenkins.osuosl.org/job/Grinder/4093/ - 2/50 failed
50x cmdLineTester_criu_nonPortableRestore_Xtrace_tracepoint_3 grinder - https://openj9-jenkins.osuosl.org/job/Grinder/4094/ - infra error

@pshipton
Copy link
Member Author

https://openj9-jenkins.osuosl.org/job/Test_openjdk24_j9_sanity.functional_aarch64_linux_Nightly_testList_0/14

This was referenced Feb 13, 2025
Merged
Merged
@pshipton
Copy link
Member Author

It's still failing.
https://openj9-jenkins.osuosl.org/job/Test_openjdk24_j9_sanity.functional_x86-64_linux_Nightly_testList_1/19

02:44:24   [OUT] Caused by: openj9.internal.criu.JVMCheckpointException: Blocking operation is not allowed in CRIU single thread mode.
02:44:24   [OUT] 	at java.base/jdk.internal.ref.PhantomCleanable.<init>(PhantomCleanable.java:77)
02:44:24   [OUT] 	at java.base/jdk.internal.ref.CleanerImpl$PhantomCleanableRef.<init>(CleanerImpl.java:172)
02:44:24   [OUT] 	at java.base/java.lang.ref.Cleaner.register(Cleaner.java:225)

@pshipton pshipton reopened this Feb 20, 2025
@JasonFengJ9
Copy link
Member

CleanerImpl.CleanableList.insert() is a synchronized method which can't be in the stack when the checkpoint is performed.
@NotCheckpointSafe should be for the caller method of list.insert(this).
Will open a PR.

This was referenced Feb 20, 2025
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JasonFengJ9 JasonFengJ9

Labels
comp:vm criu Used to track CRIU snapshot related work jdk24 test failure
Projects
None yet
Milestone
Java 24 (0.50)
2 participants
@JasonFengJ9 @pshipton