[WIP] CRaC POC

[danielkec]

Helidon MP on CRaC

Coordinated Restore at Checkpoint

Helidon MP Implicit example on CRaC

examples/crac/README.md

mvn clean package
docker build --build-arg CR_DIR=~/cr -t crac-helloworld . -f Dockerfile.crac
# First time ran, checkpoint is created, stop with Ctrl-C
docker run --privileged -p 7001:7001 --name crac-helloworld crac-helloworld
# Second time starting from checkpoint, stop with Ctrl-C
docker start -i crac-helloworld

[rvansa]

Hi @danielkec , I've played with this a bit to let CRaC checkpoint the webserver after it starts: https://github.com/rvansa/helidon/tree/crac-poc
I've manually checked that I can curl localhost:7001, then execute docker exec -it crac-helloworld jcmd helidon-examples-microprofile-hello-world-implicit.jar JDK.checkpoint and then curl it again. There might be some synchronization pecularities missing, but the code looks like the only synchronization I need is with LoomServer.start()/stop() methods as provided. Let me know if I am wrong.

[rvansa]

Referring to my changes ^: Actually, the case where a stop() (or start()) would be called in between beforeCheckpoint() and afterRestore() might need a bit more love; I am not sure if that would blow up or pass with error log leaving the executors running, but I think it's an invalid sequence of operations. So we might either explicitly throw an error, or keep the lifecycleLock locked until afterRestore().

[danielkec]

Hi, @rvansa thx for cool fix and sorry for the delay. It seems to be working for me, but when I do a little warmup before the snapshot, snapshot fails with:

An exception during a checkpoint operation:
jdk.internal.crac.mirror.CheckpointException
        Suppressed: jdk.internal.crac.mirror.impl.CheckpointOpenResourceException: FD fd=165 type=unknown path=anon_inode:[eventpoll]
                at java.base/jdk.internal.crac.mirror.Core.translateJVMExceptions(Core.java:117)
                at java.base/jdk.internal.crac.mirror.Core.checkpointRestore1(Core.java:188)
                at java.base/jdk.internal.crac.mirror.Core.checkpointRestore(Core.java:286)
                at java.base/jdk.internal.crac.mirror.Core.checkpointRestoreInternal(Core.java:299)
        Suppressed: jdk.internal.crac.mirror.impl.CheckpointOpenResourceException: FD fd=183 type=unknown path=anon_inode:[eventpoll]
                at java.base/jdk.internal.crac.mirror.Core.translateJVMExceptions(Core.java:117)
                at java.base/jdk.internal.crac.mirror.Core.checkpointRestore1(Core.java:188)
                at java.base/jdk.internal.crac.mirror.Core.checkpointRestore(Core.java:286)
                at java.base/jdk.internal.crac.mirror.Core.checkpointRestoreInternal(Core.java:299)

[rvansa]

Looks like there's a native component opening those epoll FD; I wonder why this didn't pop up with a single request for warmup. Native FDs ask for investigation through strace: https://github.com/CRaC/docs/blob/master/debugging.md#file-descriptors-in-native-code
I'll try to reproduce locally.

[rvansa]

Hi @danielkec, I can confirm this is an issue on JDK (CRaC) side. There are some codepaths in sun.nio triggered when the socket is created from a virtual threads, and we did not have test coverage for that case.

[rvansa]

Hi @danielkec , sorry for a delay. The fix should be in the last release on https://www.azul.com/downloads/#downloads-table-zulu - can you try with https://cdn.azul.com/zulu/bin/zulu22.32.17-ca-crac-jdk22.0.2-linux_x64.tar.gz ?

[rvansa]

Any luck with the latest release?

[danielkec]

I can confirm that 22.32.17 fixes the issue