Coordinated Restore at Checkpoint is an OpenJDK feature that provides a fast start and immediate performance for Java applications.
A Java application and JVM are started from an image in a warmed-up form. The image is created from a running Java instance at arbitrary point of time ("checkpoint"). The start from the image ("restore") continues from the point when checkpoint was made.
The restore in general is faster than initialization. After the restore, Java runtime performance is also on-par with the one at the checkpoint. So, after proper warm-up before the checkpoint, restored Java instance is able to deliver the best runtime characteristics immediately.
Coordinated Restore undisruptively introduces new before-checkpoint and after-restore phases in Java application lifecycle. In contrast with uncoordinated checkpoint/restore, coordination allows restored Java applications to behave differently. For example, it is possible to react on changes in execution environment that happened since checkpoint was done.
CRaC implementation creates the checkpoint only if the whole Java instance state can be stored in the image. Resources like open files or sockets are cannot, so it is required to release them when checkpoint is made. CRaC emits notifications for an application to prepare for the checkpoint and return to operating state after restore.
With more Java frameworks and libraries adopting CRaC, applications can benefit with little or no changes in the code. Moreover, the required amount of changes in the resource management code tends to be small, see examples below.
Coordinated Restore is not tied to a particular checkpoint/restore implementation and will able to use existing ones (CRIU, docker checkpoint/restore) and ones yet to be developed.
- Results
- JDK
- Projects with CRaC support
- Proof-of-Concept CRaC support implementation
- User's flow
- Programmer's flow
- Implementation details
CRaC support was implemented in a few frameworks with the following results. The source code can be found in the Projects with CRaC support section.
The environment
- laptop with Intel i7-5500U, 16Gb RAM and SSD.
- Linux kernel 5.7.4-arch1-1
- data was collected in container running
ubuntu:18.04based image - host operating system: archlinux
How to reproduce
To reproduce you need to create a workspace directory and clone along next repositories:
You need to build examples according to the Projects with CRaC support section.
Then run
host$ docker build -t full-bench -f Dockerfile.full-bench utils
host$ docker run -it --privileged -v $HOME:$HOME -v $PWD:$PWD -w $PWD full-bench
cont# JDK=<path/to/jdk> bash ./utils/full-bench.sh collect
...
cont# exit
host$ bash ./utils/full-bench.sh parse
host$ cp *.data docs
host$ make -C docs
Last command regenerates graphs in the docs.
OpenJDK CRaC Project is developed in https://github.com/openjdk/crac.
Latest release can be found in https://crac.github.io/openjdk-builds.
$ sudo tar zxf <jdk>.tar.gz
NOTE: The JDK archive should be extracted with sudo.
When using CRaC, if you see an "Operation not permitted" error, you may have to update your criu permissions with:
sudo chown root:root $JAVA_HOME/lib/criu
sudo chmod u+s $JAVA_HOME/lib/criu
You can just add crac feature at https://micronaut.io/launch!
- Micronaut CRaC Module
- Hikari DataSources coordination
- Redis coordintation
- Build System
- a single command to generate a docker image with CRaC image included
Example: https://github.com/CRaC/example-micronaut
Basic CRaC support is a part of Quarkus since 2.10.0.
Example: https://github.com/CRaC/example-quarkus
Spring Boot provides CRaC support as of version 3.2, for more details see:
- Scale to zero with Spring and Project CRaC
- Spring Boot documentation for Project CRaC support
- Spring Framework documentation for Project CRaC support
Examples:
Proof-of concept CRaC support was implemented in a few third-party frameworks and libraries.
Source code links are below.
Builds can be found in Maven Central under io.github.crac artifact-id.
AWS Libs with CRaC allows implementation of AWS Lambda functions on CRaC JDK.
Artifacts: https://mvnrepository.com/artifact/io.github.crac.com.amazonaws
Example: https://github.com/crac/example-lambda (diff) is a sample Java serverless function on CRaC.
Changes: https://github.com/CRaC/aws-lambda-java-libs/compare/master...crac
CRaC deployment scheme reflects the need to collect data required for Java application initialization and warm-up.
- a Java application (or container) is deployed in the canary environment
- the app processes canary requests that triggers class loading and JIT compilation
- the running application is checkpointed by some mean
- this creates the image of the JVM and application; the image is considered as a part of a new deployment bundle
- the Java application with the image are deployed in the production environment
- the restored Java process uses loaded classes from and JIT code from the immediately
WARNING: next is a proposal phase and is subject to change
Please refer to the Projects with CRaC support section, step-by-step guide or best practices guide to get an application with CRaC support. The rest of the section is written for the spring-boot example.
For the first, Java command line parameter -XX:CRaCCheckpointTo=PATH defines a path to store the image and also allows the java instance to be checkpointed.
By the current implementation, the image is a directory with image files.
The directory will be created if it does not exist, but no parent directories are created.
export JAVA_HOME=./jdk
$JAVA_HOME/bin/java -XX:CRaCCheckpointTo=cr -jar target/example-spring-boot-0.0.1-SNAPSHOT.jar
For the second, in another console: supply canary worload ...
$ curl localhost:8080
Greetings from Spring Boot!
... and make a checkpoint by a jcmd command
$ jcmd target/example-spring-boot-0.0.1-SNAPSHOT.jar JDK.checkpoint
1563568:
Command executed successfully
Due to current jcmd implementation, success is always reported in jcmd output, problems are reported in the console of the application.
Another option to make the checkpoint is to invoke the jdk.crac.Core.checkpointRestore() method (see API).
More options are possible in the future.
For the third, restore the cr image by -XX:CRaCRestoreFrom=PATH option
$JAVA_HOME/bin/java -XX:CRaCRestoreFrom=cr
When running checkpoint and restore on different computers you may seen an error message during restore
You have to specify -XX:CPUFeatures=[...] together with -XX:CRaCCheckpointTo when making a checkpoint file; specified -XX:CRaCRestoreFrom file contains CPU features [...]; missing features of this CPU are [...]
See more details about the CPU Features configuration.
Programs may need to be adjusted for use with Coordinated Restore at Checkpoint.
A step-by-step guide and best practices guide provide information on how to implement the CRaC support in the code.
Another option is to use an existing framework with CRaC support.
No changes required:
- Micronaut: https://github.com/CRaC/example-micronaut
- Quarkus Hello World: https://github.com/CRaC/example-quarkus
- Spring Boot: https://github.com/CRaC/example-spring-boot
With configuration changes:
- Quarkus Super Heroes migration shows a walkthrough for making an existing non-trivial Quarkus application CRaC-able.
The CRaC API is not a part of Java SE specification. We hope that eventually it will be there, until then there are different packages that can be used.
This is the API that is implemented in the CRaC JDK.
Please refer to org.crac if you are looking to add CRaC support to a code that should also work on a regular JDK/JRE.
The package is provided by org.crac compatibility library.
The org.crac is designed to provide smooth CRaC adoption.
Users of the library can build against and use CRaC API on Java runtimes with jdk.crac, javax.crac, or without any implementation.
- In compile-time,
org.cracpackage totally mirrorsjdk.cracandjavax.crac. - In runtime, org.crac uses reflection to detect CRaC implementation.
If the one is available, all requests to
org.cracare passed to the implementation. Otherwise, requests are forwarded to a dummy implementation.
The dummy implementation allows an application to run but not to use CRaC:
- resources can be registered for notification,
- checkpoint request fails with an exception.
Current OpenJDK implementation is based on using the CRIU project to create the image.
CRIU hosts a few changes made to improve CRaC usability.
You can read more about debugging C/R issues in your application in the debug guide.
Sometimes it might be difficult to alter the application to properly coordinate with the checkpoint (e.g. due to a code in a library you cannot modify). As a temporary workaround you can configure file-descriptor policies.