// CRaC - Coordinated Restore at Checkpoint
Last year I experimented a little bit with the instant restoration of started and warmed up Java programs from disk, beside a couple of other potential use cases for checkpoints. To achieve this, I accessed a rootless build of CRIU directly from Java via its C/RPC-API (using Panama as binding layer). Although it worked surprisingly well, it quickly became clear that a proper implementation would require help from the JVM on a lower level and also an API to coordinate checkpoint/restore events between libraries.
I was pleased to see that there is a decent chance this might actually happen, since a new project with the name CRaC is currently in the voting stage to be officially started as OpenJDK sub-project. Lets take a look at the prototype.
update: CRaC has been approved (OpenJDK project, github).
With a little Help from the JVM
Why would checkpoint/restore benefit from JVM and OpenJDK support? Several reasons. CRIU does not like it when files change between C/R, a simple log file might spoil the fun if a JVM is restored, shut down and then restored again (which will fail). A JVM is also in an excellent position to run heap cleanup and compaction prior to calling CRIU to dump the process to disk. Checkpointing could be also done after driving the JVM into a safe point and making sure that everything stopped.
The CRaC prototype covers all of that already and more:
- CheckpointException is thrown if files or sockets are open at a checkpoint
- a simple API allows coordination with C/R events
- Heap is cleaned, compacted and the checkpoint is made when the JVM reached a safe point
- CRaC handles some JVM produced files automatically (no need to set -
XX:-UsePerfDatafor example) - The
jcmdtool can be used to checkpoint a JVM from a shell - CRIU is bundled in the JDK as a bonus - no need to have it installed
Since CRaC would be potentially part of OpenJDK one day, it could manage the files of JFR repositories automatically, and help with other tasks like the re-seeding SecureRandom instances or updating SSL certificates in future, which would be difficult (or impossible) to achieve as a third party library.
Coordinated Restore at Checkpoint
The API is very simple and somewhat similar to what I wrote for JCRIU, the main difference is that the current implementation does not allow the JVM to continue running after a checkpoint is created (But I don't see why this can't change in future).
Core.checkpointRestore();
serves currently both as checkpoint and program exit. It is also at the same time the entry point for a restore.
Core.getGlobalContext().register(resource);
A global context is used to register resources which will be notified before a checkpoint is created and in reverse order after the process is restored.
Minimal Example
Lets say we have a class CRACTest which can write Strings to a file (like a logger). To coordinate with C/Rs, it would need to close the file before checkpoint and reopen it after restore.
public class CRACTest implements Resource, AutoCloseable {
private OutputStreamWriter writer;
public CRACTest() {
writer = newWriter();
Core.getGlobalContext().register(this); // register as resource
}
...
...
@Override
public void beforeCheckpoint(Context<? extends Resource> context) throws Exception {
System.out.println("resource pre-checkpoint");
writer.close();
writer = null;
}
@Override
public void afterRestore(Context<? extends Resource> context) throws Exception {
System.out.println("resource post-restore");
writer = newWriter();
}
public static void main(String[] args) throws IOException {
System.out.println(Runtime.version());
try (CRACTest writer = new CRACTest()) {
writer.append("hello");
try {
System.out.println("pre-checkpoint PID: "+ProcessHandle.current().pid());
Core.checkpointRestore(); // exit and restore point
System.out.println("post-restore PID: "+ProcessHandle.current().pid());
} catch (CheckpointException | RestoreException ex) {
throw new RuntimeException("C/R failed", ex);
}
writer.append(" there!\n");
}
}
}
start + checkpoint + exit:
$CRaC/bin/java -XX:CRaCCheckpointTo=/tmp/cp -cp target/CRACTest-0.1-SNAPSHOT.jar dev.mbien.CRACTest
14-crac+0-adhoc..crac-jdk
pre-checkpoint PID: 12119
resource pre-checkpoint
restore at checkpoint:
$CRaC/bin/java -XX:CRaCRestoreFrom=/tmp/cp -cp target/CRACTest-0.1-SNAPSHOT.jar dev.mbien.CRACTest
resource post-restore
post-restore PID: 12119
lets see what we wrote to the file:
cat /tmp/test/CRACTest/out.txt
hello there!
restore 3 more times as a test:
./restore.sh
resource post-restore
post-restore PID: 12119
./restore.sh
resource post-restore
post-restore PID: 12119
./restore.sh
resource post-restore
post-restore PID: 12119
cat /tmp/test/CRACTest/out.txt
hello there!
there!
there!
there!
works as expected.
What happens when we leave an io stream open? Lets remove writer.close() from beforeCheckpoint() and attempt to run a fresh instance.
./run.sh
14-crac+0-adhoc..crac-jdk
pre-checkpoint PID: 12431
resource pre-checkpoint
resource post-restore
Exception in thread "main" java.lang.RuntimeException: C/R failed
at dev.mbien.cractest.CRACTest.main(CRACTest.java:72)
Caused by: jdk.crac.CheckpointException
at java.base/jdk.crac.Core.checkpointRestore1(Core.java:134)
at java.base/jdk.crac.Core.checkpointRestore(Core.java:177)
at dev.mbien.cractest.CRACTest.main(CRACTest.java:69)
Suppressed: jdk.crac.impl.CheckpointOpenFileException: /tmp/test/CRACTest/out.txt
at java.base/jdk.crac.Core.translateJVMExceptions(Core.java:76)
at java.base/jdk.crac.Core.checkpointRestore1(Core.java:137)
... 2 more
The JVM will detect and tell us which files are still open before a checkpoint is attempted. In this case no checkpoint is made and the JVM continues. By adding this restriction, CRaC avoids a big list of potential restore failures.
Tool Integration
Checkpoints can be also triggered externally by using the jcmd tool.
jcmd 15119 JDK.checkpoint
15119:
Command executed successfully
Context and Resources
The Context itself implements Resource. This allows hierarchies of custom contexts to be registered to the global context. Since the context of a resource is passed to the beforeCheckpoint and afterRestore methods, it can be used to carry information to assist in C/R of specific resources.
Performance
As demonstrated with JCRIU, restoring initialized and warmed up Java applications can be really fast - CRaC however can be even faster due to the fact that the process image is much more compact. The average time to restore the JVM running this blog from a checkpoint using JCRIU was ~200 ms, while CRaC can restore JVMs in ~50 ms. Although this will depend on the size of the process image and IO read speed.
Potential use-cases beside instant restore
CRaC seems to be concentrating mainly on the use-case of restoring a started and warmed up JVM as fast as possible. This makes of course sense, since why would someone start a JVM in a container, on-demand, when it could have been already started when the container image was built? The purpose of the container is most likely to run business logic, not to start programs.
However, if CRaC would allow programs to continue running after a checkpoint, it would open up many other possibilities. For example:
- time traveling debuggers, stepping backwards to past breakpoints (checkpoints)
- snapshotting of a production JVM to restore and test/inspect it locally, do heap dumps etc
- maybe some niche use-cases of periodic checkpoints and automatic restoration on failure (incremental dumps)
- instantly starting IDEs (although this won't be a small task)
in any case... exciting times :)
Thanks to Anton Kozlov from Azul for immediately fixing a bug I encountered during testing.
- - - sidenotes - - -
jdk14-crac/lib/criu and jdk14-crac/lib/action-script might require cap_sys_ptrace to be set on some systems to not fail during restore.
The rootless mode for CRIU hasn't made it yet into the master branch which means that the JVM or criu has to be run with root privileges for now.
C/R of UI doesn't work at all, since disposing a window will still leave some cached resources behind (opened sockets, file descriptors etc) - but this is another aspect which could be only solved on the JDK level (although this won't be trivial).