DMTCP: Distributed MultiThreaded CheckPointing

DMTCP (Distributed MultiThreaded Checkpointing) is a tool to transparently checkpointing the state of an arbitrary group of programs spread across many machines and connected by sockets. It does not modify the user's program nor the operating system.

Among the applications supported by DMTCP are OpenMPI, MATLAB, python, perl, and many programming languages and shell scripting languages. With the use of TightVNC, it can also checkpoint and restart X-Windows applications, as long as they do not use extensions (e.g.: no OpenGL, no video). Among the Linux features supported by DMTCP are open file descriptors, pipes, sockets, signal handlers, process id and thread id virtualization (ensure old pids and tids continue to work upon restart), ptys, fifos, process group ids, session ids, terminal attributes, and mmap/mprotect (including mmap-based shared memory). See the QUICK-START file of the distribution for further details.

DMTCP does not yet support Infiniband or Myrinet for OpenMPI. This is planned for near term. Additional developers are welcome.

DMTCP is also the basis for URDB, the Universal Reversible Debugger. URDB is still experimental. Nevertheless, it currently adds reversibility to gdb, MATLAB, python (pdb), and perl (perl -d). It also supports reverse expression watchpoints, a form of temporal search within a process lifetime.

For further information, see the DMTCP Sourceforge project page.

The dmtcp manpage contains an overview of commands and usage. Similar information can also be found in here (slightly outdated).

Release 1.1.1 (Nov 13, 2009) is now available, including:

• the DMTCP .tar.gz source, and binary .deb packages for Debian GNU/Linux (both 32 bit x86 and 64 bit x86_65 architectures).

The latest stable version of DMTCP can always be obtained through the sourceforge.net downloads page.

To obtain the most recent (possibly unstable) source from subversion, run the following command:

• svn co https://dmtcp.svn.sourceforge.net/svnroot/dmtcp/trunk dmtcp

DMTCP is completely transparent and can checkpoint unmodified Linux binaries. However, if you wish to call DMTCP from within your checkpointed program, we provide an optional programming interface called DMTCP Aware. To use DMTCP Aware:

• Add dmtcpaware.h and dmtcpware.c (or libdmtcpaware.a) to your project. These files can be found in the /usr/lib/dmtcp/ directory if you are using a binary distribution or in the .../dmtcpaware/ directory of the source distribution.
• Include dmtcpaware.h from your code. You should now have access to the DMTCP Aware API.
• Documentation of the DMTCP Aware API functions is available. To test if your program is running under dmtcp_checkpoint use dmtcpIsEnabled(). Note that if your program is not running under DMTCP, dmtcpIsEnabled() will return false, and all other API functions will return -128 or NULL and have no effect.

• DMTCP: Transparent Checkpointing for Cluster Computations and the Desktop.
  Jason Ansel, Kapil Arya, and Gene Cooperman.
  23rd IEEE International Parallel and Distributed Processing Symposium (IPDPS'09).
  Rome, Italy. May, 2009. Slides. Bibtex.
  

• Transparent User-Level Checkpointing for the Native POSIX Thread Library for Linux.
  Michael Rieker, Jason Ansel, and Gene Cooperman.
  The 2006 International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA'06).
  Las Vegas, NV. Jun, 2006. Bibtex.

URDB (Universal Reversible Debugger) is a reversible debugger for gdb, MATLAB, Python, Perl, and soon others. It is based on DMTCP. A technical report on URDB is also available.

DMTCP and its standalone single-process component MTCP (MultiThreaded CheckPointing) are currently maintained by Jason Ansel, Kapil Arya, Gene Cooperman, Artem Polyakov, Mike Rieker, Praveen Solanki, and Ana Maria Visan. The list of active developers continues to evolve.