The HDF5 Subfiling Virtual File Driver (VFD) is an MPI-based File Driver that was introduced in the HDF5 1.14.0 release. This File Driver allows an HDF5 application to create HDF5 files which are distributed across a collection of ”subfiles” in equal-sized data segment ”stripes.” I/O to the logical HDF5 file is directed to the appropriate ”subfile” according to the Subfiling VFD configuration and a system of ”I/O Concentrators.” For more information on the general design of this VFD, refer to the Subfiling RFC.
A user guide for the Subfiling Virtual File Driver was created in conjunction with the ExaIO team. The document gives a brief overview of the Subfiling VFD and details how to build and use the VFD, how to get the best performance when using the VFD, and the current limitations and known problems of the VFD. The document will be integrated into the HDF5 user guide in the future, but will currently reside on GitHub until then.