Blog

HDF5 1.13.0: Introducing Experimental Releases

We are excited to announce a new strategy of delivering HDF5 features: Experimental Releases. Experimental releases allow us to get major new features into the hands of our users so that they can test the features and provide feedback before we integrate them into a subsequent maintenance release. Our first experimental release, HDF5 version 1.13.0, is now available and features support for VOL connectors, a GPU direct I/O driver, and access to data in Distributed Asynchronous Object Storage (DAOS).

Moving forward, our release schedule will include both experimental and maintenance releases. We will use the same versioning schema, the set of three integers X.Y.Z, where X is the major version number, Y is the minor version number and Z is the release number as described in the “HDF5 Release Version Numbers” document.

Under this new approach, odd-numbered minor releases (e.g., HDF5 1.13.x) will be considered experimental releases, while even-numbered minor releases (e.g., HDF5 1.10.x, 1.12.x, 1.14.x) are maintenance  releases as you know them now.

Maintenance releases that vary in the release number (i.e., the third number – 1.12.0, 1.12.1, etc.) may only contain bug fixes, performance improvements and new APIs. Changes to the file format, API signatures and public header files are not allowed. A maintenance release is binary compatible with all previous maintenance releases featuring the same minor number in the release version. (e.g., HDF5 1.12.5 will be binary compatible with any of 1.12.0 through 1.12.4).

Experimental releases allow us to test new features with our users and get feedback while we are working on the development of the next major maintenance release. These experimental releases can include substantial updates, such as changes to the file format, API signatures, or public header files. All HDF5 experimental releases will go through the usual rigorous testing and release process, including the generation of binaries. All features will be complete, tested, stable, and integrated in our development branch. Unlike maintenance releases, binary compatibility should not be expected in an experimental release series (e.g., HDF5 1.13.1 is unlikely to be binary compatible with HDF5 1.13.0). While uncommon, users may also encounter forward/backward file format incompatibility when using new features.

Today we released HDF5 version 1.13.0, our first experimental release, which includes the following features:

• An updated virtual object layer (VOL) API that supports DAOS, pass-through, and asynchronous VOL connectors. Click here for the HDF5 VOL connectors registered with The HDF Group and stay tuned for announcements on VOL connectors releases.
  • Note: The new VOL API is not backward compatible with the 1.12.x VOL API. In light of this, we will not be supporting the HDF5 1.12.x VOL API in the future and VOL development should shift to targeting the VOL API in 1.13.0 and later. For many connector authors, the changes will be minimal and involve minor changes to the VOL class structure and parameter handling. For a full discussion of the VOL changes, see the VOL connector authors guide
• Updates to the virtual file driver (VFD) layer that will support VFD plugins, a GPU VFD, and future VFD-based features like VFD SWMR and sub-filing VFD (discussed below).
• Many performance improvements to bring HDF5 1.13.0 more in line with earlier releases of the library.
• A new tool called h5dwalk. The h5dwalk utility provides a parallel alternative to creating and running script based approaches to invoke serial HDF5 tools on a collection of hdf5 files. As a means of invoking parallel instances of a serial tool, the h5dwalk application can accept directories as input arguments. This new tool provides recursive file discovery and filtering to select HDF5 files. The resulting file collection is distributed between MPI ranks and individual files are then selected for input to a user selected application. For more information we refer you to h5dwalk User’s Guide.

Our future experimental releases are:

• HDF5 1.13.1 (targeted release date mid-2022): Includes new “VFD SWMR” feature that will extend the single-writer/multiple-readers I/O pattern support to all HDF5 API calls (not just dataset I/O, as in the current SWMR implementation) and an improved parallel compression feature that addresses known bugs and scalability issues.
• HDF5 1.13.2: Includes the selection I/O feature that passes the complete I/O requests to the VFD layer, the “sub-filing” feature, which distributes I/O across a collection of sub-files and I/O concentrators in parallel HDF5, and the “onion” VFD, which allows users to track modifications to an HDF5 file and access previous states.

Our next maintenance release, HDF5 1.14.0, starts new series of maintenance releases. It is targeted for release in late 2022 and will contain all the major features listed above. At that time we would like to drop support for 1.8, 1.10 and 1.12. series.

To make HDF5 1.14.0 the best for our users, we would appreciate your feedback (both positive and negative) on the HDF5 1.13.x releases and how they perform for your application. Please send your feedback to help@hdfgroup.org.

Thank you!

The HDF Group