Sharing Experience on HDF5 VOL Connectors Development and Maintenance – public talk

The HDF Group and partners gave a talk “Sharing Experience on HDF5 VOL Connectors Development and Maintenance” on Wednesday, May 11, 1:00-2:00 PM ET as part of the ECP 2022 Community BOF days . The talk was  weighted toward new HDF5 VOL features, with the following agenda:

  • Introduction
  • VOL framework, VOL toolkit, and Spack
  • VOL connectors and experiences
    • DAOS
    • Asynchronous I/O
    • Cache
    • Log-based
  • Question and Answers

The slide deck is available to view.

HDF5 is a data model, file format, and I/O library that has become a de-facto standard for HPC applications to achieve scalable I/O and for storing and managing big data from computer modeling, large physics experiments, and observations. Designed and optimized for POSIX I/O more than 20 years ago, HDF5 was recently updated to provide access to non-POSIX storage devices via HDF5 Virtual Object Layer (VOL). The HDF5 VOL connectors created by the ECP Data and Vis SDK teams and The HDF Group showed great potential in accessing new storage (e.g., DAOS), and in improving the performance of the HDF5 ECP applications by introducing asynchronous operations into HDF5 or by reorganizing data in HDF5 file to achieve better I/O performance while preserving application semantics.

During the BOF session, HDF5 VOL developers will share their experiences creating and maintaining HDF5 VOL connectors and identifying gaps in HDF5 VOL architectural layer, VOL connector development tools, testing and documentation.

The BOF session is open to anyone interested in sharing their VOL connector development experience. It will also be valuable to anyone who wants to learn about the experience before diving into VOL connector development.

Jordan Henderson (The HDF Group)
Jerome Soumagne (The HDF Group)
Neil Fortner (The HDF Group)
Houjun Tang (LBNL)
Huihuo Zheng (ANL)
Wei-keng Liao (Northwestern University)
Suren Byna (LBNL)
Scot Breitenfeld (The HDF Group)
Larry Knox (The HDF Group)
Kai-yuan Hou (Northwestern University)

No Comments

Leave a Comment