Parallel I/O with HDF5 and Performance Tuning Techniques – Exascale Computing Project
This presentation was given at the 2022 Exascale Computing Project Annual meeting, as part of the tutorial “Using HDF5 Efficiently on HPC Systems.”
This presentation was given at the 2022 Exascale Computing Project Annual meeting, as part of the tutorial “Using HDF5 Efficiently on HPC Systems.”
We are very pleased to announce the release of HDF5 1.12.2, which can now be obtained from the HDF5 Download page.
Champaign, IL – The HDF Group announced its Board of Directors has appointed Gerd Heber as its new Executive Director. The HDF Group is a software company dedicated to creating and supporting technologies to address many of today’s big data challenges.
Dr. Heber, who has been The HDF Group’s Applications Architect since 2010, replaces Mike Folk upon his retirement. Folk will remain a member of the Board of Directors.
We are very pleased to announce the release of HDF5-1.13.1, which can now be obtained from the HDF5 Download page.
The HDF Group just released HDF5 1.13.1. All of the 1.13 series are experimental releases, which allows us to test new features with our users and get feedback while we are working on the development of the next major maintenance release. Learn more about this new release of HDF5.
Learn more about the basics needed to construct a simple terminal VOL connector, including mapping HDF5 API calls to alternative storage and setting up VOL fields and callbacks to support this. At the end of the this tutorial, viewers should be able to get started writing their own terminal VOL connectors.
The purpose of this introduction is to highlight and celebrate a community contribution the impact of which we are just beginning to understand. Its principal author, Mr. Lucas C. Villa Real, calls it HDF5-UDF and describes it as “a mechanism to generate HDF5 dataset values on-the-fly using user-defined functions (UDFs).” This matter- of-fact characterization is quite accurate, but I would like to provide some context for what this means for us users of HDF5.
The HDF Group’s formal comment to a DOE request on stewardship for scientific and high-performance computing was published in the Federal Register. We had a joint position paper at January’s ASCR Workshop on Visualization for Scientific Discovery, Decision-Making, & Communication. The paper is called Whither Visualization Logic and was written by Leigh Orf (University of Wisconsin), Lucas Villa Real
The group at BioSimulations.org has been doing some very interesting work using HSDS on Kubernetes to store biomodelling data and visualizing the results using Vega as described in the paper below. Biosimulations chose to use HSDS due to its support for very large data sets, REST API (for use with web applications), and its ability to run on Google Cloud as well as on-premise installations.
Hermes is a distributed I/O buffering system for deep distributed storage hierarchies, which are commonly found on modern HPC systems. On December 1st, 2021, members of the Hermes team gave a presentation to show Hermes in action, talk about the new release and plan for future development. Here are the materials from that session: Slide