Technical Insights

If you are looking to store HDF5 data in the cloud there are several different technologies that can be used and choosing between them can be somewhat confusing. In this post, I thought it would be helpful to cover some of the options with the hope of helping HDF users make the best decision for their deployment. Each project will have its own requirements and special considerations, so please take this as just a starting point....

Accessing large data stores over the internet can be rather slow, but often you can speed things up using multiprocessing—i.e. running multiple processes that divvy up the work needed. Even if you run more processes than you have cores on your computer, since much of the time each process will be waiting on data, in many cases you'll find things speed up nicely....

Interim Engineering Director Dana Robinson talks about The HDF Group's upcoming release schedule. As some of you may already have noticed, we now post the current HDF5 release schedule in the README.md document in the project's root on GitHub. I'll update this as circumstances change so it always reflects our current thinking....

The HDF Group just released HDF5 1.13.1. All of the 1.13 series are experimental releases, which allows us to test new features with our users and get feedback while we are working on the development of the next major maintenance release. Learn more about this new release of HDF5....

The purpose of this introduction is to highlight and celebrate a community contribution the impact of which we are just beginning to understand. Its principal author, Mr. Lucas C. Villa Real, calls it HDF5-UDF and describes it as "a mechanism to generate HDF5 dataset values on-the-fly using user-defined functions (UDFs)." This matter- of-fact characterization is quite accurate, but I would like to provide some context for what this means for us users of HDF5....

Hermes is a distributed I/O buffering system for deep distributed storage hierarchies, which are commonly found on modern HPC systems. On December 1st, 2021, members of the Hermes team gave a presentation to show Hermes in action, talk about the new release and plan for future development. Here are the materials from that session: Slide Deck - Part I - Intro, Overview, and FAQ Slide Deck - Part II - Demonstration https://youtu.be/zDmUdynklJs    ...