Technical Insights

The HDF5 European workshop, co-organized with ESRF, and sponsored by OpenIO and Omnibond took place on September 17-18, 2019. This event covered the latest HDF5 developments, HDF5 use cases from science and industry, and HDF5 Applications and Tools. This post is an archive of the recorded presentations of this event. ...

Why do we use HDF5? We moved to HDF5 for our simulation data in 2016 from using our own proprietary file format. HDF5 had been on our radar for some time and we spent a couple of years investigating it and other file formats before deciding which we should switch to. HDF5 met all the criteria we had at the time. Amongst the criteria were: performance in speed and size, an accepted standard for scientific data, being open source, providing additional tools....

Question and answers from the HDF5 C++ Webinar on January 24th, 2019. Read the followup questions and answers from presentations on H5CPP from Steven Varga, h5cpp Wrapper from Martin Shetty and Eugen Wintersberger and Ntuple: Tabular Data in HDF5 with C++ from Chris Green and Marc Paterno. ...

Join The HDF Group for a webinar consisting of three short presentations from the HDF5 community to learn about different approaches and exciting work being done by the HDF5 C++ community members. Steven Varga, Martin Shetty and Eugen Wintersberger, and Marc Paterno and Chris Green will share their vision for and experiences using HDF5 with C++....

A few years ago, I was looking for a data format with low latency block and stream support. While protocol buffers offered streams, it was lacking indexed block access. Soon, I realized I was looking for a container with file system-like properties. When I examined HDF5, I found it was very close to what I needed to store massive financial engineering datasets...

Internal compression is one of several powerful HDF5 features that distinguish HDF5 from other binary formats and make it very attractive for storing and organizing data. Internal HDF5 compression saves storage space and I/O bandwidth and allows efficient partial access to data. Chunk storage must be used when HDF5 compression is enabled....

Scot Martin, Harvard University, HDF Guest Blogger HDF5 storage is really interesting. To me, its format has no fixed structure, but instead is based on introspection and discovery. Seems great to me; Mathematica has its origins first in artificial intelligence, so we ought to be able to do something here.  Approaching twenty-two years with Mathematica and almost a “Hello, World!” ability in C, I decided to jump right in. Enter The HDF Group's P/Invoke for my salvation. Here’s how we make use of it in Mathematica: LoadNETAssembly["HDF.PInvoke.dll"] Bang! Ready to go in Mathematica. Here’s a proof of concept for how it works: Module[ (* The three symbols should have initial values so that there is *) (* memory allocation when Mathematica interfaces with P/Invoke. *) {major=0,minor=0,revision=0,return}, CompoundExpression[ (* access...