A few years ago, I was looking for a data format with low latency block and stream support. While protocol buffers offered streams, it was lacking indexed block access. Soon, I realized I was looking for a container with file system-like properties. When I examined HDF5, I found it was very close to what I needed to store massive financial engineering datasets….
The topic of software citation has been discussed in many forums recently and several major discovery repositories (e.g. zenodo and DataCite) support metadata for software in addition to datasets and other resource types. HDF5 stradles the boundary between the dataset and software worlds. It is most commonly thought of and referred to as a data format, but, as in any case, data written in the HDF formats can not be read without HDF software. So, the answer to the question: is it a format or is it software? is clearly both.
Damaris is a middleware that enriches existing HPC data format libraries (e.g. HDF5) with data aggregation and asynchronous data management capabilities. At the same time, it can be employed for in situ analysis and visualization purposes.
We are pleased to announce the launch of HDF Group’s new Support Portal and the HDF Forum. The new Support Portal, is located at https://portal.hdfgroup.org is the new home of materials previously found at support.hdfgroup.org. The old support will remain online but will no longer be updated. The new HDF Forum can be found at
The HDF Group is pleased to announce Ann Johnson has joined as the new Director of Engineering, reporting to David Pearah, CEO. Ann was most recently the Vice President of Engineering at Reservoir Labs, responsible for global engineering operations, personnel, and project management. Prior to Reservoir Labs, Ann held several executive management positions at SiCortex,
Greetings! The HDF Group is pleased to launch our new website on the evening of Tuesday, October 17th. Our new site features a new design, a new logo, and other improvements—all based on feedback from our users. You’ll find improved and simplified navigation with a better layout focused on the needs of our customers and
By Francesc Alted. He is a freelance consultant and developing author of different open source libraries like PyTables, Blosc, bcolz and numexpr and an experienced programmer in Python and C. Francesc collaborates regularly with the The HDF Group in different projects. We explain our solution for handling big data streams using HDF5 (with a little help
Tobias Weinzierl, Durham University, UK, Sven Köppel, FIAS, Germany, Michael Bader, TUM, Germany, HDF Guest Bloggers ExaHyPE develops a solver engine for hyperbolic differential equations solved on adaptive Cartesian meshes. It supports various HDF5 output formats. Exascale computing is expected to allow scientists and engineers to simulate, and ultimately understand, wave phenomena with unprecedented accuracy
Internal compression is one of several powerful HDF5 features that distinguish HDF5 from other binary formats and make it very attractive for storing and organizing data. Internal HDF5 compression saves storage space and I/O bandwidth and allows efficient partial access to data. Chunk storage must be used when HDF5 compression is enabled.
Scot Martin, Harvard University, HDF Guest Blogger HDF5 storage is really interesting. To me, its format has no fixed structure, but instead is based on introspection and discovery. Seems great to me; Mathematica has its origins first in artificial intelligence, so we ought to be able to do something here. Approaching twenty-two years with Mathematica
