Agenda – HDF5 User Group 2021

This talk will discuss the future of H5Coro and propose the formation of a subset of the HDF5 standard that is optimized for cloud computing. The value of H5Coro is not in the software itself, but in its demonstration that a narrowly focused implementation of the HDF5 standard can achieve an order of magnitude better performance than the existing HDF5 library.

Much work has been done in an effort to bring HDF5 into HEP as a standard tool for data storage and access. Early on we partnered with CMS to investigate Spark as a parallel analysis tool for columnar data stored in HDF5. This was followed up by a laboratory research grant for furthering work in columnar storage of detector data and demonstrated use of parallel processing with MPI. Soon after we had our first adopters of HDF5 “ntuples” within the NOvA experiment as the PandAna framework. This work led to work under the HEP SciDAC project HEP on HPC, where efforts expanded to cover use of Pythia on HPC systems, and parallelization and new features of PandAna. DUNE is in the process of adopting PandAna. The HEP CCE project, utilizing the established expertise, has been evaluating integration of HDF5 with ROOT. Additional projects (e.g. Exa.TrkX) are using HDF5 in the context of machine learning.

This talk will introduce rhdf5 and accompanying software packages as an R interface to working with HDF5 files. rhdf5 provides an interface to much of the HDF5 C-API, as well as simplified wrapper functions for many common operations, and can be a useful tool regardless of the type of data you are working with. I will also introduce some of the specific use cases of HDF5 for biological data analysis and highlight the role rhdf5 plays in the Bioconductor software stack.

A new detector installed at Lawrence Berkeley National Laboratory is capable of generating imaging data at 480 Gbit/s leading to data sets as large as 650 GB generated in 15 seconds. This signal is known to be sparse, and thus we have implemented parallelized post-processing software on an edge-compute platform (also compatible with high performance computing) to compress the data by 10 – 100x using electron even localization. The sparse data is represented using a linear-index encoded electron event representation (EER) where each electron event is a single entry in a list. Each frame will have a different number of events, and we utilize the ability of HDF5 to store “ragged arrays” where one dataset axis can be variable in length. The resulting sparse data fits into the RAM of a typical commercial laptop, but its unique layout is incompatible with existing post-processing codes built upon traditional dense data (i.e. numpy ndarrays). This incompatibility delays and reduces the scientific output from this (and future) detectors due to the need to develop custom processing code and Al/ML algorithms. Finally, most GPUs are incapable of handling 10’s GB sized datasets and further optimization of sparse imaging data using other sparse data types in HDF5 could enable rapid advanced data analysis on GPUs.

An example to optimize the array storage using an analytic cost model is to consolidate and place small arrays on heterogeneous data stores. Two scientific pipelines, detecting supernovae and post-processing computational fluid dynamics simulations, face scalability bottlenecks when processing massive small arrays. One solution is to organize small arrays in one big file. However, storing everything in one file does not fully leverage the heterogeneous data storage capabilities of modern clusters. We implement a system, Henosis, in the HDF5 VOL layer to intercept data accesses and transparently redirects I/O to in-memory Redis object store and TileDB array store.

Specifically, the talk will address the following: giving attributes special consideration; handling very long object names, expressing a fully-qualified HDF5 object with metadata in a one-line rendering; the techniques used to render groups with very large numbers of child objects; how to scroll individual HDF5 groups as well as an entire graph, and finally, a proper  left- and right-justification of an HDF5 graph.In a tightly-specified data environment, standardized UI components can often suffice and offer cost-effective solutions for UI applications.

However, maintaining such canonical order for parallel I/O bears an expensive overhead of inter-process communication to redistribute and reorganize data before accessing the files. One solution to mitigating such cost is to store the data in a log layout and track the location of each data relative to its canonical order. While such idea has shown a significant improvement in recent literature, it breaks the HDF5 file format specification if applied to HDF5.