HDF5 User Group (HUG) Meeting Agenda 2023

The aim of high-energy physicists is to discover the secrets of the physical universe. We do this by studying the universe at both the smallest (elementary particle) and largest (cosmological) scales. We try to answer questions like: What is the universe made of? What forces govern it? How did it become the way it is today? Scientists at the Fermi National Accelerator Laboratory (Fermilab) are engaged in investigating answers to all these questions.

The U.S. Energy Information Administration (EIA) develops and runs two large, highly visible energy models for use in the Annual Energy Outlook and International Energy Outlook publications. Each of these modular systems require careful i/o management to support increasingly long run times with increasingly complex outputs. This talk will provide highlights of EIA’s experience and challenges in converting these models to operate off of HDF5.

Sandia Laboratories uses the CGNS Database to define the model input for most analyses that involve structured mesh geometries, ranging from small-scale scoping studies to large-scale multi-physics calculations on hundreds, thousands, or tens of thousands of processors and GPUs.

The complexity of data management in HPC systems stems from the diversity in I/O behavior exhibited by new workloads, multistage workflows, and multitiered storage systems. The HDF5 library is a popular interface to interact with storage systems in HPC workloads. The library manages the complexity of diverse I/O behaviors by providing user-level configurations to optimize the I/O for HPC workloads. The HDF5 library consists of hundreds of properties that provide various I/O optimizations and improve I/O performance for different use cases. However, these configurations are challenging to set by users who lack expertise in HDF5 library internals. We propose a paradigm change through our H5Intent software, where users specify the intent of I/O operations and the software can set various HDF5 properties automatically to optimize the I/O behavior. This work demonstrates several use cases that map user-defined intents to HDF5 configurations to optimize I/O. In this study, we make three observations. First, I/O intents can accurately define HDF5 configurations while managing conflicts and improving I/O performance by up to 22x. Second, I/O intents can be efficiently passed to HDF5 using the Intent Adaptor with a small footprint of 6.84 KB per node for 1000s intents per process. Third, H5Intent vol can dynamically map I/O intents to HDF5 configurations for various I/O behaviors exhibited by our microbenchmark and improve I/O performance by up to 8.8x. In conclusion, the H5Intent software optimizes complex large-scale HPC workloads such as VPIC and BD-CATS by up to 11x better I/O performance on the Lassen supercomputer.

The current parallel I/O stack is complex and difficult to tune due to the interdependencies among multiple factors that impact data movement performance between storage and compute systems. When performance is slower than expected, end-users, developers, and system administrators rely on I/O profiling and tracing information to identify the root causes of inefficiencies. However, there is a gap between the currently available metrics, the potential bottlenecks, and the implementation of optimizations that would mitigate performance slowdowns. This talk discusses Drishti: a novel interactive, user-oriented visualization and analysis framework. Drishti seeks to aid users in pinpointing the root causes of I/O performance problems and providing actionable recommendations for improving performance based on the observed characteristics of an application. We cover its usage in applications that rely on HDF5 to store its data by providing some real study cases and discuss how the community can extend Drishti with additional analysis and visualizations.

HDF5.jl is a Julia package for accessing HDF5 files, using the HDF5 C library maintained by The HDF Group. It provides a simple, high-level interface making it easy to save and load data, as well as a more flexible interface allowing users to take advantage of many of HDF5’s features. Although the HDF5.jl package has been around since 2012, we have recently undertaken some significant changes to improve the modularity and make available newer features. An example of this is the implementation of HDF5 filter and compression plugins in Julia to connect HDF5 with the underlying C libraries directly.

Some of the earliest conversations about the development of HDF5 took place as part of the Data Models and Formats (DMF) effort of the Accelerated Strategic Computing Initiative (ASCI). ASCI-DMF was a tri-lab (LLNL, SNL and LANL) effort to develop a common and flexible scientific data modeling API and file format for the emerging generation of scalable parallel, high performance computing applications starting in the late 1990s. The NCSA HDF team participated in ASCI-DMF efforts beginning with the initial development of HDF5. Early designs of HDF5 benefited from lessons learned and knowledge gained from not only its predecessor, HDF4, but also from several other tri-lab data management technologies such as XFiles, PDB, Aio, Silo, ExodusII, Nemesis, and SAF. We’ll take a walk through this early history and some of the influence it had on the early development of HDF5.

Chunk sizes that are fine for local disk access can lead to poor performance when HDF5 files are accessed in the cloud. On the other hand, it can be impractical to rechunk datasets for large repositories in order to deploy to the cloud. To improve performance for this use case, HSDS has introduced the concept of “hyperchunking” – grouping HDF5 chunks into larger virtual chunks that optimize performance for access over the web. This talk will discuss how this works and show some real world examples.

The REST VOL is once again in active development, providing another way to interact with HDF5 stored on cloud servers. Will include an introduction to basic VOL concepts, overview of how the REST VOL works, and compare and contrast it with other tools like the ROS3 VFD.

As more HDF5 files end up in cloud object stores their producers and data managers need to be aware about the file properties that enable efficient cloud-native data access. “Cloud optimized” means that internal HDF5 file structures support requiring minimal number of requests to read the desired data from a file in a cloud object store. The talk will present what are the features of cloud-optimized HDF5 files and several ways how that can be achieved.

As technology trends increasingly towards the cloud, there has been a mass migration of services and applications to cloud-based platforms. With Google Cloud now offering HPC solutions, HDF5 would also like to provide the ability for users to use and test DAOS in the cloud. With Google’s HPC toolkit being created for ease of use, a blueprint has been created to help users set up a simple environment with HDF5, DAOS, and the DAOS VOL connector.

Zarr is a fairly recent format for multidimensional data arrays specifically targeting storage systems with key-value interface. Some scientific communities interested in implementing scalable cloud-native data analysis are considering Zarr as their chosen data format because of its straightforward implementation in cloud object stores. HDF Group had developed its own cloud-native HDF5 format, called HSDS schema, about the same time as Zarr. Only HDF Group’s developed software, HSDS, currently creates data in the HSDS schema. Since both Zarr and HSDS schema share the same design approach, it would be worthwhile to consider whether Zarr could serve as the cloud-native HDF5 format. The currently developed Zarr version 3 specification introduces the concept of extensions as a way to add more storage features. The goal of this session is to discuss pros and cons of using Zarr v3 to formulate a new cloud-native HDF5 format. Some technical information will be provided with aim to open up discussion among all attendees.

In an era characterized by an exponential surge in data generation across scientific and industrial sectors, efficiently managing and processing extensive volumes of complex HDF5 and NetCDF4 data has become a pressing challenge. It is critical to understand that many analytics tasks demand processing only specific subsets of this data, often involving extractions, aggregations, or more sophisticated operations like re-scaling. Traditional retrieval of data for external processing is viable but not optimal due to speed and cost constraints, limiting researchers in the breadth of content they can effectively analyze.

Machine learning frameworks such as TensorFlow and PyTorch may load the datasets using multiple threads to reduce the overall I/O overhead. Because of the global lock on the HDF5 library, such multi-threaded applications are effectively limited to single thread I/O. While converting the HDF5 library to support multi-threading is a daunting task, there is a feasible path to enhance the HDF5 library and enable multi-threaded VOL connectors that can mitigate the existing bottleneck.

Data provenance, or data lineage, describes the life cycle of data. In scientific workflows on HPC systems, scientists often seek diverse provenance (e.g., origins of data products, usage patterns of datasets). Unfortunately, existing provenance solutions cannot address the challenges in I/O intensive HPC workflows due to their incompatible provenance models and/or system implementations. In this work, we propose a HDF5-friendly provenance framework for scientific data on HPC systems by leveraging the HDF5 provenance Virtual Object Layer (VOL). We evaluated PROV-IO+ on two HPC systems with four workflows from different domains, including: (1) an HDF5-based acoustic sensing workflow; (2) a synthetic workflow adapted from H5Bench; (3) a Graph Neural Network (GNN) workflow; (4) a Large Language Model (LLM) workflow. Our experiments show that PROV-IO+ can address the provenance needs of the domain scientists effectively with reasonable performance.

We are losing the planet’s biodiversity at an unprecedented rate and in many cases, we do not even have the basic numbers. Photographs, taken by field scientists, tourists, automated cameras, and incidental photographers, are the most abundant source of data on wildlife today. AI can turn massive collections of images into high resolution information source about living organisms, enabling scientific inquiry, conservation, and policy decisions. This is our vision of trustworthy AI for biodiversity conservation and for the new scientific field of imageomics.

As new compression libraries become available, they are often offered to the community in the form of shell command-line tools (e.g. gzip, xz, pkzip, 7zip, bzip, pigz) and may or may not come with quality documentation necessary to use the underlying library effectively to, for example, create an HDF5 compression plugin. Next, when a compression plugin for HDF5 is developed using one of these libraries, it isn’t necessarily easy or clear how to measure performance or compare performance with the original tools. In addition, when combined with advanced features of HDF5 such as dataset chunking, chunk caching, partial I/O, fill values, etc, it can be difficult to under understand the performance within the context of HDF5, identify and correct performance issues when they are discovered or even document for users how to avoid performance pitfalls. The HDF5 community needs a compression sandbox which includes raw data sets (e.g. not HDF5 stored data but raw, binary data files) where plugin compression performance information is regularly maintained and compression libraries used in HDF5 plugins can be compared with performance of their baseline, shell command line counterparts. This talk will propose a new GitHub organization, web site, and associated repositories and compression testing workflows to build such a community-focused HDF5 compression testing sandbox and invite participants to begin contributing to it.

In this presentation, I will delve into our recent projects funded by NSF, focusing on the development of lossy compression cyberinfrastructure, encompassing software and user community expansion. We capitalize on HDF5 capabilities, such as compression filters and the Virtual Object Layer (VOL) connectors, to elevate usability. Additionally, I will offer an overview of several studies in which we utilized our lossy compression in tandem with HDF5 to significantly enhance the I/O performance of HPC applications.

What can Open Science (OS) contribute to the present and future of socio-environmental research and knowledge dissemination? Why is this question worth pursuing through a distributed network of STS researchers, OS practitioners, and socio-environmental researchers working with climate-impacted communities?

Accelerating Parallel Write via Deeply Integrating Predictive Lossy Compression with HDF5 – Sian Jin, Indiana University – Slide Deck | Video

Physics, Neutron and X-ray Scattering and Mass Spectrometry. In many use cases, only 0.1% to 10% of gathered data is of interest. HDF5, due to its proven track record and flexibility, remains the data format of choice. As the amount of data produced continues to grow due to higher instrument and detector resolution and sampling rates, there is a clear demand for efficient management of sparse data in HDF5. Adding support for sparse data will simplify data processing software and widen adoption of HDF5.

HDF5 has endured and grown for a quarter century thanks to countless contributions from people, organizations, and applications. We will share our perspective on those 25 years. We will visit some drivers that led to creating HDF5, highlight some early adopters and applications that helped HDF5 to gain acceptance, look at some interesting and fun uses of HDF5, call out the people and organizations that have kept HDF5 alive and healthy for a generation through their guidance and support, and describe the evolution of HDF5 over time.

In the early days of scientific computing (think Fortran era), most application data took the form of large arrays. These are easily stored to any of many different file formats including HDF5. As scientific computing applications have evolved and grown in sophistication and design, large arrays still play a key role but so do highly complex, pointer-linked, multiply nested, data structures involving abstract data types. Any developer confronted with persisting these complex data structures to disk, especially in a way which is machine portable, ultimately seeks a body of code that is simple and easily adaptable as that data structure changes over the life of the application in which it is used. When that goal is pitted against the relatively deep documentation dive developers must take to fully understand what HDF5 is capable of and how best to apply it in any given circumstance, this often leads to the very naïve use of recursive traversal of data structures where tiny HDF5 datasets, groups and attributes are emitted. What makes perfect sense from an HDF5 API perspective turns into a horrendous disaster from an I/O performance perspective. This talk will describe the basic use case, the problems with the naïve approach as well as a much improved approach which though more complex to code is much more I/O performant.

The Open Standard for Particle-Mesh Data (openPMD) is a metadata standard for tabular (particle/dataframe) and structured mesh data in science and engineering. We show the basic components of openPMD, its extensions to specific domains and applications from laser-plasma physics, particle accelerators, light sources, astrophysics to imaging.