An astounding variety of physical phenomena involve fluid flow including breathing and drinking, heating and ventilation, flying and sailing, engine combustion and galaxy formation. Computational fluid dynamics (CFD) uses numerical methods and algorithms to solve and analyze problems that involve fluid flows.
CFD, high-performance computing and big data all go hand-in-hand. For CFD researchers and engineers, it often seems that there is never enough computing resources available to solve the problems at hand. On the one hand, every order of magnitude increase in computing power leads to the ability to solve new CFD problems. However, the same increases in computing power lead to the creation of even more data delivered at faster speeds, furthering the need for even more computing resources.
HDF5 was designed and implemented to support some of the largest CFD applications at US Department of Energy (DOE) labs in the late 1990s. At that time, HDF5 was designed to scale well into the future as machines grew larger and larger. A recent example is a trillion particle cosmology simulation at Los Alamos National Laboratory that ran on a 100,000 cores
The large number of CFD applications and its use in so many different fields has led to a natural proliferation of data formats for handling CFD data. However, this multitude of data formats makes it extremely difficult to share data and to be able to access it with the many outstanding visualization and analysis tools available today.
Many organizations have turned to HDF5 to address this problem. The CFD General Notation System (CGNS) is an international standard for CFD data sponsored by the aerospace industry, vendors, and governments and has been optimized for HDF5. Also, individual projects, such as the Johns Hopkins Turbulence Database, archive and distribute their data in HDF5.