Webinar Followup: Parallel I/O with HDF5 and Performance Tuning Techniques
On June 26, 2020, The HDF Group employee, Scot Breitenfeld presented a webinar called “Parallel I/O with HDF5 and Performance Tuning Techniques.”
On June 26, 2020, The HDF Group employee, Scot Breitenfeld presented a webinar called “Parallel I/O with HDF5 and Performance Tuning Techniques.”
Scheduled for June 26, 2020 11:00 a.m. CDT, this webinar, presented by Scot Breitenfeld is designed for users who have had exposure to HDF5 and MPI I/O and would like to learn about doing parallel I/O with the HDF5 library.
By Francesc Alted. He is a freelance consultant and developing author of different open source libraries like PyTables, Blosc, bcolz and numexpr and an experienced programmer in Python and C. Francesc collaborates regularly with the The HDF Group in different projects. We explain our solution for handling big data streams using HDF5 (with a little help
Internal compression is one of several powerful HDF5 features that distinguish HDF5 from other binary formats and make it very attractive for storing and organizing data. Internal HDF5 compression saves storage space and I/O bandwidth and allows efficient partial access to data. Chunk storage must be used when HDF5 compression is enabled.
The current improvement of using collective I/O to reduce the number of independent processes accessing the file system helped to improve the metadata reads for cgp_open substantially, yielding 100-1000 times faster execution times over the previous implementation.
The workshop program has two main tracks, one on HPC-oriented technologies that support the industry, and one on oil & gas technologies and how they can leverage HPC.