Webinar Materials: Enabling Multithreading Concurrency in HDF5: Community Discussion

This webinar, held on November 13, 2020 was presented by Quincey Koziol.

Abstract: The HDF5 library is now threadsafe, but not concurrent. This talk will present a plan for making the HDF5 library safe for concurrent access by multiple threads. I will outline the current structure of the code, describe the infrastructure changes needed to enable multi-threaded concurrency within the library, and lay out a plan for putting those infrastructure changes to work by making the H5Dread API routine concurrently threadsafe. Community input for making more of the library concurrent is welcome and greatly desired, and I would like to build a task force for contributions to this effort.

Slide Deck

Chat Transcript

00:17:52 Lori Cooper:
00:22:36 Mark Miller: Do we all agree/believe concurrent *READ* is best first case to try?
00:22:57 Vikram Bhamidipati: Yes!
00:23:01 Michael Burkat: +1
00:23:05 Nicholas Knoblauch: +1
00:23:16 Elena Pourmal: +1
00:23:20 Mike Smith: +1
00:23:23 Marc Paterno: +1
00:23:46 Mark Miller: +1 too…I just wanted toask
00:24:07 Samuel Debionne: Concurrent writes in different files would be my first choice
00:25:06 Lori Cooper: or
00:31:59 Mark Miller: Horiz. axis is not same in both, right?
00:32:08 Mark Miller: Same length I mean?
00:36:18 Samuel Debionne: Why not start with a lock at the file level?
00:42:39 Mark Miller: How can a thread declare a *reader* lock if it expects to modify something via side-call?
00:43:01 Mark Miller: Ah, ok, Q answered
00:47:09 Nicholas Knoblauch: Approximately how many global (or almost global) objects (~order of magnitude) are written to in a H5Dread operation?
00:47:52 Gregory Sjaardema: Just to clarify. This will not stay on a branch until finished, but will periodically (based on stability, and capability) be merged into mainline HDF5 production branch/releases
00:48:03 Marc Paterno: Testing alone does not always do well for verifying multithreaded code. Do you have plans to use tools such a code sanitizers?
00:49:51 Gregory Sjaardema: +1
00:51:07 Mark Miller: During the period the library is in transition, THG is free to declare the concurrent capability is *experimental*, correct? Or, is THG on the hook to fully support whatever concurrency is available in each release?
00:52:09 Mark Miller: Thx
00:52:11 James gallagher: Suggestion: Free (for open source) code analysis tools: Coverity, SonarScan. Both are pretty good.
00:52:48 Thomas Braun: I’ve good experience with using on Mac/Linux for detecting multithreading issues.
00:53:43 Thom Caswell: please keep h5py in the loop on this (as we also have a global lock)
00:54:28 William Fawley: For those of us who use HDF5 in Fortran (or Python) codes, is the hope that successful improvements in concurrent multithreading will migrate quickly to the Fortran side of HDF5? Or is this really not an issue at all?
00:54:32 Thomas Braun: I presume you are using pthread as threading library on linux?! What are the plans for (obscure) platforms like Windows?
00:55:31 Thomas Braun: Thanks.
00:55:52 Vikram Bhamidipati: +1 for Fortran support
00:56:06 David Young: When will the lock API documentation be published?
00:56:11 Mark Miller: What’s Fortran…just kidding 😉
01:02:54 Nicholas Knoblauch: What’s the feasiblity of making something like direct chunk read re-entrant in the short to medium term?
01:09:20 Frans van der Have: Is there a way to achieve parallel compression and decompression of large multi-dimensional array data in such a way that the on-disk files remain compatible with existing other software using hdf5? The default zlib compression could be modified to do compression in parallel (see “pigz”) but not decompression.
01:12:01 Mike Smith: @Frans, maybe I’m misinterpretting the question, but are you suggesting something like using pigz as a fliter, like you can use BLOSC?
01:16:07 Lori Cooper:
01:16:24 Lori Cooper: or or
01:16:28 Mark Miller: Thank you Quincey, Lee and THG team and other contributors
01:16:31 Frans van der Have: @Mike Smit: Yes, if one would implement a parallel ZLIB library like pigz is a parallel executable for compression, then at least that part (computationally intensive and not I/O bound) would become faster. Unfortunately the deflate algorithm does not lend itself to parallel decompression, and using a better algorithm could make it less compatible.
01:16:37 Thomas Braun: Thanks.
01:16:43 Mike Smith: Thanks!

No Comments

Leave a Comment