How do you set up HDF5 so only one MPI rank 0 process does I/O?
-
Several scientific HDF5 applications use this approach, and we know it
works very well. You should use the sequential HDF5 library.
Pros: one HDF5 file
Cons: Probably a lot of communications will be going on.
How do you configure HDF5 to create separate files for each compute node in a cluster?
- If you use the Parallel HDF5 library, open a communicator for each node
and add just one process to that communicator.
- You may also try to use the sequential library, but then you have to make
sure that only a particular process creates/modifies/closes the file
that is written to. This approach has not been tested.
Pros: should be faster than 1)
Cons: multiple files to handle, not testedAlso there can be other approaches if you decide to go with multiple files. For example, each process writes a flat binary file and one can use the HDF5 external datasets storage feature to create a wrapper HDF5 file to combine all data. See the section on Dataset Creation Property Lists under Property Lists in the HDF5 Tutorial.
- - Last modified:May 16th 2011