Flexible PHDF5

Preliminary Implementation Design

Albert Cheng
March 6, 2001

Project Requirement

Problem Description

When PHDF5 was first implemented, the implementation requirements include that no threads could be used and no process could be set aside as a server daemon to coordinate file structure (metadata) changes among MPI processes. The restrictions were because threads were not commonly available then. Using threads as part of the implementation would hinder its portability to different platforms. No process was allowed to be set aside because some parallel platforms might allocate compute nodes in multiples of certain numbers, e.g., powers of two. Many parallel algorithms are designed to run with powers of two of compute nodes. Reserving one compute node would conflict with the parallel algorithm. Therefore all HDF5 function calls that involve structural changes (e.g., H5Dcreate) have to be called in the collective manner so that all processes could synchronize their structural information of the file.

This is working for the current implementation but proved to be clumsy for some applications. E.g., an application of 100 processes opens one HDF5 file that contains 100 datasets. Each process, P-i, wants to access dataset, DS-i, only. The current PHDF5 API requires each process to make 100 H5Dopen calls in a collective manner. It also requires all 100 processes to close each dataset via H5Dclose collectively. That makes the programming algorithm clumsy and causes some performance loss because the processes have to work in the collective mode unnecessarily.

Desired Solution

Permit independent datasets creation and open. Each process needs to call H5Dopen only for datasets it wants to access. As in the above example, each process, P-i, after opening the HDF5 file collectively, will make only one independent H5Dopen call gain access to dataset, DS-i. Each of them can close its dataset independently. They still need to close the file collectively.

Implementation Requirements

1. Threads or a set aside process can be used as part of the implementation.
2. The flexible PHDF5 is an optional feature such that it can be excluded during HDF5 library building time due to various reasons such as threads not supported, not willing to use threads. If it is excluded, Datasets access will be back to the collective fashion as current.
3. When the flexible PHDF5 is included, applications can still do collective datasets access as current.

Implementation Design

Two solutions are proposed to manage the file structural changes. One is via the use of Pthreads and the other is via a set aside process (SAP).

Pthreads Option

The PHDF5 is implemented with an option to use PTHREADS. For each HDF5 file opened, one of the processes that open the file spawns an internal thread. The internal thread does not return control to the user application but stays in its own "loops", coordinate all future requests for file structural changes from all the processes that open this file. The processes all returns to the user application with file handle ID. From then on, all processes can issue independent structural changes (e.g., creation of new datasets, extension of dataset sizes,) and the internal thread is there to coordinate all changes. For the above example application, each of the 100 processes will need to make only one independent dataset open call. This will make the programming model more flexible and closer to the original algorithm and may gain some performance because of the independent calls. Notice that this makes the function calls easier to use but does not change the overall I/O performance.

Advantages of the Pthreads Option

• There is no change to the current API specification. The programming model becomes more flexible with this option configured.
• I/O performance should be the same as before.

Disadvantages of the Pthreads Option

• It requires Pthreads library support. Many systems (e.g., SP2, O2K) do but some (e.g., mpich in Tflops) don't.
• If the user application also uses threads, it must use the same Pthreads library or a threads library that is compatible with the Pthreads library PHDF5 uses.
• Requires a thread-safe MPI implementation since the process that spawns the internal thread now has two threads running in its process space. Thread-safe MPI implementation is not universal yet. E.g., the current version of mpich, of which ROMIO is a component, is not thread-safe.

Set Aside Process (SAP) Option

The PHDF5 is implemented with an option to reserve an MPI process for the purpose of coordinating file structural changes for ALL parallel HDF5 files of the application. A new function, say H5Parallel_Init(), is defined. An application must call this function first before making any other parallel HDF5 related function calls. H5Parallel_Init will reserve one of the processes (e.g., process 0 of MPI_COMM_WORLD) as the Set-Aside-Process (SAP) to coordinate structural changes for all parallel HDF5 files. (Notice this is different from the Pthreads option in which one internal thread coordinates one file whereas the SAP does it for all files.) The SAP does not return to the user application space until the function H5Parallel_Close() is called by all other processes of the MPI_COMM_WORLD.

All other processes beside the SAP will create a new sub-communicator (let us name it as H5MPI_COMM_WORLD) and return it to the user application. So, H5MPI_COMM_WORLD consists of one less process than MPI_COMM_WORLD. From then all, the user application uses H5MPI_COMM_WORLD as if it is the MPI_COMM_WORLD until it calls H5Parallel_Close(). After H5Parallel_Close() returns, (including the return of SAP for the call of H5Parallel_Init()), the application may use the MPI_COMM_WORLD again.

Advantages of the SAP option

• It does not require any extra library and can be implemented in all platforms that PHDF5 supports.
• User applications can use threads as long as it is compatible with the MPI library being used.

Disadvantages of the SAP option

• The user application loses the use of the SAP during the use of the PHDF5 library calls. One may use this option only when there are many MPI processes invoked. Thus losing one out of many will be just a small percentage of the overall compute power.
• Two new functions introduced and the user application must use H5MPI_COMM_WORLD in place of MPI_COMM_WORLD between the calls of H5Parallel_Init and H5Parallel_Close. This changes the programming model a little bit. On the other hand, it is common for MPI applications to create sub-communicators anyway.
• The SAP can become a choke point since it has to interact with all the other processes. If the application prefers to use the independent calls, its I/O requests from different processes should be spread out more randomly. If its I/O requests all come around the same time, it is better for this application to use collective PHDF5 calls.
• The SAP needs to hold all structural information of all PHDF5 files currently open. It may run into memory limits. If the SAP option is implemented, a future extension is to allow the designation of multiple SAP's to handle subsets of PHDF5 files opened.
• In some parallel platforms, it may have a requirement that processors are allocated in multiples of some numbers (e.g., powers of two). Losing the use of the SAP will conflict with some parallel algorithms that require powers of two processes.

Conclusion (so far)

Since the Pthreads option takes more software requirements and is less likely to work on as many platforms as the SAP option may, I am in favor to try the SAP option implementation first.