Proposed Solution for Goal 2: Allow files renamed and moved ============================================================ 1) Current Implementation Suppose we have a set of files opened, each containing SDS's and we are looking for an SDS within these files. To identify the SDS, we need to know which file stores the specified SDS and where the file is. When a chunked and distributed SDS is created for the first time, it is not difficult to keep track of each of the sub-SDS. To find an existing SDS, the "address" of the SDS must be known. Therefore, whenever a chunked and distributed SDS is created for the first time, the chunking information of the chunked and distributed SDS is stored in MHF. The chunking information includes the "address" of the sub-SDS represented by pair and the dimension size, chunk name and origin of each sub-SDS. To find a sub-SDS, MHP does a table lookup and gets the "address" represented by of the specified sub-SDS and knows that the sub-SDS can be found in the file with name filename at the machine with name hostname. 2) Limitation & Problem with the Current Implementation 2.1 Limitation This approach is restrictive in that for an application to access a set of existing files, it must have the same configuration as the one that the files were created the first time. This means that the file name can never be changed once it is created and if a file was created by a process running on machine M for the first time, then in order to access the SDS's in the file, the process which opens the file must run on machine M. 2.2 Problem The problem with this approach is that it doesn't allow machine hostnames or file names to be changed because this would cause the pair stored in MHF to be invalid. To show the problem with an example, assume an SDS is stored in file F1 at machine M and the pair is stored in MHF F0. For some reason, F1 is renamed to F2. The info stored in F0 is no longer valid because the SDS is stored in F2 now. 3) Proposed Solution The proposed solution is to assign an integer, FileId, to a file when the file is created for the first time. When n files are created by DSDstart, the number 0 .. n-1 is stored in each of them, respectively. Whenever a SDS is created, MHP stores the integer i as the "address" of the SDS in MHF, where i is the FileId of the file which owns the SDS. To find a SDS, MHP does a table look up and gets the FileId instead of the filename and hostname pair. Since we don't use hostname and file name to find an SDS, the above problem is solved. To be more specific, let's first see how DSDstart works. When a CP (Compute Process) wants to open a set of HDF files, it specifies which processes are MHP and AHP's and which files are MHF and AHF's and then calls DSDstart. Then the MHP and AHP's open the MHF and AHF's, respectively, by calling SDstart. After performing SDstart, each AHP sends the status of SDstart to MHP. If all AHP's open AHF's successfully, MHP sends the file id of MHF to the requesting CP. Otherwise, MHP sends FAIL to the requesting CP. When an AHP opens an existing AHF, it gets the FileId of the AHF. The AHP registers the FileId of the AHF at the MHP by including the FileId in the message in which AHP reports the status of SDstart to the MHP. When a CP wants to access an SDS, it sends the name of the SDS to the MHP. The MHP finds the FileId of the file where the SDS is stored in the chunking table, then forwards the request to the AHP which owns the SDS. To show that this approach works, reconsider the above example. Assume the FileId for F1 is 100. Instead of storing , FileId 100 is stored in F0. (This means that the SDS is stored in the file with FileId 100.) If F1 is renamed to F2, F2 becomes the file with FileId 100. 4) Implementation Design 4.1 File_Info and SetId Besides FileId, HDF files need to store a table called File_Info and a string SetId to make the design complete. 4.1.1 File_Info o What's File_Info table? A File_Info table is a table which has fields of FileId, filename, and hostname. o Why File_Info? With the proposed solution, there is no symbolic information about files. The only information that users can get from MHF by using HDF dumper is the metadata of the SDS (dim size, rank, etc ) and where the SDS is (for example, the SDS is stored in the file with MachID 16.) It is hard to figure out the file name of the file with FileId 16. With the File_Info table, the file name with FileId 16 can be found easily. 4.1.2 SetId o What is SetId? SetId is a string generated by the DSD library for a group of HDF files. A group of HDF files has one MHF and one or more AHF's. MHF and AHF's in the same group have the same SetId. o Why SetId? SetId is used to detect whether a group of HDF files provided by users is valid or not. A group of HDF files should have the matching SetId. 4.2 HDFNOW file structure intn FileId o an integer assigned by DSDstart when files are created. o stored as an attribute of the file. o the name of the file attribute is DHDF_FileId. o FileId 0 is reserved for MHF. char SetId[255] o a string generated by MHP during DSDstart. o stored as an attribute of the file. o the name of the file attribute is DHDF_SetId. o MHF and AHF's have the same SetId. o a concatenated string of time stamp, name of MHF and process rank. vdata File_Info o the class name of vdata is DHDF_FileInfo. o a table with fields FileId, filename, and hostname. __ FileId | MHF --|-- SetId | |-- Files_Info | |-- for each SDS --- metadata of SDS | (name, dim size, data type, and rank) | |- chunking info (class name = Distributed_SDS_Chunk_Table) (FileId, chunkname, dimension size, and origins) __ FileId | AHF --|-- SetId | |-- Files_Info | |__ for each SDS -- metadata and values of sub-SDS 4.2 An Example Assumption: o three files, mhf.hdf, ahf1.hdf, and ahf2.hdf. o a 100x100 array is divided into two subarrays, chunk1_40x100 and chunk2_60x100. 4.2.1 mhf.hdf FileId = 0 SetId = "mhf.hdf" ------------------------------------ | FileId | filename | hostname | ------------------------------------ | 0 | mhf.hdf | H0 | ------------------------------------ | 1 | ahf1.hdf | H1 | ------------------------------------ | 2 | ahf2.hdf | H2 | ------------------------------------ SDS : sdsname = "array_100x100" // metadata of array_100x100 datatype = int rank = 2 dimsize = 100x100 Vdata : --------------------------------------------------- | FileId | chunkname | dimsize | origin | --------------------------------------------------- | 1 | chunk1_40x100 | 40x100 | (0,0) | --------------------------------------------------- | 2 | chunk2_60x100 | 60x100 | (40,0) | --------------------------------------------------- 4.2.2 ahf1.hdf and ahf2.hdf o ahf1.hdf FileId = 1 SetId = "mhf.hdf" -------------------------------------- | FileId | filename | hostname | -------------------------------------- | 0 | mhf.hdf | H0 | -------------------------------------- | 1 | ahf1.hdf | H1 | -------------------------------------- sdsname = chunk1_40x100 datatype = int rank = 2 dimsizes = 40x100 Data : 0 1 2 3 4 .... o ahf2.hdf FileId = 2 SetId = "mhf.hdf" -------------------------------------- | FileId | filename | hostname | -------------------------------------- | 0 | mhf.hdf | H0 | -------------------------------------- | 2 | ahf2.hdf | H2 | -------------------------------------- sdsname = chunk2_60x100 datatype = int rank = 2 dimsizes = 60x100 Data : 6000 6001 .... 4.3 Implementation for the Proposed Solution o The major change is in the data structures. - the structure of HDF files. - the internal data structure used by MHP and AHP. o The protocols for DSD routines are the same as before. o The DSD routine interface is the same as before.