Abstract
Noncontiguous data access is a very common access pattern in many scientific applications. Using POSIX I/O to access many pieces of noncontiguous data segments will generate a lot of amount of I/O requests that cause the I/O system perform poorly. Tow-phase I/O, also called collective I/O, applied by MPI-IO provides a good method to optimize noncontiguous I/O operations. In two-phase I/O, the collections of independent I/O operations that make up the collective operation are analyzed to determine what data segments must be redistributed and transferred. This redistribution needs some information been calculated in advance and lots of data segments are transmitted via network more than once. With I/O-intensive applications, the aggregate size of these data segments being redistributed becomes significant large. This situation will consume much more CPU time to compute redistributing information and network resources to send the data segments. This additional overhead will degrade the system performance. In this thesis, we extend the collective I/O method and propose a new I/O scheme to avoid re-transmission of data segments by applying RDMA Gather/Scatter operations supported by InfiniBand hardware. We also extend the “view” and “datatype” concepts of MPI into the file system to help complete our design. The experiments show that the method we design improves the performance and is more efficient than the collective I/O approach.