Abstract
Scientific experiments, observations, and large-scale simulations generate massive amounts of data. The size of these datasets typically ranges from hundreds of gigabytes to tens of petabytes. Therefore, indexing technique has become an essential tool which enables scientists to directly access the most relevant data records instead of shifting through the whole dataset. In recent years, many data management tools or techniques have been made to accelerate the data access process, including ADIOS, SciDB, and FastBit. However, the time and space required for building and storing these indexes are often too expensive. In this thesis, we propose a light-weight indexing technique called "block index", which exploits the I/O characteristics of storage systems to significantly reduce index size and index building time without sacrificing query performance. After investigating the challenges and benefits of using block index technique, we further develop three optimization techniques to improve query performance. All these techniques are driven by our extensive effort in characterizing and modeling real scientific datasets and HPC I/O systems. As a result, our optimizations significantly improve query performance by up to a factor of 2.3 comparing to the original block index implementation.