Abstract
Hive is a widely-used open-source framework for data warehouse system. Based on the Hadoop execution engine and distributed storage systems, Hive adopts high-level SQL statements to simplify the difficulties for developing big data analytic applications. As more attention has been drawn to optimize the performance of Hive, the performance estimation has been an important role in finding the appropriate parameters. However, since execution time of Hive queries is affected by over hundreds of configurations, resulting in different execution plans and job behaviors, performance prediction becomes more challenging. In this thesis we proposed a time prediction model for optimizing the execution of Hive. Our proposed prediction model is called Oracle, which is a data-driven solution based on deep learning techniques. The prediction also employs recurrent neural networks(RNN) to consider dependencies between stages in a DAG work-flow. We have implemented Oracle with intensive evaluation for TPC-H benchmark queries in different complexity running on the in-house cluster. The experiment results show that Oracle achieved 5.8\% error rate and outperformed three other comparison approaches. Based on prediction models, we can bring about 40% performance improvements without any modification on architecture.