Abstract
Hadoop is a popular computing framework to deliver timely and cost-effective data processing on a large cluster of commodity machines. It relieves the burden of the programmers dealing with distributed programming, and an ecosystem of Big Data solutions have developed around it. However, Hadoop's job execution time can be greatly depending on its runtime configurations and resource selections. Given more than 100 job configuration settings from Hadoop, and diverse resource instance options in a cloud or virtualized computing environment, running Hadoop jobs still requires a substantial amount of expertise and experience. To address this challenge, we applied deep neural network to predict Hadoop job time based on historical execution data, and we proposed optimization methods to reduce job execution time and cost. The results showed that our prediction method achieved almost 90\% of time prediction accuracy and clearly out-performed three other state-of-art regression-based prediction methods. Based on the time prediction, our proposed configuration search method and job scheduling algorithm successfully shorten the execution time of a single Hadoop job by more than 2 times and reduce the execution cost of processing a batch of Hadoop jobs by more than 2.7 times without relying on any human knowledge and intervention.