Abstract
Benchmarks are important tools to evaluate the efficacy of operating systems, especially when scalability of operating systems is becoming an important issue due to the increasing number of cores. Based on the designers’ understandings, various benchmarks are designed as applications to evaluate the performance of operating systems in different aspects without systematic ways to verify the correctness and workload of the benchmarks. Different from existing heuristic-based methods, we propose a machine-learning-based benchmark characterization method to characterize the workload of benchmarks systematically with the correctness verification to the characterized benchmarks. In particular, the support vector machine is adopted to characterize benchmarks automatically with the training data collected by kernel tracing tools. The capability of the proposed method is evaluated by characterizing the workload of existing popular, well-known benchmarks, and the results are very encouraging. Benchmarks are widely used to evaluate the performance between different machines or software implementation. Operating system designers usually improve an operating system’s performance based on the experimental results of benchmarks. However, the correctness of benchmarks for operating system are seldom be verified systematically. Due to the complexity of an operating system’s kernel, benchmarks’ results are not directly related to the implementation of applications. Those misapprehended improvements could not identify the actual bottlenecks inside an operating system's kernel. In order to characterize workload accurately, we examine the detailed relationships between operating systems and the applications running on them. Based on the observed kernel events, we proposed a novel workload characterization method that uses the support vector machine (SVM) to automatically classify various applications’ impact to operating system's runtime behaviors. The classification could help identifying bottleneck inside an operating system kernel. Therefore, the performance improvement can be done efficiently.