Abstract
As new vulnerabilities on Windows systems are reported endlessly, it is more practical to stop polymorphic malicious code from exploiting these vulnerabilities by building an behavior-based monitor, rather than adopting a signature-based detection system or fixing these vulnerabilities. Many behavior-based monitors have been proposed for Windows systems to serve this purpose. Some of them hook high-level system APIs to detect the suspicious behaviors of code. However, they cannot detect malicious code that directly invokes Native APIs. In this paper, we present a novel security scheme that hooks Native APIs in the kernel mode. This method effectively prevents malicious code calling Native APIs directly. It introduces an average eight percent computation overhead into the system. Analyses and a series of experiments are given in the paper to support our claims. © 2011 IEEE.