Abstract
Worms and Exploits attacks are currently the most prevalent security problems; they are responsible for over half of the CERT advisories issued in the last three years. Based on the principles, Worms and Exploits are very similar. In order to initiate an infection or intrusion, both of them inject a small size of binary code into software applications through buffer overflow or heap overflow vulnerabilities. They are differentiated by their behavior after entering the system. For example, worms will automatically try to find and infect other hosts that have the same vulnerability continuously; however, exploits focus on creating an interface such as connect cmd.exe to one of ports, by connect to this port hackers will get a “Command Line Shell”. In this thesis, we proposed a performance effective API monitoring system to get rid of Worms and Exploits attacks. The Microsoft Windows users in this system can defeat these attacks without hardware support. This mechanism focuses on how shellcodes (the hooks of worms or exploits) work on Windows NT series system (including Windows 2000, Windows XP and Windows 2003 server) and the methods that can be used to stop them. Both of them inject a small size of binary code into the target software process. These small size binary codes are called ShellCode and their sizes are about several hundred bytes. ShellCode has unique structures that are essential for it to utilize system calls to interact with target operating system. Its structure and functioning methods vary among Unix-like and Microsoft Windows series operating system. For instance, ShellCode can use system calls directly in Unix-like operating systems, but it needs to go through more complex steps to acquire Win32 API calls, which are needed from DLL-Dynamic Load Library in Microsoft Windows. These differences make direct impact on the size of ShellCode. In general, for ShellCode that runs on Unix-like operating system, the size is about 20 bytes to 100 bytes. Conversely, it is about 200 bytes to 1000 bytes for ShellCode running on Microsoft Windows system. Despite the difference in ShellCode size, Windows ShellCode also acts in a more complicated way than the Unix ones. Since Shellcode requires more steps to get address of system’s Win32 API Calls to work on Microsoft Windows platform, we have found some ways specifically for Microsoft Windows system to stop them. Through the API monitoring system we purposed, we can stop the attacks made by worms and exploits to vulnerable software processes successfully by making Shellcode run incorrectly on windows system. In addition, the efficiency of Win32 API Calls hooking and monitoring system can be improved. Through the mechanism, our incapability to disassemble and analysis the protected software processes can be overcame as well.