Abstract
Nowadays, there is a growing trend of attacks using vulnerabilities to compromise victim's computer system. Attackers often write shellcodes through these vulnerabilities to compromise the victim's system. These shellcodes are a kind of "harmful code", and use the specific Windows APIs to destroy the system or disclose sensitive data. However, the shellcodes should get the APIs’ addresses before they use it. In this thesis, we proposed an effective system using API hooking and finite state machine to detect shellcodes. Our system detects the shellcodes by monitoring the Windows APIs detect the shellcodes through the API hooks. We use a finite state machine (FSM) to evaluate the behavior of shellcodes. In Recoding Phase, the FSM stores the feature codes and compare in Monitor Phase. If the feature codes mismatch, our system detects shellcodes. Furthermore, our system is low cost and convenient since it does not need hardware support and source code of programs.