Automated Detection of Stealth Attacks on the Operating System Kernel
September 28, 2009
Speaker: Arati Baliga, Rutgers
Time: Monday, September 28, 2PM
Location: Babbio 221
Host: Vivek Pathak
The operating system kernel is implicitly trusted by applications running on a computer system. An attack on the operating system kernel that alters its state is critical because it puts all applications at risk. A compromised system can be stealthily exploited by the attackers, in several ways, such as exfiltration of sensitive information, wasteful usage of the system's resources, adversely affecting system performance or involving it in fraudulent or malicious activities without the user's knowledge or permission. The lack of appropriate detection tools allows such systems to lie within the attackers' control for indefinite periods of time.
Stealth attacks on the kernel are carried out by malware commonly known as rootkits. Though rootkits have considerably increased in sophistication over the past few years, their primary purpose is to conceal the presence of the attacker and therefore, focus on hiding user level objects. In this talk, I will present a new class of stealth attacks on the kernel that we have identified, which do not attempt to hide objects but are inherently stealthy by design. They achieve their malicious objectives by solely modifying data within the kernel. I will also describe an automated technique that can be used for detection of such stealthy data-centric attacks. The key idea behind this technique is to automatically identify and extract invariants exhibited by kernel data structures during a training phase on a clean kernel. The hypothesis is that rootkits that manipulate kernel data violate some of
these invariants and therefore, can be detected. These inferred invariants are then used as specifications of data structure integrity and are enforced during runtime.
Arati Baliga is a Research Associate at the Wireless Information Network Laboratory (WINLAB), Rutgers University. Her current research includes improving the security and reliability of application programs using transactional memory and securing cognitive radio networks. She completed her Ph.D in January 2009 from the department of Computer Science at Rutgers. Her research interests span system security, security in wireless and emerging networks, operating systems and distributed systems.