Program analysis—the automated analysis of program behavior—has been a fundamental tool for security researchers, having been applied to a multitude of problems ranging from vulnerability discovery to malware detection. Program analysis approaches can classified as either static or dynamic, depending on whether the program is executed during the course of the analysis. For instance, a typical application of static analysis used during program compilation is type checking, where a compiler statically determines the values program fragments can compute and guarantees that a program conforms to a given type system. On the other hand, a dynamic analyzer such as Valgrind might execute a program and monitor heap allocations for evidence of memory leaks or other misbehavior.

Static analysis relies on abstraction of the program under consideration to a model of that program. Typically, static analyses are sound—that is, any property that is true of the model is also true of the original program. On the other hand, static analyses are generally incomplete, which is to say that not every property that is true of the model is true of the original program. Static analysis has been proven to be undecidable in general; however, useful approximate results can often be obtained depending on the complexity of the particular property to be verified.

In contrast to static approaches, dynamic analysis has the desirable property that complex properties can be verified over a subset of the possible paths and inputs to a program that are observed during testing. However, dynamic analyses cannot reason about paths that have not been explored. Therefore, dynamic analyses critically depend on the quality of test inputs used in order to provide adequate path coverage of the program under test.

Static Program Analysis

My research has explored the use of both static and dynamic techniques to verify program security properties. On the static side, my work has studied how static analysis can be used to accurately disassemble obfuscated binaries often encountered during the analysis of packed malware. I have also studied the use of static analysis to automatically derive whitelists of legitimate kernel module behavior in order to detect kernel-level rootkits at load time before they have the chance to execute on a system. Another application of static analysis I have studied involved the automated generation of mimicry attacks against host-based intrusion detection systems that utilize a program control flow graph as a model of legitimate program behavior. Finally, I have studied how statically-derived control flow graphs can be used to reliably detect polymorphic worm variants for use in network-based intrusion detection.

Dynamic Program Analysis

Several of my research projects have also involved the application of dynamic techniques. For instance, my work has explored the use of dynamic analysis to automatically derive the set of detection contraints used in closed source intrusion detection systems. A subsequent study examined how such information could be used to drive the automatic generation of network exploits using dynamic data flow analysis of closed source intrusion detection systems.