Network-based Buffer Overflow Attack Detection
Buffer overflow attack is the main attack method that most if not all existing malicious worms use to propagate themselves from machine to machine. Many solutions to the buffer overflow attack problem have been proposed in the last decade, including compiler transformation approaches that detect and/or prevent tampering of control-sensitive data structures, library rewriting approaches that ensure incoming traffic never steps beyond the corresponding receiving buffer's bound, and operating system approaches that prevent malicious code injected by buffer overflow attacks from being executed. In theory, these efforts have largely successfully solved the buffer overflow attack problem. In practice, however, new buffer overflow vulnerabilities are still discovered and reported on a routine basis. This discrepancy between theory and practice arises because almost all existing solutions to the buffer overflow attack problem require substantial modification to the computing infrastructure in which network applications are developed or executed, and thus have met substantial resistance in actual deployment. One way to overcome this deployment problem is to develop a network-based buffer overflow attack detection mechanism that can detect arbitrary buffer overflow attacks without requiring any changes to the network applications or the hosts they run on.
Existing network-based intrusion detection systems (NIDSs) compare incoming packets against an attack signature database, and raise an alert when one or multiple matches are found. Typically, a separate signature is created for each distinct buffer overflow attack. Obviously, this approach cannot effectively detect zero-day attacks, whose signature is unavailable by definition, or variants of known attacks. Moreover, under this approach, false positives are inevitable and tend to be numerous, mainly because the signature matching logic in NIDSs rarely takes into account the context in which buffer overflow attacks take place.
This project develops a network-based buffer overflow attack detection system called Nebula (NEtwork-based BUffer overfLow Attack detection), which can detect both known and zero-day buffer overflow attacks based solely on the traffic observed without requiring any modifications to the end hosts. Moreover, instead of deriving a specific signature for each individual buffer overflow attack instance, Nebula uses a generalized signature that can capture all known variants of buffer overflow attacks while reducing the number of false positives to a negligible level. The key finding of this project is that there is a generic signature in buffer overflow attack: the attacker will use the injected content on the stack and there will be stack addresses in the attack traffic. By detecting the stack address on the network, buffer overflow exploits can be detected.
Furthermore, the following techniques are adopted to decrease false negatives and false positives:
l A transparent TCP proxy is used to defeats NIDS evasion techniques (false negatives)
l A payload identification mechanism is used to reduce the false positive rate and scales the detection scheme to gigabit network links
l Fu-Hau Hsu, Fanglu Guo, and Tzi-cker Chiueh, ``Scalable Network-Based Buffer Overflow Attack Detection, '' under submission
l CTCP: Centralized TCP Router http://www.ecsl.cs.sunysb.edu/kctcp_uctcp/index.html
l
StackGuard:
Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks
C Cowan, C Pu, D Maier, H Hinton, P Bakke, S … - 7th USENIX Security Conference, 1998
l
RAD:
A compile-time solution to buffer overflow attacks
T Chiueh, F Hsu - 21st IEEE International Conference on Distributed Computing
…, 2001
l
Buffer Overflow Attacks and
Their Countermeasures
S Grover - Linux Journal, March, 2003
l
Buttercup:
On network-based detection of polymorphic buffer overflow vulnerabilities
A Pasupulati, J Coit, K Levitt, F Wu, SH Li, JC … - IEEE/IFIP Network
Operation and Management Symposium, 2004
l
Network-Based
Buffer Overflow Detection by Exploit Code Analysis
l
Network
Intrusion Detection: Evasion, Traffic Normalization, and End-to-End Protocol
Semantics
M Handley, V Paxson, C Kreibich
- USENIX Security Symposium, 2001
l Vendicator, ``Stack Shield,'' http://www.angelfire.com/sk/stackshield/