Scalability of Hash-based Pattern Matching for High-speed Network Security and Monitoring

hash functions, high-speed networks, scalability, pattern matching

Gradually increasing throughput of high-speed networks puts continuous pressure on the performance of operations over a stream of network data. Probably the most affected operation in the area of network security and monitoring is pattern matching, which is in the core of widely deployed intrusion detection systems (IDSes) like Snort, Suricata and Bro. This paper therefore proposes several optimizations of a hash-based pattern matching architecture that together allow to increase its throughput to 100 Gbps and beyond. Proposed optimizations target an interconnection network between parallel hash function engines and independent memory blocks addressed by hashes computed over short strings of input data. Specifically, the optimizations reduce resource utilization by sharing parts of the full interconnection network among its several outputs and lower collision rate in these shared parts by aggregation and distributed buffering of memory access requests. The optimized pattern matching architecture is therefore able to utilize a higher number of parallel hash functions, each of which can use the interconnection network to access any memory block. This allows not only to increase the throughput of a key component within IDSes to more than 100 Gbps, but also to support a larger set of network threat patterns and to update this set dynamically.