Towards Building Network Outlier Detection System for Network Traffic Monitoring

Traffic monitoring is important for supporting network security and management. Recent advancements have explored machine learning-based approaches to classify encrypted traffic, yet the challenge of obtaining current threat datasets persists, leaving supervised models reliant on outdated information. Outlier detection, which identifies anomalous network behavior without requiring labeled data, addresses this limitation by flagging suspicious deviations from expected patterns. This paper proposes a novel Network Outlier Detection System (NODS), a platform based on open-source software designed to detect outliers in network traffic by leveraging forecasting models. Our system was deployed and tested on a large ISP infrastructure. The evaluation of detected outliers over a one-month period showed key insights into system performance and provided valuable lessons for future deployment of outlier detection methods. This paper details the architecture of NODS, deployment, and performance while highlighting the challenges and lessons learned in building an effective outlier detection system for network traffic.