Unsupervised Learning Based Network Anomaly Detection using One-Class SVM, Isolation Forest, and LOF in IoT Systems
Keywords:
Anomaly detection, Convolutional Neural Networks (CNN), Deep learning, Intrusion detection, Network securityAbstract
This project focuses on detecting network anomalies using unsupervised learning techniques, a powerful approach especially suited for environments where labelled data is scarce or unavailable. The system starts by collecting comprehensive network log data from various sources, such as routers, switches, firewalls, and endpoint devices. This raw data is subjected to pre-processing, including steps like normalization to scale numerical features and encoding to convert categorical variables into machine-readable formats. Following preprocessing, Exploratory Data Analysis (EDA) is conducted to gain insights into data distribution, feature importance, correlations, and potential anomalies. This stage helps in understanding the baseline patterns of normal network behavior. The core of the system employs advanced unsupervised learning algorithms One-Class SVM, Local Outlier Factor (LOF), Isolation Forest, and deep learning-based autoencoders. These models are trained on normal traffic patterns, enabling them to detect deviations without the need for labelled anomalies. By leveraging statistical properties and internal representations learned during training, these models can effectively flag suspicious activity in real time. The system is designed for scalability and can adapt to evolving threats, making it suitable for deployment in dynamic IoT and enterprise network environments. Its ability to detect both known and unknown threats enhances overall cybersecurity posture.
References
I. Petrariu, A. Moscaliuc, C. E. Turcu, and O. Gherman, "A comparative study of unsupervised anomaly detection algorithms used in a small and medium-sized enterprise," International Journal of Advanced Computer Science and Applications, vol. 13, no. 9, pp. 931–940, 2022, doi: https://doi.org/10.14569/IJACSA.2022.01309108
Y. Sayed and R. Ahmed, "Education quality, and teaching and learning in the post-2015 education agenda," International Journal of Educational Development, vol. 40, pp. 330–338, Jan. 2015, doi: https://doi.org/10.1016/j.ijedudev.2014.11.005
V. Chandola, A. Banerjee, and V. Kumar, "Anomaly detection: A survey," ACM Computing Survev., vol. 41, no. 3, pp. 1–58, 2009, doi: https://doi.org/10.1145/1541880.1541882
M. Schmidt-Heydt, E. Graf, J. Batzler, and R. Geisen, "The application of transcriptomics to understand the ecological reasons of ochratoxin A biosynthesis by Penicillium nordicum on sodium chloride rich dry cured foods," Trends in Food Science & Technology, vol. 22, pp. S39–S48, Nov. 2011, doi: https://doi.org/10.1016/j.tifs.2011.02.010
C. M. Forsyth, A. Graesser, A. M. Olney, K. Millis, B. Walker, and Z. Cai, "Moody agents: affect and discourse during learning in a serious game," in Artificial Intelligence in Education: 17th International Conference, AIED 2015, Madrid, Spain, June 22–26, 2015. Proceedings 17, Cham, Switzerland: Springer, 2015, pp. 135–144. doi: https://doi.org/10.1007/978-3-319-19773-9_14
N. K. Sahu and I. Mukherjee, "Machine Learning based anomaly detection for IoT Network: (Anomaly detection in IoT Network)," 2020 4th International Conference on Trends in Electronics and Informatics (ICOEI), Tirunelveli, India, 2020, pp. 787–794, doi: https://doi.org/10.1109/ICOEI48184.2020.9142921
V. Hodge and J. Austin, "A survey of outlier detection methodologies," Artificial Intelligence Review, vol. 22, pp. 85–126, Oct. 2004, doi: https://doi.org/10.1023/B:AIRE.0000045502.10941.a9
J. Paparrizos, F. Yang, and H. Li, "Bridging the gap: A decade review of time-series clustering methods," Arxiv Preprint Arxiv: 2412.20582, Dec. 29, 2024. doi: https://doi.org/10.48550/arXiv.2412.20582
K. Singh, A. S. Chaudhary, and P. Kaur, "A machine learning approach for enhancing defence against global terrorism," 2019 Twelfth International Conference on Contemporary Computing (IC3), Noida, India, 2019, pp. 1–5, doi: https://doi.org/10.1109/IC3.2019.8844947
O.-H. Kwon, T. Crnovrsanin, and K.-L. Ma, "What would a graph look like in this layout? A machine learning approach to large graph visualization," IEEE Transactions on Visualization and Computer Graphics, vol. 24, no. 1, pp. 478–488, Jan. 2018, doi: https://doi.org/10.1109/TVCG.2017.2743858
T. Bashir and N. A. Al-Sammarraie, "Revolutionizing network security with AI and machine learning solutions," International Journal of Computer Applications, vol. 186, no. 53, pp. 35–42, Nov. 2024, doi: https://doi.org/10.5120/ijca2024924217
H. Jia, J. Liu, M. Zhang, X. He, and W. Sun, "Network intrusion detection based on IE-DBN model," Computer Communications, vol. 178, pp. 131–140, Oct. 2021, doi: https://doi.org/10.1016/j.comcom.2021.07.016
J. Liu, J. Wang, C. Xu, H. Jiang, C. Li, L. Zhang, J. Lin, and Z. X. Shen, "Advanced energy storage devices: basic principles, analytical methods, and rational materials design," Advanced science, vol. 5, no. 1, p. 1700322, Jan. 2018, doi: https://doi.org/10.1002/advs.201700322
G. R. Ginni and S. L. Chakravarthy, "A Hybrid Framework for Robust Anomaly Detection: Integrating Unsupervised and Supervised Learning with Advanced Feature Engineering," International Journal of Computational and Experimental Science and Engineering, vol. 11, no. 2, pp. 1993–2017, 2025, doi: https://doi.org/10.22399/ijcesen.1383
I. G. A. Poornima and B. Paramasivan, “Anomaly detection in wireless sensor network using machine learning algorithm,” Computer Communications, vol. 151, pp. 331–337, Feb. 2020, doi: https://doi.org/10.1016/j.comcom.2020.01.005.
