Cooperative Intrusion Response under Noisy Alerts in a Controlled Cloud IoT Network Simulation
Keywords:
Cloud IoT simulation, Intrusion response, Multi-agent reinforcement learning, Service cost, Shared rewardAbstract
Automated intrusion response requires an environment in which defensive actions alter subsequent network states. This study evaluates cooperative tabular reinforcement learning in a transparent four-agent cloud IoT simulation with noisy intrusion alerts. Each agent can leave traffic unchanged, inspect it, or isolate its local service. Compromise, propagation, recovery, and action costs are explicitly specified. Independent learners use local alerts, whereas cooperative learners also observe whether a neighbour is alarming and optimise a shared immediate reward. Both methods train for 1,500 episodes on a ring topology and are evaluated for 200 episodes on a star topology across five seeds. Fixed baselines include no intervention, continuous inspection, and isolation on an alert. The alert isolation rule obtains a mean per-agent per-step cost of 0.1569, compared with 0.1793 for cooperative learning and 0.1862 for independent learning. Cooperative learning reduces the compromised fraction to 0.0229, compared with 0.0423 for the rule, but increases isolation to 0.1696 from 0.0855. The measured trade-off therefore does not support a claim of overall superiority for cooperative learning. The results are simulation evidence under declared transition and cost assumptions, not measured containment effects on CICIDS2017 or CICIoT2023 traffic.
References
I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward generating a new intrusion detection dataset and intrusion traffic characterization,” Proceedings of the 4th International Conference on Information Systems Security and Privacy ICISSP, 2018, pp. 108–116.
E. C. P. Neto, S. Dadkhah, R. Ferreira, A. Zohourian, R. Lu, and A. A. Ghorbani, “CICIoT2023: A real-time dataset and benchmark for large-scale attacks in IoT environment,” Sensors, vol. 23, no. 13, Jun. 2023.
T. T. Nguyen and V. J. Reddi, “Deep reinforcement learning for cyber security,” in IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 8, pp. 3779–3795, Aug. 2023.
R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” Proceedings of the 31st International Conference on Neural Information Processing Systems, Dec. 2017, pp. 6382–6393.
T. Rashid, M. Samvelyan, C. S. de Witt, G. Farquhar, J. Foerster, and S. Whiteson, “QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning,” Journal of Machine Learning Research, vol. pp. 1–51, 2020.
C. Yu et al., “The surprising effectiveness of PPO in cooperative, multi-agent games,” Advances in Neural Information Processing Systems, 2022.
J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” Proceedings of the 34th International Conference on Machine Learning, 2017, pp. 22–31.
J. García and F. Fernández, “A comprehensive survey on safe reinforcement learning,” Journal of Machine Learning Research, vol. 16, no. 1, pp. 1437–1480, Jan. 2015.
K. Zhang, Z. Yang, and T. Başar, “Multi-agent reinforcement learning: A selective overview of theories and algorithms,” Handbook of Reinforcement Learning and Control, 2021, pp. 321–384.
D. Arp et al., “Dos and don'ts of machine learning in computer security,” Proceedings of the 31st USENIX Security Symposium, 2022, pp. 3971–3988.
M. Standen, M. Lucas, D. Bowman, T. J. Richer, J. Kim, and D. Marriott, “CybORG: A gym for the development of autonomous cyber agents,” arXiv, 2021.
J. K. Terry et al., “PettingZoo: Gym for multi-agent reinforcement learning,” arXiv, Sept. 2020.
S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” arXiv, May 2020.
A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative q-learning for offline reinforcement learning,” arXiv, Jun. 2020.
N. Jiang and L. Li, “Doubly robust off-policy value evaluation for reinforcement learning,” Proceedings of The 33rd International Conference on Machine Learning, 2016, pp. 652–661.
P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, “Deep reinforcement learning that matters,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018.
R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, and M. G. Bellemare, “Deep reinforcement learning at the edge of the statistical precipice,” arXiv, 2021.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv, Jul. 2017.
T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” arXiv, Sept. 2016.
W. L. Hamilton, R. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” Proceedings of the 31st International Conference on Neural Information Processing Systems, Dec. 2017, pp. 1025–1035.
P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” arXiv, Oct. 2017.
P. W. Koh et al., “WILDS: A benchmark of in-the-wild distribution shifts,” Proceedings of the 38th International Conference on Machine Learning, 2021, pp. 5637–5664.
R. Geirhos et al., “Shortcut Learning in Deep Neural Networks,” Nature Machine Intelligence, vol. 2, pp. 665–673, 2020.
T. Saito and M. Rehmsmeier, “The Precision-Recall Plot is more informative than the ROC Plot when evaluating binary classifiers on imbalanced datasets,” PLoS ONE, vol. 10, no. 3, 2015.
M. Mitchell et al., “Model cards for model reporting,” arXiv, 2018.
T. Gebru et al., “Datasheets for Datasets,” Communications of the ACM, vol. 64, no. 12, pp. 86–92, Nov. 2021.
S. Lundberg and S. I. Lee, “A unified approach to interpreting model predictions,” Proceedings of the 31st International Conference on Neural Information Processing Systems, Dec. 2017, pp. 4768–4777.
M. T. Ribeiro, S. Singh, and C. Guestrin, ““Why should i trust you?”: Explaining the predictions of any classifier,” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Aug. 2016, pp. 1135–1144.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” Proceedings of the 34th International Conference on Machine Learning, Aug. 2017, pp. 1321–1330.
S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine learning-based science,” Patterns, vol. 4, no. 9, Sept. 2023.
