Hybrid Machine Learning Approach for Financial Fraud Detection: Integrating XGBoost and TabNet
Keywords:
Deep learning, Financial security, Fraud detection, Hybrid model, Machine learning, TabNet, XGBoostAbstract
One of the most crucial responsibilities of the banking and financial industries is detecting fraud in financial transactions, which calls for strong and efficient machine learning models that can accurately identify fraudulent transactions. To increase the effectiveness of fraud detection, we provide a hybrid machine learning approach in this research that combines XGBoost with TabNet. While TabNet takes advantage of one of the most important responsibilities of the banking and financial sectors is identifying fraud in financial transactions, which requires strong and efficient machine learning models that can precisely detect fraudulent activities. To improve fraud detection effectiveness, this research offers a hybrid machine learning approach combining XGBoost with TabNet. While TabNet leverages deep learning for better pattern recognition, XGBoost provides excellent accuracy and feature selection. With an AUC-ROC of 0.99, a precision of 0.89, and superior fraud detection performance, our empirical results on a fraud detection dataset show that the hybrid model surpasses traditional machine learning and deep learning methods. Additionally, SHAP analysis helps make the model more understandable by highlighting the main factors influencing decisions. Our suggested approach delivers a high-performing, interpretable, and scalable financial fraud detection system that can be effectively implemented in core financial institutions. Deep learning for better pattern recognition, XGBoost offers excellent accuracy and feature selection. With an AUC-ROC of 0.99, a precision of 0.89, and superior fraud detection performance, our empirical results on a fraud detection dataset demonstrate that the hybrid model outperforms traditional machine learning and deep learning techniques.
References
S. D. Penmetsa and S. Mohammed, “Ensemble Techniques for Credit Card Fraud Detection,” International Journal of Smart Business and Technology, vol. 9, no. 2, pp. 33–48, Sep. 2021, doi: https://doi.org/10.21742/ijsbt.2021.9.2.03.
A. D. Pozzolo, O. Caelen, R. A. Johnson, and G. Bontempi, “Calibrating Probability with Undersampling for Unbalanced Classification,” 2015 IEEE Symposium Series on Computational Intelligence, Dec. 2015, doi: https://doi.org/10.1109/ssci.2015.33.
F. Carcillo, Y.-A. Le Borgne, O. Caelen, Y. Kessaci, F. Oblé, and G. Bontempi, “Combining unsupervised and supervised learning in credit card fraud detection,” Information Sciences, vol. 557, May 2019, doi: https://doi.org/10.1016/j.ins.2019.05.042.
A. C. Bahnsen, D. Aouada, and B. Ottersten, “Example-Dependent Cost-Sensitive Logistic Regression for Credit Scoring,” International Conference on Machine Learning and Applications, Dec. 2014, doi: https://doi.org/10.1109/icmla.2014.48.
S. Misra, S. Thakur, M. Ghosh, and S. K. Saha, “An Autoencoder Based Model for Detecting Fraudulent Credit Card Transaction,” Procedia Computer Science, vol. 167, pp. 254–262, 2020, doi: https://doi.org/10.1016/j.procs.2020.03.219.
R. Ayyadurai, K. Parthasarathy, N. Kumar, R. Panga, J. Bobba, and R. Lakshmi Bolla, “Deep Convolutional Autoencoders for Fraud Detection in Digital Banking: Anomaly Detection with Reconstruction Error,” International Journal of Engineering, Management and Humanities (IJEMH), vol. 6, no. 2, pp. 2584–2145, 2025, Available: https://ijemh.com/issue_dcp/Deep%20Convolutional%20Autoencoders%20for%20Fraud%20Detection%20in%20Digital%20Banking%20%20Anomaly%20Detection%20with%20Reconstruction%20Error.pdf
D. Cheng, Y. Zou, S. Xiang, and C. Jiang, “Graph Neural Networks for Financial Fraud Detection: A Review,” arXiv (Cornell University), Oct. 2024, doi: https://doi.org/10.1007/s11704-024-40474-y.
W. Hamilton, R. Ying, and J. Leskovec, “Inductive Representation Learning on Large Graphs,”
Social and Information Network, 2017, doi: https://doi.org/10.48550/arxiv.1706.02216.
E. Btoush, X. Zhou, R. Gururajan, K. C. Chan, and O. Alsodi, “Achieving Excellence in Cyber Fraud Detection: A Hybrid ML+DL Ensemble Approach for Credit Cards,” Applied Sciences, vol. 15, no. 3, p. 1081, Jan. 2025, doi: https://doi.org/10.3390/app15031081.
G. Yu and Z. Luo, “Financial fraud detection using a hybrid deep belief network and quantum optimization approach,” Deleted Journal, vol. 7, no. 5, May 2025, doi: https://doi.org/10.1007/s42452-025-06999-y.
S. Shi, W. Luo, and G. Pau, “An attention-based balanced variational autoencoder method for credit card fraud detection,” Applied Soft Computing, vol. 177, p. 113190, Apr. 2025, doi: https://doi.org/10.1016/j.asoc.2025.113190.
E. Nopiel, Abiodun Okunola, E. Phine, and A. L. Rasaq, “Reinforcement Learning for Adaptive Fraud Detection Systems in Dynamic Financial Environments,” Apr. 09, 2025. https://www.researchgate.net/publication/390629307_Reinforcement_Learning_for_Adaptive_Fraud_Detection_Systems_in_Dynamic_Financial_Environments.
S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” Advances in Neural Information Processing Systems, vol. 30, 2017. https://doi.org/10.48550/arXiv.1705.07874
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pretraining of deep bidirectional transformers for language understanding,” Proceedings of NAACL-HLT, 2019. https://doi.org/10.48550/arXiv.1810.04805
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738, 2020. https://doi.org/10.48550/arXiv.1911.05722
