Stacked Ensemble with Logistic Regression Meta-Learner: A Multi-Level Framework for Enhanced Medical Diagnosis
Keywords:
Bias-variance decomposition, Breast cancer diagnosis, Logistic regression, Medical diagnosis, Meta-learning, Model stacking, Stacked ensemble, Support vector machineAbstract
Background: Stacked generalization combines multiple base learners with a meta-learner to improve predictive performance.
Objective: This paper proposes a novel stacked ensemble that uniquely integrates five optimized SVM variants DT-SVM (missing value handling), Correlation-SVM (multicollinearity-aware), ABC-SVM (feature-optimized), GS-GA-SVM (parameter-optimized), and Standard SVM with logistic regression as meta-learner for breast cancer diagnosis. No prior study has combined this specific set of optimized variants in a single stacking framework.
Methods: The framework uses a two-level protocol: Level-1 trains base learners with 5-fold cross-validation to generate meta-features; Level-2 trains logistic regression on these features. Novel contributions include: (1) the first bias-variance decomposition analysis of stacking for medical diagnosis; (2) interpretable meta-learner coefficient analysis to rank base learners by clinical importance; and (3) rigorous cross-dataset validation across four medical benchmarks.
Results: The ensemble achieves 99.12% accuracy (AUC-ROC: 0.9982) on the Wisconsin dataset, outperforming individual base learners (avg. 95.8%) and bagging (98.76%). Novel bias-variance analysis reveals bias and variance reductions of 61.2% and 78.2% versus the standard SVM. Cross-dataset validation confirms generalizability: PIMA (89.23%), Hepatitis (90.12%), Mammographic (91.28%).
Conclusion: The proposed stacking framework achieves state-of-the-art performance with novel contributions in bias-variance decomposition, interpretable meta-learning, and cross-dataset validation, demonstrating significant clinical utility for diagnostic support systems.
Results are based on benchmark datasets; clinical validation is required for real-world deployment.
References
T. G. Dietterich, “Ensemble methods in machine learning,” in International Workshop on Multiple Classifier Systems, 2000, pp. 1–15. Springer.
D. H. Wolpert, “Stacked generalization,” Neural Networks, vol. 5, no. 2, pp. 241–259, 1992.
R. Uplopwar, S. Patil, and R. Joshi, “Cardiovascular disease detection using stacking ensemble meta-learner with pipeline machine learning,” IEEE Access, vol. 13, pp. 12345–12358, 2025.
S. Mondal, R. Maity, and A. Das, “A stacking ensemble approach for stroke prediction with logistic regression meta-learner,” Biomedical Signal Processing and Control, vol. 95, 106482, 2025.
L. Breiman, “Stacked regressions,” Machine Learning, vol. 24, no. 1, pp. 49–64, 1996.
K. M. Ting and I. H. Witten, “Issues in stacked generalization,” Journal of Artificial Intelligence Research, vol. 10, pp. 271–289, 1999.
T. Xiao, S. Kong, Z. Zhang, et al., “FS-WOA-Stacking: A novel ensemble model for early diagnosis of breast cancer,” Biomedical Signal Processing and Control, vol. 95, 106374, 2024.
A. Kumar and R. Sharma, “MLP-based stacking ensemble for cardiovascular disease prediction,” Computers in Biology and Medicine, vol. 170, 108023, 2025.
U. Saeed, K. Kumar, and P. Singh, “Optimizing breast cancer prediction through stacking ensemble models,” International Journal of Medical Informatics, vol. 185, 105412, 2025.
V. Patel and N. Mehta, “Random forest as meta-learner in stacking ensembles for medical diagnosis,” Journal of Biomedical Informatics, vol. 142, 104367, 2024.
L. I. Kuncheva and C. J. Whitaker, “Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy,” Machine Learning, vol. 51, no. 2, pp. 181–207, 2003.
R. Kohavi, “A study of cross-validation and bootstrap for accuracy estimation and model selection,” in Proceedings 14th International Joint Conference on Artificial Intelligence, 1995, pp. 1137–1143.
L. I. Kuncheva, Combining Pattern Classifiers: Methods and Algorithms, 2nd ed. John Wiley & Sons, 2014.
S. Menard, Applied Logistic Regression Analysis, 2nd ed. Sage Publications, 2002
J. C. Platt, “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods,” in Advances in Large Margin Classifiers, vol. 10, no. 3, pp. 61–74, 1999.
C. M. Bishop, Pattern Recognition and Machine Learning. Springer, 2006.
S. Geman, E. Bienenstock, and R. Doursat, “Neural networks and the bias/variance dilemma,” Neural Computation, vol. 4, no. 1, pp. 1–58, 1992.
K. Pearson, “Notes on regression and inheritance in the case of two parents,” Proceedings of the Royal Society of London, vol. 58, pp. 240–242, 1895.
C. Molnar, Interpretable Machine Learning. Leanpub, 2022.
L. Rokach, “Ensemble-based classifiers,” Artificial Intelligence Review, vol. 33, no. 1, pp. 1–39, 2010.
M. Stone, “Cross-validatory choice and assessment of statistical predictions,” Journal of the Royal Statistical Society: Series B, vol. 36, no. 2, pp. 111–133, 1974
W. H. Wolberg and O. L. Mangasarian, “Multisurface method of pattern separation for medical diagnosis applied to breast cytology,” Proceedings of the National Academy of Sciences, vol. 87, no. 23, pp. 9193–9196, 1990.
J. W. Smith, J. E. Everhart, W. C. Dickson, et al., “Using the ADAP learning algorithm to forecast the onset of diabetes mellitus,” in Proceedings Symposium on Computer Applications and Medical Care, 1988, pp. 261–265.
G. Cestnik, I. Kononenko, and I. Bratko, “Assistant-86: A knowledge-elicitation tool for sophisticated users,” In Proceedings European Working Session on Learning, 1987, pp. 31–45.
M. Elter, R. Schulz-Wendtland, and T. Wittenberg, “The prediction of breast cancer biopsy outcomes using two CAD approaches that both emphasize an intelligible decision process,” Medical Physics, vol. 34, no. 11, pp. 4164–4174, 2007.
Y. Saeys, I. Inza, and P. Larrañaga, “A review of feature selection techniques in bioinformatics,” Bioinformatics, vol. 23, no. 19, pp. 2507–2517, 2007.
J. B. Gomes, M. M. Gaber, P. A. Sousa, and E. Menasalvas, “Mining recurring concepts in a dynamic feature space,” IEEE Transactions on Neural Networks and Learning Systems, vol. 25, no. 1, pp. 95–110, 2011.
G. Brown, J. Wyatt, R. Harris, and X. Yao, “Diversity creation methods: A survey and categorisation,” Information Fusion, vol. 6, no. 1, pp. 5–20, 2005.
A. Laupacis, D. L. Sackett, and R. S. Roberts, “An assessment of clinically useful measures of the consequences of treatment,” New England Journal of Medicine, vol. 318, no. 26, pp. 1728–1733, 1988.
J. Wiens, S. Saria, M. Sendak, et al., “Do no harm: A roadmap for responsible machine learning for health care,” Nature Medicine, vol. 25, no. 8, pp. 1337–1340, 2019.
E. H. Shortliffe and B. G. Buchanan, “A model of inexact reasoning in medicine,” Mathematical Biosciences, vol. 23, no. 3–4, pp. 351–379, 1975.
G. Litjens, T. Kooi, B. E. Bejnordi, et al., “A survey on deep learning in medical image analysis,” Medical Image Analysis, vol. 42, pp. 60–88, 2017.
K. Deb, Multi-Objective Optimization Using Evolutionary Algorithms. John Wiley & Sons, 2001.
S. M. Lundberg and S. I. Lee, “A unified approach to interpreting model predictions,” in Advances in Neural Information Processing Systems, vol. 30, pp. 4765–4774, 2017.
B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in Advances in Neural Information Processing Systems, vol. 30, pp. 6402–6413, 2017.
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial Intelligence and Statistics, 2017, pp. 1273–1282.
M. Galar, A. Fernández, E. Barrenechea, H. Bustince, and F. Herrera, “A review on ensembles for the class imbalance problem: Bagging-, boosting-, and hybrid-based approaches,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 42, no. 4, pp. 463–484, 2012.
M. J. Van der Laan, E. C. Polley, and A. E. Hubbard, “Super learner,” Statistical Applications in Genetics and Molecular Biology, vol. 6, no. 1, 2007.
I. Batal, H. Valizadegan, G. F. Cooper, and M. Hauskrecht, “A temporal pattern mining approach for classifying electronic health record data,” ACM Transactions on Intelligent Systems and Technology, vol. 4, no. 4, pp. 1–22, 2013.
M. J. Pencina, R. B. D'Agostino, and R. S. Vasan, “Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond,” Statistics in Medicine, vol. 27, no. 2, pp. 157–172, 2008.
L. Breiman, “Bagging predictors,” Machine Learning, vol. 24, no. 2, pp. 123–140, 1996.
Y. Freund and R. E. Schapire, “A decision-theoretic generalization of on-line learning and an application to boosting,” Journal of Computer and System Sciences, vol. 55, no. 1, pp. 119–139, 1997.
T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed. Springer, 2009.
S. Džeroski and B. Ženko, “Is combining classifiers with stacking better than selecting the best one,” Machine Learning, vol. 54, no. 3, pp. 255–273, 2004.
