Review on Malware Detection and Classification Using Machine Learning and Deep Learning Techniques
DOI:
https://doi.org/10.54536/ajdsai.v2i2.8150Keywords:
Cybersecurity, Deep Learning, Machine Learning, Malware Classification, Malware DetectionAbstract
Malicious software (malware) is increasing at an alarming rate, and modern malware routinely uses obfuscation techniques to evade detection, making reliable identification an open problem in cybersecurity. Signature- and heuristic-based approaches remain fast and effective against known malware but fail against unknown and zeroday variants, while behavior-based, model-checking-based, and cloud-based approaches perform better against unknown and complex threats; deep-learning-, mobile-, and IoT-based approaches have since emerged to address specific portions of the known and unknown malware landscape. This paper presents a structured literature review of malware detection approaches spanning signature-based, behavior-based, heuristic-based, model-checking-based, and deep-learning-based techniques, drawing on more than 110 primary studies published mainly between 2000 and 2020. For each approach, the underlying methodology, representative studies, reported detection performance, and documented limitations are summarized and compared. The review finds that no single approach reliably detects all malware in the wild: signature- and heuristic-based methods are fast but limited to known threats, behavior- and model-checking-based methods generalize better to unknown malware but carry higher false-positive rates and resource overhead, and deeplearning-based methods substantially reduce manual feature engineering yet remain vulnerable to adversarial evasion. These findings underscore both the difficulty of the underlying detection problem, which has been shown to be NP-complete, and the need for further research. The review concludes by motivating hybrid detection strategies that combine complementary techniques, such as behavior-based analysis with model checking or deep learning with cloud-based resources, as a promising direction for future malware detection research.
References
Adleman, L. M. (1990). An abstract theory of computer viruses. In Advances in Cryptology—CRYPTO (pp. 354– 374). Springer-Verlag.
Alosefer, Y. (2012). Analysing web-based malware behaviour through client honeypots [Doctoral dissertation, Cardiff University]. https://orca.cardiff.ac.uk/id/eprint/29469/
Anderson, B., Quist, D., Neil, J., Storlie, C., & Lane, T. (2011). Graph-based malware detection using dynamic analysis. Journal of Computer Virology, 7(4), 247–258.
Anderson, S., & Roth, P. (2018). EMBER: An open dataset for training static PE malware machine learning models. arXiv:1804.04637.
Arp, D., Spreitzenbarth, M., Hübner, M., Gascon, H., & Rieck, K. (2014). Drebin: Effective and explainable detection of Android malware in your pocket. In Proceedings of the Network and Distributed System Security Symposium (Vol. 14, pp. 23–26).
Battista, P., Mercaldo, F., Nardone, V., Santone, A., & Visaggio, C. A. (2016). Identification of Android malware families with model checking. In Proceedings of the 2nd International Conference on Information Systems Security and Privacy.
Bazrafshan, Z., Hashemi, H., Fard, S. M. H., & Hamzeh, A. (2013). A survey on heuristic malware detection techniques. In Proceedings of the 5th Conference on Information and Knowledge Technology.
Borojerdi, H. R., & Abadi, M. (2013). MalHunter: Automatic generation of multiple behavioral signatures for polymorphic malware detection. In Proceedings of the International Conference on Computer and Knowledge Engineering (ICCKE) (Vol. 1). Ferdowsi University of Mashhad.
Chandramohan, M., Tan, H. B. K., Briand, L. C., Shar, L. K., & Padmanabhuni, B. M. (2013). A scalable approach for malware detection through bounded feature space behavior modeling. In Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering (pp. 312–322).
Chinonye, A. E., & Onah, B. I. (2025). Assessing the efficacy of AI-based techniques in anomaly detection in financial institutions. American Journal of Data Science and Artificial Intelligence, 1(2), 11–17. https://doi.org/10.54536/ajdsai.v1i2.5062
Cohen, F. (1987). Computer viruses: Theory and experiments. Computers & Security, 6(1), 22–35.
Dahl, G. E., Stokes, J. W., Deng, L., & Yu, D. (2013). Large-scale malware classification using random projections and neural networks. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing.
Griffin, K., Schneider, S., Hu, X., & Chiueh, T.-C. (2009). Automatic generation of string signatures for malware detection. In Proceedings of the International Workshop on Recent Advances in Intrusion Detection. Springer.
Grosse, K., Papernot, N., Manoharan, P., Backes, M., & McDaniel, P. (2016). Adversarial perturbations against deep neural networks for malware classification. arXiv:1606.04435.
Howard, C. C., Otobo, F. N., & Odogu, E. (2026). Analyzing temporal dependency structures in cyber security: A case study of network traffic anomalies. American Journal of Data Science and Artificial Intelligence, 2(1), 22–32. https://doi.org/10.54536/ajdsai.v2i1.5081
Huang, W., & Stokes, J. W. (2016). MtNet: A multi-task neural network for dynamic malware classification. In Proceedings of the International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer.
Islam, R., Tian, R., Batten, L. M., & Versteeg, S. (2013). Classification of malware based on integrated static and dynamic features. Journal of Network and Computer Applications, 36(2), 646–656.
Kinder, J., Katzenbeisser, S., Schallhart, C., & Veith, H. (2005). Detecting malicious code by model checking. In Proceedings of the International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer.
Kolbitsch, C., Comparetti, P. M., Kruegel, C., Kirda, E., Zhou, X.-Y., & Wang, X. (2009). Effective and efficient malware detection at the end host. In Proceedings of the USENIX Security Symposium (Vol. 4, No. 1, pp. 351–366).
Kolosnjaji, B., Demontis, A., Biggio, B., Maiorca, D., Giacinto, G., Eckert, C., & Roli, F. (2018). Adversarial malware binaries: Evading deep learning for malware detection in executables. In Proceedings of the 26th European Signal Processing Conference (EUSIPCO).
Lanzi, A., Balzarotti, D., Kruegel, C., Christodorescu, M., & Kirda, E. (2010). AccessMiner: Using system-centric models for malware protection. In Proceedings of the 17th ACM Conference on Computer and Communications Security (pp. 399–412).
Lashkari, A. H., Kadir, A. F. A., Gonzalez, H., Mbah, K. F., & Ghorbani, A. A. (2017). Towards a network-based framework for Android malware detection and characterization. In Proceedings of the 15th Annual Conference on Privacy, Security and Trust (PST).
Morgan, S. (2019). 2019 cybersecurity almanac: 100 facts, figures, predictions and statistics. Cisco and Cybersecurity Ventures. https://cybersecurityventures.com/cybersecurity-almanac-2019
Naval, S., Laxmi, V., Rajarajan, M., Gaur, M. S., & Conti, M. (2015). Employing program semantics for malware detection. IEEE Transactions on Information Forensics and Security, 10(12), 2591–2604.
Pajouh, H. H., Dehghantanha, A., Khayami, R., & Choo, K.-K. R. (2018). Intelligent OS X malware threat detection with code inspection. Journal of Computer Virology and Hacking Techniques, 14(3), 213–223.
Ronen, R., Radu, M., Feuerstein, C., Yom-Tov, E., & Ahmadi, M. (2018). Microsoft malware classification challenge. arXiv. https://arxiv.org/abs/1802.10135
Samani, R., & Davis, G. (2019). McAfee mobile threat report: Q1 2019. McAfee. https://www.mcafee.com/enterprise/en-us/assets/reports/rp-mobile-threat-report-2019.pdf
Santos, I., Penya, Y. K., Devesa, J., & Bringas, P. G. (2009). N-grams-based file signatures for malware detection. In Proceedings of the 11th International Conference on Enterprise Information Systems (Vol. 9, pp. 317–320).
Saxe, J., & Berlin, K. (2015). Deep neural network based malware detection using two dimensional binary program features. In Proceedings of the 10th International Conference on Malicious and Unwanted Software (MALWARE).
Sikorski, M., & Honig, A. (2012). Practical malware analysis: The hands-on guide to dissecting malicious software. No Starch Press.
Song, F., & Touili, T. (2012). Efficient malware detection using model-checking. In Proceedings of the International Symposium on Formal Techniques. Springer.
Song, F., & Touili, T. (2014). Pushdown model checking for malware detection. International Journal on Software Tools for Technology Transfer, 16(2), 147–173.
Spinellis, D. (2003). Reliable identification of bounded-length viruses is NP-complete. IEEE Transactions on Information Theory, 49(1), 280–284.
Szor, P. (2005). The art of computer virus research and defense. Pearson Education.
Tang, Y., Xiao, B., & Lu, X. (2009). Using a bioinformatics approach to generate accurate exploit-based signatures for polymorphic worms. Computers & Security, 28(8), 827–842.
Tavallaee, M. (2009). A detailed analysis of the KDD CUP 99 data set. In Proceedings of the IEEE Symposium on Computational Intelligence for Security and Defense Applications (pp. 1–6).
Wagener, G., State, R., & Dulaunoy, A. (2008). Malware behaviour analysis. Journal in Computer Virology, 4(4), 279–287.
Ye, Y., Li, T., Jiang, Q., & Wang, Y. (2010). CIMDS: Adapting postprocessing techniques of associative classification for malware detection. IEEE Transactions on Systems, Man, and Cybernetics, Part C, 40(3), 298–307.
Ye, Y., Chen, L., Hou, S., Hardy, W., & Li, X. (2018). DeepAM: A heterogeneous deep learning framework for intelligent malware detection. Knowledge and Information Systems, 54(2), 265–285.
Yuan, Z., Lu, Y., Wang, Z., & Xue, Y. (2014). Droid-Sec: Deep learning in Android malware detection. ACM SIGCOMM Computer Communication Review, 44(4), 371–372.
Zheng, M., Sun, M., & Lui, J. C. (2013). Droid analytics: A signature based analytic system to collect, extract, analyze and associate Android malware. In Proceedings of the 12th IEEE International Conference on Trust, Security and Privacy in Computing and Communications.
Zhu, D., Jin, H., Yang, Y., Wu, D., & Chen, W. (2017). DeepFlow: Deep learning-based malware detection by mining Android application for abnormal usage of sensitive data. In Proceedings of the IEEE Symposium on Computers and Communications (ISCC).
Zuo, Z., Zhu, Q., & Zhou, M. (2005). On the time complexity of computer viruses. IEEE Transactions on Information Theory, 51(8), 2962–2966.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Umesh Balami, Bhoj Raj Ghimire
This work is licensed under a Creative Commons Attribution 4.0 International License.
