Metaheuristics for Machine Learning

2A Comprehensive Review of Metaheuristics for Hyperparameter Optimization in Machine Learning

Ramachandran Narayanan¹* and Narayanan Ganesh²

¹Department of Computer Science and Engineering, Sona College of Technology, Salem, India

²School of Computer Science & Engineering, Vellore Institute of Technology, Chennai, India

Abstract

Hyperparameter optimization is a critical step in the development and fine-tuning of machine learning (ML) models. Metaheuristic optimization techniques have gained significant popularity for addressing this challenge due to their ability to search the hyperparameter space efficiently. In this review, we present a detailed analysis of various metaheuristic techniques for hyperparameter optimization in ML, encompassing population-based, single solution-based, and hybrid approaches. We explore the application of metaheuristics in Bayesian optimization and neural architecture search, two prominent areas within the field. Moreover, we provide a comparative analysis of these techniques based on established criteria and evaluate their performance in diverse ML applications. Finally, we discuss future directions and open challenges with special emphasis on the opportunities for improvement in metaheuristics. Other crucial issues like adaptability to new ML paradigms, computational complexity, and scalability issues are also discussed critically. This review aims to provide researchers and practitioners with a comprehensive understanding of the state-of-the-art ...