Advanced Analytics and Deep Learning Models

11Optimization Techniques in Deep Learning Scenarios: An Empirical Comparison

Ajeet K. Jain¹*, PVRD Prasad Rao² and K. Venkatesh Sharma³

¹Department of Computer Science and Engineering, Koneru Lakshmaiah Education Foundation, Vaddeswaram, AP, India; (Association: CSE, KMIT, Hyderabad, India)

²CSE, KLEF, Vaddeswaram, AP, India

³CSE, CVR College of Engineering, Hyderabad, India

Abstract

Machine learning has enormously contributed toward optimization techniques motivating new approaches for optimization algorithms and their usage that have played a pivotal role in data science. The optimization approaches in deep learning has wide applicability with resurgence of novelty starting from Stochastic Gradient Descent to convex and non-convex and derivative-free approaches. Selecting an optimizer is an important choice in deep learning scenarios, and the optimization algorithm chosen having convexity principles in their core determines the training speed and final performance predicted by the DL model. The complexity further increases with growing deepness due to hyperparameter tuning and as the datasets become larger and, in turn, they require a fitting optimizer.

In this chapter, we examine the most popular and widely optimizers algorithms in an empirical way. The augmenting behaviors of these are tested on MNIST, SKLEARN datasets. We empirically compare them pointing out their similarities, differences, and likelihood of their suitability for a given applications. Recent variants ...