In-Depth Case Study on Artificial Neural Network Weights Optimization Using Meta-Heuristic and Heuristic Algorithmic Approach

Abstract

The Meta-heuristic and Heuristic algorithms have been introduced for deep neural network optimization in this paper. Artificial Intelligence and the most used Deep Learning methods are getting popularity in these days, thus we need faster optimization strategies for finding more accurate results in the future. Neural Network Optimization with Particle Swarm Optimization, Backpropagation (BP), Resilient Propagation (Rprop), and Genetic Algorithm (GA) have been used for numerical analysis of different datasets and compared with each other to find out which algorithms work better for finding optimal solutions by reducing training loss. Meta-heuristic algorithms GA and PSO are higher-level formulas and problem-independent techniques that may be used for a diverse number of challenges. The characteristic of heuristic algorithms has extremely specific features that vary depending on the problem. The conventional backpropagation (BP) based optimization, genetic algorithm, particle swarm optimization, and resilient propagation (Rprop) are all fully presented, and how to apply these procedures in artificial deep neural networks optimization is also thoroughly described. Applied numerical simulation over several datasets shows that the Meta-heuristic algorithm particle swarm optimization and also the genetic algorithm performed better than the conventional heuristic algorithm like backpropagation and resilient propagation over these datasets. Evaluation of these algorithms was done based on training epoch and their error convergence.