Although offline handwritten Indic script recognition has been explored for decades, it is still a challenging task to recognize handwritten characters and digits accurately because of complex structure and similar shaped characters in Indic scripts. Like other computer vision problems, handwritten Indic scriptsrecognition has achieved impressive state-of-the-art results using deep learning-based techniques. However, designing a successful handcrafted Deep Neural Network (DNN) right from scratch requires a lot of problem domain knowledge and involves a significant amount of trial and error. This approach intuitively appears to consume substantial time and computational resources. To solve this problem, we simplified the search process by using a meta-heuristics evolutionary technique to automatically evolve the optimal Convolutional NeuralNetwork (CNN) architecture. More specifically, this work proposes a novel framework based on improved and fast converging Adaptive Particle Swarm Optimization (APSO) to design CNN architecture without manual intervention. The computational experiments are subsequently carried out on eight handwritten isolated characters and digits datasets belonging to three popular Indic scripts, namely Bangla, Devanagari, and Dogri. The experimental results clearly show that the proposed APSO-CNN technique yields better performance than the state-of-the-art methods for all the datasets.