In order to rapidly build an automatic and precise system for image recognition and categorization, deep learning is a vital technology. Handwritten character classification also gaining more attention due to its major contribution in automation and specially to develop applications for helping visually impaired people. Here, the proposed work highlighting on fine-tuning approach and analysis of state-of-the-art Deep Convolutional Neural Network (DCNN) designed for Devanagari Handwritten characters classification. A new Devanagari handwritten characters dataset is generated which is publicly available. Datasets consist of total 5800 isolated images of 58 unique character classes: 12 vowels, 36 consonants and 10 numerals. In addition to this database, a two-stage VGG16 deep learning model is implemented to recognize those characters using two advanced adaptive gradient methods. A two-stage approach of deep learning is developed to enhance overall success of the proposed Devanagari Handwritten Character Recognition System (DHCRS). The first modelachieves 94.84% testing accuracy with training loss of 0.18 on new dataset. Moreover, the second fine-tuned model requires very fewer trainable parameters and notably less training time to achieve state-of-the-art performanceon a very small dataset. It achieves 96.55% testing accuracy with training loss of 0.12. We also tested the proposed model on four different benchmark datasets of isolated characters as well as digits of Indic scripts. For all the datasets tested, we achieved the promising results