Offline handwriting recognition is an important application of pattern recognition that has attracted a lot of interest from researchers. Transforming any handwritten material into machine-readable text data by extracting hidden patterns and comprehending the texts from the documents is a complex process. There are 22 scheduled languages in India and Gujarati is one among them. There are several optical character recognition issues (OCR) in Gujarati and it is difficult to identify universal invariant patterns and irregularities in handwritten Gujarati script. The lack of a big benchmark dataset is another important issue with handwritten Gujarati script. This issue was identified, and we built a dataset with 75600 images spanning 54 Gujarati character classes. Although, this dataset is reasonably large, it is still not large enough to learn deep neural networks from scratch due to overfitting concerns. To address this problem, we have integrated transfer learning with CNN for Gujarati handwritten character recognition. We have used 5 distinct pre-trained models and have achieved approximately 97% accuracy on images of 54 different classes.