Activity recognition is the interest gaining research area, as the need for monitoring and controlling the public and the society to ensure the detection of the suspects and the illegal activities is of prime importance. The process of recognizing the activity of the humans is employed in various applications, mainly in the field ofsecurity to identify and detect the suspects. Accordingly, this paper uses a novel method named as the Fuzzy- DDBN classifier to categorize the human activities in the video. At first, the keyframes are extracted from the video based on the Bhattacharya similarity measure, and the keyframes are subjected to feature extraction using the Scale Invariant Feature Transform (SIFT) and the Spatio-Temporal Interest (STI) descriptor. The features extracted from the descriptors are fed to the classifier for classification that in turn, uses the GMM clustering.The classified output from the proposed Fuzzy-DDBN classifier, which is the combination of the fuzzy and the Dragon Deep Belief Neural Network (DDBN) classifier is merged using the correlation coefficients. The proposed method is experimented using two standard datasets to prove the superiority of the method with an accuracy of 0.98, specificity at a rate of 0.981, and sensitivity at a rate of 0.98 respectively.