Recognizing the person from a sample of their voice print or speech samples is known as Speaker recognition. It is an emerging technology, in recent years machine learning (ML) based classification schemes have been observed as one kind of alternate solution for speaker recognition. In this paper adaptive boosting(AdaBoost) combined with a powerful ML classifier (Random Forest) is proposed to handle multi-class imbalanced speaker data classification. Based on weights AdaBoost integrates several sub-classifiers and constructs a robust classifier. A new strong and more accurate technique is proposed by employing Random Forests as the initial stage classifier and AdaBoost as the subsequent stage classifiers, to decide the class that the speaker sample belongs to. Three dissimilar datasets are utilized to estimate the robustness of the proposed hybridAdaBoost technique. The classification results of the hybrid RF-AdaBoost are evaluated against other state-ofthe- art algorithms (kNN, SVM, RF, kNN- AdaBoost, and SVM- AdaBoost), experimental results convey the proposed algorithm improves accuracy as well as stability for the imbalanced speaker data. The f1_score for RFAdaBoost is 92%, as well as it produces minimum root mean squared value. The stability of the hybrid algorithm is evaluated using Matthews correlation coefficient (MCC), g-means metric value, and variance, it shows RFAdaBoost outperforms the other state-of-the-art algorithms in all the aspects of speaker recognition.