This study aims to design a real-time application to detect several human beings’ universal emotional levels simultaneously. The intra-class and inter-class variations present in images make it one of the most challenging recognition problems. In this regard, a simple solution for facial expression recognition using a combination of convolutional neural network (CNN) with minimal parameters and transfer learning (TL) has been proposed here. The proposed CNN architecture named LiveEmoNet has been jointly trained with wild (FER-2013) and lab-controlled (CK+) datasets for real-time detection, contributing to versatile emotion detection. The observed experimental results demonstrate that the proposed method outperforms the other related researche concerning accuracy and time. The accuracy of 68.93%, 97.66%, and 96.67% has been achieved on FER-2013, JAFFE, and 7-classes of the CK+ dataset, respectively. Also, real-time detection takes 46.85 ms/frame with an intel i5 2.60 GHz CPU, which is significantly better than other works in the literature.