Mathieu equation is a well-known ordinary differential equation in which the excitation term appears as the non-constant coefficient. The mathematical modelling of many dynamic systems leads to Mathieu equation. The determination of the locus of unstable zone is important for the control of dynamic systems. In this paper, the stable and unstable regions of Mathieu equation are determined for three cases of linear and nonlinear equations using the homotopy perturbation method. The effect of nonlinearity is examined in the unstable zone. The results show that the transition curves of linear Mathieu equation depend on the frequency of the excitation term. However, for nonlinear equations, the curves depend also on initial conditions. In addition, increasing the amplitude of response leads to an increase in the unstable zone.