In this paper we show that the h-p spectral element method developed in [3,8,9] applies to elliptic problems in curvilinear polygons with mixed Neumann and Dirichlet boundary conditions provided that the Babuska-Brezzi inf-sup conditions are satisfied. We establish basic stability estimates for a non-conforming h-p spectral element method which allows for simultaneous mesh refinement and variable polynomial degree. The spectral element functions are non-conforming if the boundary conditions are Dirichlet. For problems with mixed boundary conditions they are continuous only at the vertices of the elements. We obtain a stability estimate when the spectral element functions vanish at the vertices of the elements, which is needed for parallelizing the numerical scheme. Finally, we indicate how the mesh refinement strategy and choice of polynomial degree depends on the regularity of the coefficients of the differential operator, smoothness of the sides of the polygon and the regularity of the data to obtain the maximum accuracy achievable.