In this paper, a new type of flux-controlled memristor model with fifth-order flux polynomials is presented. An equivalent circuit which realizes the action of higher-order flux-controlled memristor is also proposed. We use the memristor model to establish a memristor-based four-dimensional (4D) chaotic system, which can generate three-scroll chaotic attractor. By adjusting the system parameters, the proposed chaotic system performs hyperchaos. Phase portraits, Lyapunov exponents, bifurcation diagram, equilibrium points and stability analysis have been used to research the basic dynamics of this chaotic system. The consistency of circuit implementation and numerical simulation verifies the effectiveness of the system design.