In this paper, the AB system with time-dependent coefficients for the ultrashort pulses in an inhomogeneous optical fibre or the marginally unstable baroclinic wave packets in an atmospheric or oceanic system is investigated via the Lie symmetry analysis. We obtain the Lie symmetries, reduced equations and groupinvariant solutions. The nonlinear self-adjointness of the AB system is proved, and the conservation laws associated with the Lie symmetries are constructed. For the amplitude of the electric field in the inhomogeneous optical fibre or the amplitude of the wave packet in the atmospheric or oceanic system, and for the quantity associated with the occupation number which gives a measure of the atomic inversion in the inhomogeneous optical fibre or the quantity measuring the correction of the basic flow in the atmospheric or oceanic system, we get some solitons through the Lie symmetry transformations, whose amplitudes,widths, velocities and backgrounds are different from those of the given ones and can be adjusted via the Lie group parameters. We find a family of the ultrashort pulses propagating in the inhomogeneous optical fibre or a family of the marginally unstable baroclinic wave packets propagating in the atmospheric or oceanic system.

The non-local symmetries for the generalized Broer–Kaup (BK) system are obtained with the truncated Painlevé expansion method. The non-local symmetries can be localized to the Lie point symmetries for the extended system by introducing auxiliary dependent variables. The finite symmetry transformations related to the non-local symmetries are computed. The consistent Riccati expansion and consistent tanh expansion integrability of the generalised BK system are proved, and the single soliton, two-resonant soliton and soliton–cnoidal wave interaction solutions are derived.