The classical and quantum mechanics of a system of directly interacting relativistic particles is discussed. We first discuss the spin-zero case, where we basically follow Rohrlich in introducing a set of covariant centre of mass (CM) and relative variables. The relation of these to the classic formulation of Bakamjian and Thomas is also discussed. We also discuss the important case of relativistic potentials which may depend on total four-momentum squared. We then consider the quantum mechanical case of spin-half particles. The negative energy difficulty is solved by introducing a number of first class constraints which fix the spinor structure of physical solutions and ensure the existence of proper CM and relative variables. We derive the form of interactions consistent with Lorentz invariance, space inversion, time reversal and charge conjugation and with the above mentioned first class constraints and find that it is analogous to that for the non-relativistic case. Finally the relationship of the present work with some previous work is briefly discussed.