Linear response theory is used to express the anelastic response (creep function and generalized compliance) of a system under an applied stress, in terms of the equilibrium strain auto-correlation. These results extend an earlier analysis to cover inhomogeneous stresses and the tensor nature of the variables. For anelasticity due to point defects, we express the strain compactly in terms of the elastic dipole tensor and the probability matrix governing dipole re-orientation and migration. We verify that re-orientations contribute to the deviatoric strain alone (Snoek, Zener, etc. effects), while the dilatory part arises solely from the long-range diffusion of the defects under a stressgradient (the Gorsky effect). Our formulas apply for arbitrary orientational multiplicity, specimen geometry, and stress inhomogeneity. The subsequent development of the theory in any given situation then reduces to the modelling of the probability matrix referred to. In a companion paper, we apply our formalism to work out in detail the theory of the Gorsky effect (anelasticity due to long-range diffusion) for low interstitial concentrations, as an illustration of the advantages of our approach to the problem of anelastic relaxation.