No study has focussed on the longitudinal effect of APOE-ɛ4 genotype on the logical memory delayed recall total (LDELTOTAL) score in late-onset Alzheimer’s disease (AD). The LDELTOTAL scores were collected at baseline, 12, 24, 36 and 48 months from 382 participants with AD, 503 with cognitive normal (CN), 1293 with mild cognitive impairment (MCI) in the Alzheimer’s Disease Neuroimaging Initiative (ADNI). A linear mixed model (LMM) was used to investigate the effect of APOE-ɛ4 on the longitudinal changes in the LDELTOTAL scores adjusted for age, gender, education and baseline Mini Mental State Examination score. There were significant differences in LDELTOTAL scores among AD, CN, and MCI (P<0.0001) and among APOE-ɛ4 alleles at baseline (P<0.0001). In the multivariable LMM, elders with 75+ years (P = 0.0051), females (P<0.0001), lower education (P<0.0001), AD and MCI (both P values <0.0001) were associated with decreased LDELTOTAL values, while the individuals with 1 or 2 APOE-ɛ4 allele revealed significantly lower LDELTOTAL scores (both P values<0.0001) compared with individuals without APOE-ɛ4 allele. Further, APOE-ɛ4 alleles had significant interactions with four follow-up visits, where all follow-up visits showed significantly higher LDELTOTAL score. In addition, gender showed interaction with age, education and APOE-ɛ4 with follow-up visits. Our findings provide the first evidence of the effect of APOE-ɛ4 genotype on the logical memory declines related to AD. Further, APOE-ɛ4 alleles showed interactions with gender and follow-up visits.