Erbium ions (Er³⁺) in cerium dioxide (CeO₂) represent a promising spin-photon interface for quantum communication, but the mechanisms limiting their optical coherence remain poorly understood. Using periodic hybrid density functional theory calculations with finite-size corrections, we identify Ce³⁺ polarons and their complexes with oxygen vacancies and Er³⁺ dopants as likely sources of optical decoherence. These defects exhibit finite photoionization cross-sections at 0.8 eV, coinciding with both the laser excitation energy used experimentally and the emission energy of Er³⁺. This resonance enables photoionization of the polarons and photoluminescence quenching of Er³⁺, leading to the broadening of optical linewidths, shortening of excited-state lifetimes, and introduction of charge noise. Our concentration-dependent photocurrent measurements in Er³⁺-doped CeO₂ films under 0.8 eV illumination validate the predicted decoherence pathway. Our combined computational and experimental results identify a concrete defect-engineering target for improving the Er³⁺-doped CeO₂ platform, and point to a decoherence mechanism likely relevant to other Er³⁺-doped multivalent-oxide quantum platforms.