The charge state stability of nitrogen-vacancy (NV) centers critically affects their application as quantum sensors and qubits. Understanding charge state conversion and equilibration is important not only for NV centers in diamond but also for defects and impurities in wide-bandgap materials in general. The mechanisms by which these centers change charge state upon optical or electronic excitation without the presence of mobile carriers remain unclear. Here, we elucidate this issue for the case of photoionization of NV center ensembles. Using pump-probe spectroscopy, we ionize negatively charged NV centers and monitor the recovery of NV − on timescales of up to several seconds. We find that the recovery rate depends strongly on the concentration of surrounding nitrogen donors. Remarkably, the equilibration dynamics exhibit no discernible dependence on temperature, ruling out thermally activated processes. We develop a multiphonon-assisted electron-tunneling model, supported by density-functional calculations, that explains the measurements and identifies tunneling as the equilibration mechanism. Because tunneling rates depend exponentially on donor separation, the random distribution of nitrogen donors spreads recovery times from nanoseconds to hours or longer.