We prove that the colored Hofstadter butterfly has a many-body interpretation for a broad class of weakly interacting lattice fermion systems. Starting from a spectral gap of a Hofstadter-like one-particle Hamiltonian at arbitrary magnetic flux b, we construct an open region in the three-dimensional parameter space (b,μ,λ) of magnetic field, chemical potential, and interaction strength on which the infinite-volume interacting system has locally unique gapped ground states. The construction combines quasi-adiabatic continuation in the interaction strength with denominator-independent magnetic perturbation estimates, and therefore covers both commensurate and incommensurate fluxes, where no finite magnetic unit cell exists. On connected uniformly gapped regions meeting the non-interacting plane λ=0, we prove a many-body gap-labeling theorem: the Hall conductivity appearing in the macroscopic Ohm's law is constant and quantized, satisfying 2πσ^H∈Z. Thus the integer colors of the non-interacting Hofstadter butterfly persist as Hall-conductivity labels of interacting quantum Hall phases.