We successfully stabilized a Fabry-Pérot optical cavity incorporating a bulk diamond crystal at millikelvin temperatures in a cryogen-free dilution refrigerator with the pulse-tube cryocooler running. In stark contrast to previous demonstrations where lasers were locked to the cavities, our setup locks the cavity to a laser. Our measurements of cavity length fluctuation suggest that the setup could stabilize a cavity up to a finesse of 1.2× 10⁴ without the diamond and 5.8 ×10³ with the diamond crystal. The finesse with a diamond crystal of approximately 90 is primarily limited by the absorption loss inside the diamond.