Time-resolved infrared (TR-IR) spectroscopy reveals molecular structural changes and charge-carrier dynamics on their intrinsic timescales, in photosynthesis, catalysis and optoelectronic materials, but requires complex sources of ultrashort infrared pulses, cryogenically cooled detectors to overcome thermal background, and intense probes that can perturb the sample. Quantum infrared spectroscopy avoids these constraints: harnessing quantum interference between the generation processes of frequency-entangled photon pairs, it reads out the infrared response of a sample by detecting only the visible partner photon, for which efficient detectors exist. To date, it has been limited to steady-state measurements. Here we demonstrate its ultrafast, time-resolved form with a picosecond visible pump pulse and a silicon detector. Each visible pulse generates photon pairs within a picosecond window; the infrared photon of a pair probes the excited sample at a set delay after excitation, and the transient response is read out from the count rate of its visible partner. We resolve picosecond carrier dynamics in a semiconductor saturable absorber mirror near 1529 nm with a picowatt-level probe, unaffected by thermal background and by sample IR photoluminescence 50 times stronger than the probe. The approach is applicable to the mid-infrared, fingerprint region, where infrared detection is hardest, and to the terahertz region.