Van der Waals heterostructures exhibit self-cavity resonances from sub-terahertz to terahertz frequencies and thus offer an attractive route toward realizing cavity-modified phases of matter. Spectroscopy of such systems calls for high frequency resolution at millikelvin temperatures, which is extremely challenging with existing methods. Here we demonstrate a new platform for realizing and probing such systems. Superconducting layers define self-cavity structures with electromagnetic volume compression of order 10⁷. With an integrated graphene layer, we observe underdamped cavity plasmon polaritons at frequencies as low as 0.04 THz and which exhibit avoided crossings with cavity modes. To measure such spectra, we deploy a millikelvin-compatible sub-THz spectrometer based on a superconducting circuit. This work paves the way for simultaneous spectroscopy and transport measurements of cavity-matter hybrid systems at millikelvin temperatures.