Due to the inherent scattering in topological edge states, the topological resonant cavity (TRC) formed by their interference exhibits a relatively low quality factor (Q). To address this issue, we propose the mechanism of engineering the transverse evanescent depth of edge states to achieve high-Q in irregular shaped TRCs, which further enables strong photon-exciton coupling at the single-quantum level. By adjusting the intra- and inter-cell coupling strengths in the pseudo-spin topological photonic crystal, the transverse evanescent depth of edge states is greatly reduced, thereby suppressing the scattering of TRCs. Consequently, the Q of various shaped TRCs with 2 ~ 3 um size is up to 10^5 ~ 10^6 in the visible, so that the strong photon-exciton coupling is achieved. The high-Q topological cavities hold promising application in integrated lasers, sensors, cavity quantum electrodynamics, and quantum photonics.