Two-dimensional infrared (2D IR) spectroscopy is a powerful nonlinear technique for probing vibrational couplings, anharmonicities, coherence, and molecular dynamics. Among the microscopic parameters encoded in 2D IR spectra, transition dipoles are fundamental quantities that characterize molecular structure and govern light-matter interactions. However, reconstructing these quantities from spectroscopic data remains a challenging inverse problem. Here, we introduce a physics-informed convolutional neural network (CNN), Spec2DipoleNet, trained on 2D IR spectra generated via the non-Hermitian Hamiltonian (NHH) method. This multi-task CNN reconstructs the transition dipole matrices under physically-motivated masking constraints. On the validation set, the model achieves a root-mean-square error (RMSE) of 0.085, an R² of 0.97, and a normalized cross-correlation of 0.98. It reliably recovers matrix elements associated with allowed transitions, while larger deviations are observed for nominally forbidden transitions. Our results suggest that transition dipole operators can be quantitatively inferred from 2D IR spectra, highlighting the potential of physics-informed deep learning for extracting microscopic quantum parameters from multidimensional spectroscopic data.