The time-evolving matrix product operator (TEMPO) method has proven to be a powerful method to study the long-time dynamics of bosonic impurity problems where a small system is linearly coupled to a noninteracting bosonic bath. However, current developments of TEMPO have mostly focused on the case of diagonal system-bath coupling, i.e., ∑ₖ Aop(Vₖ bdopₖ + h.c.), with Aop a Hermitian operator of the system. Based on the process tensor framework, we extend TEMPO to the more general case of off-diagonal system-bath coupling in the form ∑ₖ (VₖAopbdopₖ + h.c.), where Aop could be non-Hermitian. As applications, we study the real-time dynamics of a spin that is coupled to a sub-ohmic bath via the Jaynes-Cummings-type system-bath coupling and compare it against the standard spin-boson model, where we show that the commonly used rotating-wave approximation could be very poor for this bath. We also study the imaginary-time evolution of a bosonic impurity with nonzero on-site interaction that is coupled to a sub-ohmic bath, to illustrate the flexibility of our method. Our method provides a unified framework to understand different variants of TEMPO, and is a promising building block for an impurity solver in the bosonic dynamical mean field theory for the normal phase with a scalar hybridization function.