Two-dimensional magnetic semiconductors provide a unique platform for exploring the interplay between charge, lattice, spin, and light. The strong Coulomb interaction combined with long-range magnetic order enables efficient coupling between excitons and both lattice vibrations and spin excitations, thereby strongly influencing exciton dynamics and transport in these materials. Here, we develop a microscopic many-body theory of exciton–phonon and exciton–magnon interactions in mono- and bilayer CrSBr. We demonstrate that the quasi-one-dimensional character of excitons in CrSBr gives rise to strongly anisotropic exciton–phonon and exciton–magnon scattering, resulting in direction-dependent diffusion coefficients. Furthermore, we identify a distinct temperature dependence originating from one-phonon and two-magnon scattering processes, providing characteristic signatures of the underlying microscopic interactions. Our work establishes a microscopic framework for understanding exciton transport in magnetic van der Waals semiconductors and provides guidance for future optical spectroscopy and exciton transport experiments.