Aluminum nitride-based ferroelectrics combine high remanent polarization, thermal stability, weak size effects, a wide bandgap and back-end process compatibility, making them promising for high density non-volatile memory. However, their high coercive field (Ec) and unclear polarization switching mechanism limit further applications. Theoretical predictions indicate preferential reversed domain growth along the polar axis, whereas experiments show nearly isotropic growth or preferential growth perpendicular to this axis. Here, we combine density functional theory, Deep Potential molecular dynamics and experimental characterization to determine how nitrogen vacancies (VN) regulate domain growth and Ec. In ideal crystal, strong lattice coupling along the polar axis drives preferential domain growth in this direction. VN locally weakens this coupling and promotes polarization nucleation. The spatial distribution of VN and the temporal sequence of vacancy induced switching jointly determine the direction and morphology of reversed domain growth. We further show that scandium doping promotes VN formation, thereby enhancing defect mediated nucleation and reducing Ec. These findings establish a multiscale mechanism governing thin-film domain-growth kinetics and macroscopic Ec, reconciling experimental observations with ideal crystal simulations and providing defect engineering principles for low-voltage, reliable and scalable ferroelectric devices.