Transition-metal chalcophosphates (TMCs) are two-dimensional (2D) van der Waals materials supporting a broad range of electronic and magnetic properties. Among quaternary TMCs, AgVP₂Se₆ is a rare ferromagnet (FM) with a triangular V sublattice, in contrast to the antiferromagnetic (AFM) zigzag chains of the sulfide AgVP₂S₆. Prior Raman studies of AgVP₂Se₆ have been performed far above the Curie temperature (T_C), preventing studies of spin-phonon coupling. Here, we report polarization-resolved, temperature-dependent Raman and magneto-Raman spectroscopy of AgVP₂Se₆ single crystals grown by chemical vapor transport (CVT) and flux methods, across the FM transition, complemented by density functional theory (DFT) calculations of phonon, optical and magnetic properties. Cryogenic Raman spectra resolve up to thirty peaks, while angle-resolved measurements enable their symmetry assignment. Spectral differences between CVT- and flux-grown crystals are traced back to the presence of different interlayer stacking domains in these samples. Temperature- and field-dependent Raman spectra remain largely unchanged across T_C, suggesting weak spin-phonon coupling. However, pronounced spectral alterations near 100 K suggest a potential structural phase transition. Our DFT calculations further reconcile the large discrepancy between the transport activation gap (≈ 0.325 eV) and optical absorption edge (≈ 2.14 eV), attributing the former to a transition to the V-d upper Hubbard band that is optically dark but thermally accessible. We also suggest a theoretical explanation of the qualitative difference (reproduced by DFT calculations) between the magnetic properties of the sulfide and selenide. These results provide key information regarding the lattice dynamics of AgVP₂Se₆ and will guide future applications in spin-based electronics.