We study topological band structures and superfluid phases in two-dimensional square-lattice systems with homogeneous SU(N) non-Abelian gauge fields. Starting from an SU(4) gauge-field model related to the Hofstadter model with flux α=1/4, we show that the lowest and highest bands are isolated Chern bands that carry Chern numbers C=-4 and give rise to four chiral edge modes in a strip geometry. Remarkably, although the two middle bands touch and form 16 gapless Dirac cones, their combined Chern number is C=8. We then generalize the construction to SU(N) systems and reveal an even–odd structure of the band topology: when N is odd, all bands are isolated and carry nonzero Chern numbers; when N is even, the two middle bands touch at N² Dirac points, while all other bands remain isolated and topologically nontrivial. We find that the uppermost and lowermost bands become increasingly flat and their Berry curvature becomes more uniform as N increases, providing a promising platform for realizing fractional Chern insulating phases. We further examine the spin-3/2 SU(4) model with on-site attractive Hubbard interactions, exploring its superfluid phases at partial filling. We find that the non-Abelian gauge field breaks the hidden SO(5) degeneracy of the quintet pairing and selects distinct nematic superfluid states. For fillings in the middle-band regime, the resulting spectrum can host topologically protected Bogoliubov Fermi surfaces. Our results provide a starting point for exploring both topological band physics and unconventional superfluidity in synthetic SU(N) cold-atom systems.