The post-reionization epoch (z∼4-6) marks a critical phase in galaxy evolution, during which galaxies rapidly assemble their stellar mass, enrich their interstellar medium, and build up substantial dust reservoirs. The physical processes responsible for such rapid dust enrichment remain uncertain. We investigate the dust enrichment history of the ALPINE-CRISTAL-JWST (ACJ) sample using the semi-analytic galaxy formation model GAlaxy Evolution and Assembly (GAEA), which follows galaxy assembly, chemical enrichment, and dust evolution. We select model galaxies matching the observed stellar masses, star formation rates, and metallicities of the ACJ sample, compare their dust properties with observed scaling relations, and trace their enrichment histories and environments. The model reproduces the observed dust mass-stellar mass relation and the dust-to-stellar mass, dust-to-gas, and dust-to-metal ratios as functions of stellar mass and metallicity. Reproducing these constraints requires efficient dust growth in the interstellar medium; stellar dust production alone cannot account for the observed dust reservoirs, even under the most optimistic assumptions. The combination of stellar mass, star formation rate, and metallicity provides a good predictor of galaxy dust content. Our model suggests rapid dust build-up consistent with the previously noted rapid metal enrichment, with evidence for a transition around z∼8-9 below which dust growth becomes the dominant dust formation channel. Finally, ACJ-like galaxies preferentially reside in locally overdense environments, suggesting that environment may have contributed to their accelerated evolution. Taken together, these results support rapid and coupled stellar, metal, and dust build-up in ACJ galaxies during the post-reionization era.