Manipulating magnetism with light is crucial for both fundamental understanding and technological advancements of information storage devices. The layered antiferromagnet CrSBr, part of the recently emerging class of two-dimensional van der Waals magnets, offers a unique path towards optical control of magnetism via magneto-excitons, which allow optical readout of the spin alignment and also provide a strong absorption channel. Here, we use exciton absorption and photoluminescence to demonstrate optical switching of the magnetic order in bi- and trilayer CrSBr. Using a continuous-wave laser with a power as low as a few microwatts at near-critical external magnetic fields, we demonstrate both local and remote switching of the magnetic configuration in extended lateral domains and further employ this mechanism to deterministically prepare the zero-field magnetic configuration. Our results establish optical switching as a useful means of controlling magnetism in CrSBr, with potential applications in magneto-optoelectronic devices.