Altermagnetism has emerged as a new class of magnetism with vanishing magnetization and macroscopic time reversal symmetry breaking. Hexagonal MnTe is a g-wave altermagnet with a 307 K Néel temperature, large spin splitting, and an anomalous Hall effect. However, the relationship between altermagnetism and the anomalous Hall effect in MnTe remains debated. Here, using optical magnetic circular dichroism, we observe hysteresis and a memory effect in a MnTe bulk crystal below 250 K. No hysteresis is seen after polishing the bulk crystal and in thin films. We conclude that the hysteresis in magnetic circular dichroism in the bulk crystal is from surface defects. The similar temperature dependence and a sign change between circular dichroism and the anomalous Hall effect indicate that the origin of the anomalous Hall effect in transport is likely not intrinsic.