In time-reversal symmetric systems such as topological and higher-order topological insulators, 1D spin-momentum locked edge and hinge states are theoretically ``perfectly conducting'', being immune to backscattering by non-magnetic disorder. Here, we reveal a fundamental ``topological fragility'': these states exhibit a bilinear magnetoelectric resistance significantly larger than in 2D systems. This effect requires two ingredients: (i) spin-momentum locking, which maximizes time-reversal symmetry breaking in the non-linear regime, and (ii) random spin-orbit interaction – the same mechanism behind Elliott - Yafet spin relaxation in heavy elements. Together, these generate a robust backscattering channel when a modest external magnetic field is applied. Our theory requires no gap opening or complex many-body effects, offering a simple and general mechanism that quantitatively explains recent observations in Bismuth hinge states.