Using supersonic expansion, we create a time-gated, directional beam of Xe atoms with a narrow momentum distribution and detect their collisions with a levitated nanosphere cooled to its quantum ground state. We observe individual momentum kicks below 50 keV/c with 96% confidence and distinguish directional momentum transfer from the atomic beam against the background of thermal collisions. Our experiments constitute a first step toward the exploration of distance-dependent short-range interactions between atoms and levitated systems, as well as toward the use of levitated platforms for impulsive force sensing in previously unexplored parameter regimes.