Optomechanical atom-cavity interaction in the sub-recoil regime

We study the optomechanical interaction of a Bose-Einstein condensate with a single longitudinal mode of an ultra-high finesse standing wave optical resonator. As a unique feature the resonator combines three extreme regimes, previously not realized together, i.e., strong cooperative coupling, cavity dominated scattering with a Purcell factor far above unity, and sub-recoil resolution provided by a cavity damping rate smaller than four times the single photon recoil frequency. We present experimental observations in good agreement with a two-mode model predicting highly nonlinear dynamics with signatures as bistability, hysteresis, persistent oscillations, and superradiant back-scattering instabilities.