Precise radial velocities of giant stars VI. A possible 2:1 resonant planet pair around the K giant star η Ceti

We report the discovery of a new planetary system around the K giant eta Cet (HIP 5364, HD 6805, HR 334) based on 118 high-precision optical radial velocities taken at Lick Observatory since July 2000. Since October 2011 an additional nine near-infrared Doppler measurements have been taken using the ESO CRIRES spectrograph (VLT, UTI). The visible data set shows two clear periodicities. Although we cannot completely rule out that the shorter period is due to rotational modulation of stellar features, the infrared data show the same variations as in the optical, which strongly supports that the variations are caused by two planets. Assuming the mass of 77 Cet to be 1.7 M-circle dot, the best edge-on coplanar dynamical fit to the data is consistent with two massive planets (m(b) sin i = 2.6 +/- 0.2 M-jup, m(c) sin i = 3.3 +/- 0.2 M-jup), with periods of P-b = 407 +/- 3 days and P-c = 740 +/- 5 days and eccentricities of e(b) = 0.12 +/- 0.05 and e(c) = 0.08 +/- 0.04. These mass and period ratios suggest possible strong interactions between the planets, and a dynamical test is mandatory. We tested a wide variety of edge-on coplanar and inclined planetary configurations for stability, which agree with the derived radial velocities. We find that for a coplanar configuration there are several isolated stable solutions and two well defined stability regions. In certain orbital configurations with moderate e(b) eccentricity, the planets can be effectively trapped in an anti-aligned 2: 1 mean motion resonance that stabilizes the system. A much larger non-resonant stable region exists in low-eccentricity parameter space, although it appears to be much farther from the best fit than the 2: 1 resonant region. In all other cases, the system is categorized as unstable or chaotic. Another conclusion from the coplanar inclined dynamical test is that the planets can be at most a factor of similar to 1.4 more massive than their suggested minimum masses. Assuming yet higher inclinations, and thus larger planetary masses, leads to instability in all cases. This stability constraint on the inclination excludes the possibility of two brown dwarfs, and strongly favors a planetary system.