Exotic quantum phase transitions of strongly interacting topological insulators

Using determinant quantum Monte Carlo simulations, we demonstrate that an extended Hubbard model on a bilayer honeycomb lattice has two novel quantum phase transitions. The first is a quantum phase transition between the weakly interacting gapless Dirac fermion phase and a strongly interacting fully gapped and symmetric trivial phase, which cannot be described by the standard Gross-Neveu model. The second is a quantum critical point between a quantum spin Hall insulator with spin S-z conservation and the previously mentioned strongly interacting fully gapped phase. At the latter quantum critical point the single-particle excitations remain gapped, while spin and charge gaps both close. We argue that the first quantum phase transition is related to the Z(16) classification of the topological superconductor He-3-B phase with interactions, while the second quantum phase transition is a topological phase transition described by a bosonic O(4) nonlinear sigma model field theory with a Theta term.