Breakdown of gauge invariance in ultrastrong-coupling cavity QED

We revisit the derivation of Rabi- and Dicke-type models, which are commonly used for the study of quantum light-matter interactions in cavity and circuit QED. We demonstrate that the validity of the two-level approximation, which is an essential step in this derivation, depends explicitly on the choice of gauge once the system enters the ultrastrong-coupling regime. In particular, while in the electric dipole gauge the two-level approximation can be performed as long as the Rabi frequency remains much smaller than the energies of all higher-lying levels, it can dramatically fail in the Coulomb gauge, even for systems with an extremely anharmonic spectrum. We extensively investigate this phenomenon both in the single-dipole (Rabi) and multidipole (Dicke) case, and consider the specific examples of dipoles confined by double-well and square-well potentials, and of circuit QED systems with flux qubits coupled to an LC resonator.