Spin-orbital order modified by orbital dilution in transition-metal oxides: From spin defects to frustrated spins polarizing host orbitals

We investigate the changes in spin and orbital patterns induced by magnetic transition-metal ions without an orbital degree of freedom doped in a strongly correlated insulator with spin-orbital order. In this context, we study the 3d ion substitution in 4d transition-metal oxides in the case of 3d3 doping at either 3d2 or 4d4 sites, which realizes orbital dilution in a Mott insulator. Although we concentrate on this doping case as it is known experimentally and more challenging than other oxides due to finite spin-orbit coupling, the conclusions are more general. We derive the effective 3d-4d (or 3d-3d) superexchange in a Mott insulator with different ionic valencies, underlining the emerging structure of the spin-orbital coupling between the impurity and the host sites, and demonstrate that it is qualitatively different from that encountered in the host itself. This derivation shows that the interaction between the host and the impurity depends in a crucial way on the type of doubly occupied t2g orbital. One finds that in some cases, due to the quench of the orbital degree of freedom at the 3d impurity, the spin and orbital order within the host is drastically modified by doping. The impurity either acts as a spin defect accompanied by an orbital vacancy in the spin-orbital structure when the host-impurity coupling is weak or favors doubly occupied active orbitals (orbital polarons) along the 3d-4d bond leading to antiferromagnetic or ferromagnetic spin coupling. This competition between different magnetic couplings leads to quite different ground states. In particular, for the case of a finite and periodic 3d atom substitution, it leads to striped patterns either with alternating ferromagnetic or antiferromagnetic domains or with islands of saturated ferromagnetic order. We find that magnetic frustration and spin degeneracy can be lifted by the quantum orbital flips of the host, but they are robust in special regions of the incommensurate phase diagram. Orbital quantum fluctuations modify quantitatively spin-orbital order imposed by superexchange. In contrast, the spin-orbit coupling can lead to anisotropic spin and orbital patterns along the symmetry directions and cause a radical modification of the order imposed by the spin-orbital superexchange. Our findings are expected to be of importance for future theoretical understanding of experimental results for 4d transition-metal oxides doped with 3d3 ions. We suggest how the local or global changes of the spin-orbital order induced by such impurities could be detected experimentally.