Abstract
This thesis elaborates the exotic two-dimensional ultracold states, in particular the superfluidity and its phase transitions, for both bosonic and fermionic systems. In Part I, the basic concepts of ultracold systems, such as Bose-Einstein condensation, superfluid, the BCS-BEC crossover, and polar molecules, are briefly mentioned as an overview for pedagogical purposes. In Part II, the two-dimensional bosonic systems in optical lattices are investigated via path integral quantum Monte Carlo method. In Part III, the fermionic systems in the presence of long-ranged interaction are investigate for the unconventional properties of superfluidity. In Chapter 1, the single-band Bose-Hubbard model and its phase diagram are introduced to describe bosons in optical lattices. We briefly review the main features of the phase transitions in Bose-Hubbard model, namely, the superfluid-to-Mott insulator quantum phase transition at zero-temperature and the superfluid-to-normal Berezinskii-Kosterlitz-Thouless transition at finite temperature. Then, the quantum Monte Carlo method is introduced in the following section of the same Chapter. After a very brief overview on how Monte Carlo methods work in general, we focus on the construction of the path integral quantum Monte Carlo, based on the path integral to build up the algorithm with the graphical worldline diagram, together with the common numerical issues. In Chapter 2, by probing the correlation functions, we investigate the critical behaviors of the superfluid-to-normal and superfluid-to-Mott insulator transitions in two-dimensional Bose-Hubbard model. In particular, by observing the divergence laws of the correlation length, the critical regimes of the Berezinskii-Kosterlitz-Thouless transition and of quantum phase transition are distinguished. Then the quantum critical regimes in the vicinity of quantum phase transition are investigated as well. Later on we discuss the possible quantities and necessary conditions to observe the critical regimes in experiments. In Chapter 3, we extend the path integral quantum Monte Carlo in two different perspectives. In the fi rst section, we consider anisotropic two-dimensional cases. We observe the change of the superfluid transition temperature from low anisotropy to high anisotropy, since it is expected to vanish in pure 1D, and compare with the analytical results from self-consistent harmonic approximation. In the second section, we start from the regime where the few-body interaction is so strong that the single-band Bose-Hubbard model needs corrections. We analyse the interaction-dependent tunneling strength, and generalize the Bose-Hubbard model with occupation-dependence. In Chapter 4, we fi rst introduce the Fulde-Ferrell-Larkin-Ovchinnikov (FFLO) state, which is an unconventional superfluid/superconducting state occurring when the Fermi surfaces of different component are imbalanced. Then we explain the particular interests and the benefi ts of the polar molecules in layered structures, particularly in the realization of the BCS-BEC crossover in fermionic cases. The experimental progresses and the theoretical proposals of the polar molecules in layered structures are also mentioned. In Chapter 5, we analyse the FFLO state made by polar molecules in bilayer system via various methods including the Bogoliubov transformation, Green's functions and Ginzburg-Landau functional. We demonstrate how the dipolar interaction can enhance the stability of FFLO state via the p-wave channel, at both zero- and finite-temperature. The possible structures of the FFLO state are investigated as well, and we show the possibility to realize such state in realistic parameters of experiments. In Chapter 6, as an extension, we show the interest of loading attractive fermionic Rydberg atoms in a bilayer geometry. In particular, the Cooper pairs formed in the same layer can intertwine the interlayer Cooper pairs. The competition, and moreover the coexistence, of the two kinds of Cooper pairs are investigated.