Probe-based confocal laser endomicroscopy (pCLE) is an emerging tool for epithelial cancer diagnosis, which enables in-vivo microscopic imaging during endoscopic procedures and facilitates the development of automatic recognition algorithms to identify the status of tissues. In this paper, we propose a transfer recurrent feature learning framework for classification tasks on pCLE videos. At the first stage, the discriminative feature of single pCLE frame is learned via generative adversarial networks based on both pCLE and histology modalities. At the second stage, we use recurrent neural networks to handle the varying length and irregular shape of pCLE mosaics taking the frame-based features as input. The experiments on real pCLE data sets demonstrate that our approach outperforms, with statistical significance, state-of-the-art approaches. A binary classification accuracy of 84.1% has been achieved.