Classification of small lesions is of great importance for early detection of breast cancer. The small size of lesion makes handcrafted features ineffective for practical applications. Furthermore, the relatively small data sets also impose challenges on deep learning based classification methods. Dynamic Contrast Enhancement MRI (DCE-MRI) is widely-used for women at high risk of breast cancer, and the dynamic features become more important in the case of small lesion. To extract more dynamic information, we propose a method for processing sequence data to encode the DCE-MRI, and design a new structure, dense convolutional LSTM, by adding a dense block in convolutional LSTM unit. Faced with the huge number of parameters in deep neural network, we add some semantic priors as constrains to improve generalization performance. Four latent attributes are extracted from diagnostic reports and pathological results, and are predicted together with the classification of benign or malignant. Predicting the latent attributes as auxiliary tasks can help the training of deep neural network, which makes it possible to train complex network with small size dataset and achieve a satisfactory result. Our methods improve the accuracy from 0.625, acquired by ResNet, to 0.847.