Manual diagnosis of crops diseases is not an easy process; thus, a computerized method is widely used. From a couple of years, advancements in the domain of machine learning, such as deep learning, have shown substantial success. However, they still faced some challenges such as similarity in disease symptoms and irrelevant features extraction. In this article, we proposed a new deep learning architecture with optimization algorithm for cucumber and potato leaf diseases recognition. The proposed architecture consists of five steps. In the first step, data augmentation is performed to increase the numbers of training samples. In the second step, pre-trained DarkNet19 deep model is opted and fine-tuned that later utilized for the training of fine-tuned model through transfer learning. Deep features are extracted from the global pooling layer in the next step that is refined using Improved Cuckoo search algorithm. The best selected features are finally classified using machine learning classifiers such as SVM, and named a few more for final classification results. The proposed architecture is tested using publicly available datasets–Cucumber National Dataset and Plant Village. The proposed architecture achieved an accuracy of 100.0%, 92.9%, and 99.2%, respectively. A comparison with recent techniques is also performed, revealing that the proposed method achieved improved accuracy while consuming less computational time.