Automatic tobacco grading systems can promote the standardization and unification of tobacco acquisition activities, improve production efficiency, and promote the long-term stable development of this industry. In this paper, we first extract the shape and color features of tobacco leaf structure, use a support vector machine to determine the difficult samples, add the difficult samples to the training set, and use the deep convolutional neural network method to assign initial values to the model, which reduces the occurrence of a model overfitting phenomenon and improves the prediction accuracy. Constructing a convolutional neural network based on EfficientNetV2-S for an automatic tobacco grading model. The performance of the model was tested on HSV and RGB images, and finally, automatic tobacco grading was applied to two batches of high-quality and good tobacco. It is found that the accuracy of the model training set and test set of this paper is 0.077 and 0.094 higher than that of the VGG16 model on HSV and RGB images, respectively, and there is no upward trend in the loss function. The model’s training operation time for grading two batches of tobacco is 27.56 seconds, which is more efficient. The accuracy rate of the two batches of tobacco is higher than 90%. The model in this paper in the practical application performance in addition to a higher degree of grading efficiency and accuracy, which is stronger than the manual identification in terms of efficiency and accuracy recognition stability. This paper’s research offers a feasible path and useful exploration to effectively improve production efficiency and optimize the process in China’s tobacco industry.