The surface of semiconductor materials can produce defects such as scratches during use, significantly affecting their performance. In this paper, we use advanced machine learning techniques to detect defective regions on the surface of semiconductor materials by employing the Canny operator. The characteristics of defects on the surface of semiconductor materials, such as geometry, grayscale, and texture, are extracted. Based on the TensorFlow framework, a machine learning model for recognizing defects on the surface of semiconductor materials has been established. The model in this paper can achieve 94.53% accuracy in the comprehensive recognition of eight types of defects on the surface of semiconductor materials. In addition to random-type defects, the recognition accuracy of this paper’s model for the other 7 types of defects is above 94.59%. The model shows the best performance in the task of recognizing six types of semiconductor material surface defect patterns, namely center, torus, marginal local, edge ring, local, and random, and its F-value reaches 94.08 and 88.40 for the two types of defect patterns, namely nearly full and scratches. This is close to the highest F-value of all algorithms for the recognition of these two defects. In the five-fold cross-validation, the defect recognition accuracy of this paper’s model was as high as 96.82%, which fully demonstrates the advanced performance of this model in the task of recognizing defects on semiconductor material surfaces.