The detection of teaching factors is mainly from the perspectives of both teachers and students, and the multi-feature fusion of the teaching factor detection method is used to add possibilities for the improvement of the quality of blended teaching of college English. The use of auditory features in this paper is to separate the teaching scene video and obtain the teacher and student video scenes. Teacher and student voice clustering algorithm based on teaching rules is proposed to complete the feature division, and the combination of visual and auditory features is used to extract the blended teaching behaviors of college English and classify the types of teacher and student behaviors and behavioral probabilities. Create a multi-feature fusion teaching factor detection model by optimizing the multi-feature fusion process and multi-feature fusion strategy. Design an experimental configuration to detect multi-feature teaching factors that affect students’ concentration, attention, and teaching state. The multi-feature fusion model’s recognition accuracy for excellent concentration, excellent concentration, and poor concentration all reached 75.29% and above, and the probability of correctly recognizing excellent concentration was the greatest, gaining 96.80%.