Classroom learning behavior analysis is an important means for teachers to understand the classroom situation, which helps teachers grasp the learning situation of each student, adjust teaching strategies, and then improve teaching quality. The article first collects lecture videos from real English classroom teaching in universities, summarizes six typical classroom behaviors, and designs and constructs a student classroom learning behavior dataset for the classroom learning behavior recognition model. In order to better integrate the spatio-temporal features of students’ classroom learning behaviors, an end-to-end student classroom learning behavior recognition model based on the dual-attention mechanism integrating spatio-temporal features (DA-YOWO) is designed. Through experimental comparison with YOLOv5 and FasterR-CNN models, it has been verified that the model in this paper has a high behavioral recognition accuracy for student learning behaviors in the English classroom. Then, in the real classroom scenario, the comprehensive weights of the student behavior indicators were determined by principal component analysis and factor analysis, and the analysis of student learning engagement was carried out. According to the results of the analysis, the learning status of each student in the English classroom was clarified so that the teachers had enough time to reflect on the deficiencies and problems in their teaching process according to the student’s performance in the classroom, which helped the teachers to better grasp the whole classroom.