Research on intelligent analysis and evaluation of teachers’ teaching behavior still suffers from the lack of teacher behavior datasets in real classroom scenarios, difficulties in migrating existing behavior recognition algorithms, and low accuracy of teacher classroom behavior recognition. This paper focuses on the research of teacher teaching behavior for real teaching scenarios, explores the effective method of teacher teaching behavior recognition based on deep learning, and provides technical support for carrying out the realization of personalized teaching assistance, intelligent evaluation and decision-making. First, the spatial and temporal characteristics of each English teacher’s goal are extracted by constructing a deep learning-based model for detecting and tracking English teacher goals. Then, on the basis of Multi-Stream Adaptive Graph Convolutional Network (MS-AAGCN), the mechanism of Graph Attention Network (GAT) is embedded to construct a model for recognizing the teaching behavior of classroom English teachers. Finally, the model’s experimental results and its application to real teaching scenarios are examined. As the number of iterations increases, the loss function value of the model gradually tends to 0, while the accuracy rate tends to 100%, indicating that the model training effect is better. On the validation dataset, this paper’s model achieves a recognition accuracy of 87.61%, which are higher than the other comparison models, indicating that the teacher behavior analysis network model constructed in this paper is suitable for analyzing the teaching behavior of classroom teachers.