In this paper, we first establish a multimodal knowledge graph and process the course teaching video using by PaddleOCR algorithm to distinguish and understand the teaching text. This paper also improves the traditional classroom behavior recognition method by using the skeletal covariance matrix extraction strategy to aggregate the skeletal sequence features while combining the courseware information features to jointly achieve the recognition of classroom behavior. Based on the above algorithms, this paper summarizes the English multimodal teaching system based on intelligent learning, which improves traditional English teaching methods in terms of learning environment, teaching method, and mode. Finally, this paper conducted an experimental test on the effects of a multimodal English teaching method. The results show that the class scores of the class applying the multimodal English teaching method have been cumulatively improved by 22.06 points, and the rankings have been improved from 4 or 5 to 1 or 2 among the six tested classes. Students’ satisfaction with the pedagogy increased from an initial score of 3.4 to 4.5. The results show that multimodal English teaching promotes the change of English classrooms in the smart era.