Intelligent methods such as deep learning provide an innovative optimization path for English teaching in colleges and universities. In this paper, the attention mechanism is based on traditional CNN. Embedded attention and multilayer convolutional feature fusion are used to complete the design of streamlined bilinear attention networks. The streamlined bilinear attention network is used to obtain the characteristics of students’ classroom behavior in the process of English teaching, combine the characteristics of students’ classroom behavior and computer programming language to design and implement the intelligent interactive teaching system of English in colleges and universities, and test and apply the analysis of the intelligent interactive teaching system of English. The results show that in the query performance test report, even though the number of requests issued in the test is 2,000, the error rate is always kept at 0, the throughput is 39.8, and the response time is kept at 95~1144s, which indicates that this system can meet the performance requirements with a moderate number of users. In the application analysis, the total score of the teaching method of English Intelligent Interactive Teaching System (mean=94.62) is higher than that of the traditional English teaching method (mean=89.65), which indicates that the teaching method of English Intelligent Interactive Teaching System is more capable of improving the interaction and communication ability of the students in English teaching. The system constructed in this study can meet the diverse needs of students and contribute to the quality improvement of English teaching in colleges and universities.