With the rapid development of science and technology, the use of artificial intelligence in the field of higher education is becoming more and more common. The use of intelligent speech recognition technology in English teaching in colleges and universities is on the rise. The research utilizes speech recognition logic to create a hybrid CTC/Attention model for teaching spoken English in colleges and universities. Then we compare and analyze the performance of artificial speech recognition models, such as the HMM model and CTC/Attention model, and explore the effects of teaching English speaking in colleges and universities using this model. In terms of the overall recognition rate, with the increase of the test vocabulary, the recognition rates of both the artificial speech recognition model and the HMM model decrease, while the CTC/Attention model shows an increasing trend. Except for intonation, the CTC/Attention model has the best recognition rate performance across all dimensions. Through the college-speaking teaching experiment based on the CTC/Attention model, the students’ total English speaking performance, accuracy, fluency, and complexity were improved. The Sig values of total speaking achievement, accuracy, fluency, and complexity are less than 0.05. College English-speaking teaching with intelligent speech recognition has a significant positive impact on the improvement of student’s English-speaking achievement, as indicated.