Currently, intelligent speech recognition technology has become an important tool to improve the quality of English teaching, and the flipped classroom combined with intelligent speech recognition technology not only optimizes the teaching process but also enhances students’ listening and speaking abilities. This paper briefly introduces the text representation and classification technology used in speech recognition results to extract MFCC speech features. Using the acoustic model of long and short-term memory networks and the neural network as the language model, an improved intelligent speech recognition model (LSTM-NNLM) is constructed based on English teaching. The model is applied to a smart classroom at a university to achieve deep integration between the two. Through examples, we analyze whether the model in this paper has practical application value. In this paper, the loss and error rates of the intelligent speech recognition model converge to 0.13 and 0.10, respectively, during the 25 rounds of training. Speech recognition technology can be used in the English flipped classroom to improve students’ speaking levels. The model in this paper has the most effective effect on students’ speaking scores. Meanwhile, the use of speech recognition technology in flipped classrooms can improve students’ interest in English classes. To conclude, the use of intelligent speech recognition technology in English flipped classrooms can enhance teaching efficiency and increase interactivity.