Spoken English teaching is a teaching scenario characterized by multimodality, which requires teachers’ verbal and nonverbal to cooperate. In the context of the high degree of integration of artificial intelligence and educational technology, the construction of an English speaking multimodal assessment method is of great significance for English speaking training and teaching. This paper constructs a multimodal teaching model of spoken English based on three modalities: audio, text, and video. Taking the English speech of Chinese college students as the research object, we improve the traditional computerized English pronunciation quality evaluation method, consider the multi-parametric evaluation indexes such as pitch, speech rate, rhythm and intonation, and convert the recognition problem into a scoring problem by utilizing the soft classification property of the ANN model. The confidence probability output value of the ANN model is used as the metric for evaluation results. Through experiments, it is proved that the accurate agreement rate of the spoken English assessment method in this paper is 89.87% and R2 is 0.528 under the gold standard, and the data results prove that the assessment method is credible. It can give learners of spoken English timely, accurate and objective evaluation and feedback guidance and help learners find out the differences between their pronunciation and the standard pronunciation so as to improve the efficiency of spoken English learning.