In order to improve the quality of teaching spoken English, this paper reconstructs the semantic structure through the weight matrix, and the association matrix describes the mapping process in the context of computer-assisted language learning. Pooling layer vectors are computed, and translation transformation matrices are encoded in real time so as to extract language vector features. Feature regression evaluation is implemented in a restricted Boltzmann machine to enhance the decoding language target through human-computer interaction, described using nonlinear functions, and the text to be translated is input to the deeply structured semantics, which is processed by weighted averaging to form a spatial vector to complete the input layer mapping process for the utterance. To verify the feasibility of this teaching model, the effectiveness of the teaching strategy is analyzed. The results showed that the variance of spoken English of college students with computer assistance was 0.534, the memory rate increased by at least 0.07, and the value of the number of words memorized by college students increased by more than 98.72%. It can be seen that the proposed information-based teaching model of spoken English in colleges and universities breaks the traditional education method and makes the teaching strategy approach more diversified.