First, This paper constructs a multimodal teaching model based on teaching contents, activities, learning environment and learning means and examines measures to illustrate teaching English and American literature under language acquisition theory. Secondly, it uses Fourier transform and discrete cosine transform to preprocess audio signals, extracts emotional features from MFCC according to the bidirectional long and short-term memory network algorithm of attention, proposes to use representational learning and attention mechanism to focus on the detected emotional regions, and analyzes the multimodalized teaching of English and American literature course and the emotion of teaching speech reading in multimodal English and American literature. The results showed that on the multimodalized English and American literature course, the experimental class had mean scores of 21.81, 8.04, 6.71, and 7.08 on each test, respectively, and the experimental class outperformed the control class on all tests of English and American literature instruction. On the analysis of speech reading aloud emotion evaluation, the model assessment score and teacher assessment error were 2.3, the correlation coefficient was 0.6982, and the overall performance of the model of speech reading aloud emotion analysis of English and American literature was relatively reliable. Students' literary emotional literacy is developed through this study, and a multidimensional interpretation of English and American literature is achieved.