This paper analyzes the characteristics and changes of influence on college business English teaching in the context of the new era, constructs a multimodal college business English teaching model based on vision and hearing, and carries out a multimodal fusion of graphics, text and audio through deep learning. To solve this problem, the time series analysis problem is easily affected by noise. On the basis of the basic time series algorithm, the GBDT-KF algorithm based on the Kalman filter is designed. It also predicts the effectiveness of college business English teaching. The results show that listening ability is improved by 0.3, speaking ability by 0.25, writing ability by 0.21, and reading ability by 0.18; the effect of college business English teaching has also been improved in different degrees.