This paper analyzes the optimization problem of the English teaching process, uses the proposed Newton algorithm to solve the optimization problem, and provides the calculation process for the BFGS algorithm. Based on the optimization theory, the principle of optimization of English teaching in colleges and universities is discussed, and the mixed teaching mode of English in colleges and universities is constructed. To verify the practical effectiveness of the application of the English mixed teaching mode in English teaching work, an empirical analysis was carried out. The results show that the English hybrid teaching mode based on the optimization theory can effectively improve students’ English oral expression ability, and the difference in oral accuracy of the experimental class compared with the control class is 11.98 points, and 20.41 points improve fluency. It can also provide effective help for students to improve the content, language, and chapter structure of English writing and promote the innovative development of English teaching in colleges and universities.