This paper firstly focuses on the automatic detection of mental health in colleges and universities and the sequential quadratic programming algorithm, proposes a psychological abnormality detection model for college students based on the sequential quadratic programming algorithm, and in the feature vectorization process, uses constraint functions and coefficient matrices to process various types of features into vector sequences of length k weeks to help quantify the trajectory of students’ school activities, and trains and establishes a student psychological abnormality detection model. Then, on the technical basis of the psychological abnormality detection model, the mental health education system is constructed, the online and offline integration teaching development is realized through the teaching system, and the performance of the education system is verified. The results show that the average score of the traditional psychological teaching model 1-10 week test is 76.52, compared to the average score of 78.51 in the mental health education system teaching 1-10 test, with an error value of 1.89. This study uses the mental health teaching system to retrieve information on students with current mental health problems at any time, and there this is to realize dynamic online and offline coeducation.