The advent of the big data era brings opportunities and challenges to the innovation of mental health education for college students in higher education institutions. In this paper, we monitor the mental health of college students in higher education institutions by collecting students’ consumption data, access control data, network data and historical achievement data, and use a convolutional neural network to extract features from behavioral data and then input the feature set into random forest model for training. The recognition effect of the RF algorithm is analyzed by comparing it with other algorithms, and the problems of college students’ mental health education are analyzed based on the correlation of features. The accuracy of identifying mental health problems of college students based on RF analysis of big data reached 0.72, the recall rate reached 0.58, and the F1-Measure was 0.67. For the correlation of different features, the most significant correlation effect was the time spent in the dormitory on rest days and the number of swiping access cards, with the correlation coefficients reaching 0.2719 and -0.2191, respectively. The analysis based on big data can accurately grasp the current situation and problems of mental health education of college students in higher education institutions and promote the comprehensive development of mental health education of college students.