To explore the correlation between college students’ social media behaviors and their mental health in the context of the network era in order to provide some empirical evidence for college students’ mental health education in the new period and new situation. In this paper, we first introduce the attention mechanism to optimize the LSTM network model to extract the features of college students’ online social media behaviors and combine it with the weighted cross-entropy cost function to improve the speed of the model when extracting behavioral features. Then, the fitness function of the genetic algorithm is used to analyze the distance between the feature samples, and the optimal features are selected through the selection strategy and cross-variance operation to obtain the college students’ online social media behavior. The analysis of the impact of social media behavior on mental health found that there is a significant difference in the level of misplaced fear and self-differentiation of students with different average use times of social media (P=0.024<0.05). The score of misplaced fear of students with a use time of less than 1h is only 22.18 points, and the level of self-differentiation reaches 119.86 points. The positive detection rate for depression was 82.61% among students who engaged in negative social media behavior for more than 6 hours. The analysis results of this paper provide reference data for managing college students’ social media behavior and new ideas for improving college students’ mental health.