Virtual reality technology helps to improve students’ engagement and interest in learning and has a good potential to be combined with the mental health education of college students in higher vocational colleges and universities. This paper aims to address the current emotion deficiency problem in virtual reality education by designing a cascade classifier for expression classification training and proposing an ICABoost expression classification algorithm. According to the expression classification results, the incentive factors are designed to detect and motivate learned emotions. Furthermore, a proposal is made for applying the AER emotion interaction algorithm to teaching. The ICABoost classification algorithm is experimentally tested to have an F1 value of 93.52%, a recall rate of 93.50% and an accuracy rate of 95.10%, which is excellent among all similar algorithms. Applying improved virtual reality technology as a mental health teaching method in this paper significantly improves students’ mental health, and students generally have a high degree of acceptance of the new method.