This paper selects the evaluation indexes that fit with the teaching mode and evaluation objectives of university innovation and entrepreneurship education in the new media environment. The limitations of the existing evaluation indexes are analyzed and combined with the characteristics of students’ physical and mental development and cognitive level; the indexes are screened and integrated, and a new evaluation system is constructed. Based on the improved particle swarm algorithm, the BP neural network is optimized to establish the evaluation model of students’ innovation ability and improve the defects of the particle swarm algorithm, covering the selection of the network topology, the number of neurons in each layer, the setting of the parameters of the improved particle swarm algorithm, and the dynamic adjustment of the weights and thresholds, so as to build the evaluation model. Analyzing the influencing factors of college innovation and entrepreneurship education in terms of college factors and individual factors, 45.7% of students want college innovation and entrepreneurship education courses to focus on entrepreneurial opportunities or environment analysis in the teaching process, 41.4% of students feel average in terms of teaching resources, and more than 50% of college students are conservative in the direction of employment or entrepreneurship in terms of personal factors. Comprehensive analysis shows that the influencing factors, such as inflexible teaching methods, unsound curriculum systems, imperfect practice platforms, and college students’ own ideological awareness, are very significant and worth noting.