In the era of the knowledge economy, colleges and universities, as knowledge-intensive units, urgently need to seek a new management model that meets the requirements of the times, supports the high quality of teaching, and helps colleges and universities to realize sustainable development. Based on this demand, this paper focuses on the concept of knowledge management to innovate the existing management mode and takes the construction of a knowledge-sharing system, the establishment of a knowledge innovation incentive mechanism, the development of organizational learning, and the creation of knowledge alliance as the specific implementation strategies. Using the multi-level gray correlation evaluation method and fuzzy comprehensive evaluation method, we evaluate the actual effects of the knowledge management strategy of colleges and universities in this paper. The multi-level gray correlation evaluation method is used to assess the knowledge management level of 60 colleges and universities, and the results of their scoring order are compared with the reference order. The order-preserving test parameter δ=0.0446<0.1 indicates that the basic trend of this paper’s sorting is consistent with the reference sorting, which verifies the practicality of the knowledge management evaluation system for colleges and universities constructed in this paper. Taking the university H, which implements the knowledge management strategy of this paper as an example, the five levels of the evaluation set V={excellent, good, medium, poor, very poor} are assigned a value of 5~1 respectively, and the final evaluation score of university H is 3.9394, and its knowledge management level tends to be excellent. This paper’s knowledge management strategy has a positive implementation effect and can be a valuable reference for knowledge management innovation in colleges and universities.