In this paper, we first studied the basic content of deep learning in college sports and explored the principles of physical education evaluation in colleges and universities in the context of deep learning. Then, the framework of the physical education evaluation index system is constructed, the indexes are screened and optimized by combining the Delphi method, and the weights of the indexes are calculated by using the fuzzy comprehensive evaluation method. Then, based on the expectation and satisfaction data of students in the sample colleges and universities, the gap is analyzed according to the perception-expectation matrix, and the SERVQUAL model is used to analyze each issue of the quality of physical education in colleges and universities one by one. The maximum value of the expectation of the quality of physical education in colleges and universities is 6.558 points, the minimum value is 6.348 points, the maximum value of the mean value of perception is 6.349 points, the minimum value is 5.797 points, and the scores are above 5 points, which makes the quality of education in colleges and universities better. This study further enriches the theory of quality evaluation of education in sports colleges and establishes the evaluation model and method with specific specifications.