College students’ physical fitness test scores are an important criterion for evaluating students’ physical fitness, and scientific and effective prediction and analysis of physical fitness test scores can provide a theoretical basis for subsequent physical education teachers to carry out teaching. This study proposes a combined prediction model of the gray system and neural network to deal with the small sample data of college students’ physical fitness test scores, introduces the basic concepts of the GM(1,1) model and BP neural network, respectively, and explains the advantages and complementarities between the gray prediction and the neural network prediction, which provides theoretical support for the combined prediction model. By capturing 2000 college students’ physical fitness test scores from a university as the research object, 1600 of them were used as training samples and the remaining 400 as test samples, and different data sets were divided by gender. The model was applied to predict individual specific item scores and classify the total assessment. Taking the girls’ 50-meter running performance as an example to draw the comparison curve of the prediction model, it was found that the error of the gray neural network model prediction was within 0.5 seconds. In addition, the RMSE values of the prediction results of other sports performance were all below 0.06, and the MAPE values were all below 3%, which means that the model can meet the practical requirements of the prediction of the physical fitness test. The horizontal ladder plot and confusion matrix plot reflect that the model is relatively accurate in predicting the overall rating level of students’ physical fitness test scores, with an accuracy of 95.142% in the boys’ dataset and 95.425% in the girls’ dataset.