Physical fitness assessment is of great significance for the design and adjustment of youth basketball training, and talent selection and evaluation of training teams. In this paper, a dynamic assessment model of physical fitness and health is designed, and a convolutional neural network and autoencoder are used to achieve feature learning of raw body side data. According to the learning results, the Gaussian mixture model is selected for physical fitness assessment, and the quantitative evaluation method of physical fitness is established based on the parameter-solving results of the EM algorithm. The ablation experiment demonstrates that the model in this paper has a low feature loss and excellent convergence, and the mAP value is 89.12%, which is the most accurate performance. The comprehensive accuracy of the assessment reaches 97.5%, indicating that the assessment performance of the dynamic assessment model of physical fitness and health proposed in this paper is better and can provide help for youth basketball training.