Factors such as class, class time, teachers, courses, and classrooms influence the optimization of the physical education course structure, making it challenging for the traditional physical education course structure to adapt to the current standards. For this reason, the article uses a genetic algorithm to optimize the structure of the physical education course. Class time, teachers, classrooms, courses, and classes are selected as the variables of the physical education course structure optimization model, and according to the actual situation of physical education course structure in colleges and universities, hard and soft rules for course optimization are formulated, and the construction of the optimization model of physical education course structure is completed based on the principles of physical education course structure optimization. We optimize the traditional genetic algorithm using the simulated annealing algorithm, which addresses the local optimal solution issue in the sports course structure optimization model. Examine the parameters of the improved genetic algorithm, and use the algorithm from this paper to conduct a case study on optimizing the structure of physical education courses. The fitness function value of this algorithm is higher than that of the traditional genetic algorithm from both the students’ and teachers’ points of view. The improved genetic algorithm can get rid of the issue of course structures that don’t work well with each other, which makes the best use of physical education course resources.