With the continuous improvement of high-performance machinery manufacturing capacity, the mechanical properties of high-performance materials and processing efficiency and quality put forward higher requirements. This paper takes CF/UHMWPE fiber hybrid composites as the main research body, carries out experiments on the mechanical properties of the material, and constructs an optimization model of the processing parameters of the high-performance material based on its mechanical properties, so as to improve the processing efficiency and processing quality. Based on the NSGA-II algorithm, the Pareto optimal solution set is obtained, and then the best parameter scheme is selected after screening. In the machining parameter optimization experiment, the spindle speed x1 of the Pareto front is taken in the range of 9500~9700r/min, and the feed speed x2 is taken in the range of 598~600mm/min, which achieves the improvement of machining efficiency. The prediction accuracy of the surface roughness of the workpiece is up to 98.66%, and the minimum prediction accuracy is 93.12%, and the average prediction accuracy reaches 95.86%, and the optimization results of the machining parameters have strong reliability.