The quality of the milled surface affects the performance of the affiliated workpiece, since it plays a vital role in determining the precision of the geometry and duration of service time. In this paper, a modified convolution recurrent neural network (CRNN) is proposed to effectively predict the surface quality of the end milling workpiece. First, the validated features of milling force data in the machining process are extracted based on the proposed artificial network model. Second, a modified CRNN model is constructed by merging residual neural network with the help of bidirectional long- and short-term memory as well as attention mechanism. Third, the model’s weight is optimised according to the changes in the loss function and directional propagation principle, which significantly improves the effectiveness of the proposed model. Finally, the actual experiment is carried out on a 5-axis milling centre to validate our model. Also, the surface quality predicted by the CRNN model is in good accordance with the experimental result. In our experiment, an accuracy of 98.35% is achieved, which is a significant improvement compared to the classic CRNN method.