For the current commercially available fabric yarn detection, which is based on traditional machine vision methods and relies heavily on manually designed features, an improved Faster R-CNN algorithm is proposed in this paper. In this paper, based on the Faster R-CNN algorithm, the deformable convolutional Resnet-50 network is fused to improve the learning ability of woven yarn features. By designing a multi-scale model for the detection of fine features in fabric yarns, a cascade network is introduced to improve the detection accuracy and localization accuracy of woven yarns, and an optimized loss function is constructed to reduce the effect of sample imbalance. Through the experiments of the improved fast R-CNN algorithm for fabric yarn detection, we can find that the confidence level of SSD algorithm detection is 58%, and the confidence level of the original Faster R-CNN algorithm detection is 78%, while the improved Faster R-CNN can not only accurately frame the finesse problem, but also the confidence level is as high as 97%. So this paper, based on the improved Faster R-CNN algorithm, can pinpoint the problem of fabric yarn detection, improve the learning ability of detection, and can meet the current demand for fabric yarn detection in the market.