In this paper, we use Caffe framework to implement the improved Faster R-CNN recognition technique for building images in civil engineering under Linux system and add feature pyramid network and regional feature aggregation into the ResNet-50 network and ResNet-101 network, respectively, to strengthen the training effect, and establish ResNet-101+FPN+ROI Align image recognition technique. Simulated crack experiments and concrete surface quality defect detection experiments confirm that the ResNet-101 FPN ROI Align method is accurate and detects defects at a high rate. The method established in this paper has a minimum error of only 0.4% in the simulated crack experiment, and the detection rate is much higher than that of other detection methods when detecting quality defects on the concrete surface, and the accuracy can reach up to 94% at the same time. In civil engineering, the image recognition technology established in this paper has practical significance and high application value, as demonstrated by the experiment.