The Picture of a Lady with a Fan in the Tang Dynasty has become essential for studying ancient costume art with its exquisite painting skills and rich cultural value. However, due to the passage of time, the costume details in the paintings are seriously damaged, which brings challenges to the research. This study uses computer-aided digital restoration technology to reproduce the original appearance of ladies’ costumes in The Picture of a Lady with a Fan. This study realized the digital reconstruction of ladies’ costumes in the Tang Dynasty through high-precision scanning and image processing of paintings combined with historical documents and physical materials. During the research process, we first scanned “The Lady with a Fan” using a 40-megapixel high resolution to guarantee the clarity of the image details. Afterwards, an image restoration algorithm based on deep learning was employed to handle the damaged portion of the painting, and 95% of the clothing area was restored. On the restoration of costume colors, through the statistical analysis of costume colors in the Tang Dynasty, we constructed a database containing 120 typical colors from the Tang Dynasty, and based on this, we made color matching and rendering of costumes. The experimental results show that the color saturation of the restored clothing is increased by 30%, and the pattern clarity reaches 98%. Compared with the existing costumes of the Tang Dynasty, the style accuracy rate of restored costumes has reached 90%. In addition, we also used three-dimensional modeling technology to restore the costume in three dimensions, and its structure was 85% similar to that of the Tang Dynasty costume recorded in the literature.