Color, as the first element affecting human vision, is particularly important in the process of oil painting art creation. This paper constructs a color-matching optimization algorithm using convolutional neural network and generative adversarial network algorithms to optimize the four optimization factors of Chinese oil painting: color brightness, hue, contrast and orderliness. Color beauty is used to measure the color matching and design effect of Chinese oil paintings, and the order factor and complexity factor between each color of the color matching scheme are calculated. Finally, the effectiveness of the color matching optimization scheme is verified by taking the Chinese oil painting ‘Eight Nine Geese Coming’ as the research object. The results show that the harmonies of the four color-matching optimization schemes are 3.53, 2.86, 3.43 and 2.34, respectively, with an overall average improvement of 5.1175 percentage points compared with that before optimization. The color matching of Chinese oil paintings is based on a theoretical basis provided by this study, and it also introduces a new line of thought.