The development of art education in colleges and universities is a direct reflection of the soft power of culture. In this paper, based on two models, BP neural network and long and short-term memory neural network in deep learning model, we study the factors affecting the development of college art education. Firstly, the article introduces the basic models of the BP neural network and LSTM neural network and selects the indicators under three dimensions of college art education scale, college art education input, and output to visually analyze the development status of college art education and find the possible problems. Secondly, five dimensions, namely, population scale, policy support, economic strength, industrial structure, and faculty level, were selected as the influential indicators to study the level of art education in colleges and universities. The population size passed the 1% significance test, GDP per capita and faculty strength passed the 5% significance test, the influence coefficient of art education expenses was 0.349, and only the industrial structure failed the test. According to the analysis, we conclude that the factors affecting the development of art education in colleges and universities are mainly four aspects: population size, policy support, economic strength, and faculty level.