In the growth process of crops, the growth information of crops is an important basis for judging the growth trend and yield of crops, and it is also important research for monitoring the changes in crop growth. In this study, we constructed a monitoring system based on improved Faster R-CNN, selected the situation of different rice varieties in three cities of Jiangxi Province, and used the data to analyze the relationship between the growth and development of early rice and late rice and climate in Jiangxi Province. Based on the data results, for the case of the correlation of rice growth to climate ability in Jiangxi Province, it was concluded that the total growing season temperatures of both early and late rice passed the significance test. By using the monitoring system to identify the growth trend of rice shoots, the image recognition of rice shoots was adopted, and after pre-processing the images, the length, width, area and number of rice shoots were finally analyzed with respect to the law of time, and the growth process of shoots was monitored. The period of rising rice growth area is in July-August, which is one of the months of rice shoot area growth. The monitoring system employed in this paper is capable of effectively monitoring the impact of climate on the growth cycle of crops.