In order to promote the development of the Chinese gardening field, this paper proposes a data-driven, multi-sensory design for Chinese gardens. The multisensory design of Chinese gardens, which combines data-driven technology and VR technology, is explored from two perspectives: the visual and auditory senses. It is found that the design easily leads to the loss of the natural beauty of the garden. For this reason, the layout optimization model is constructed using the improved PSO algorithm, and the corresponding analysis method is applied to analyze the effect of the multi-sensory design of the garden. The results show that the model in this paper is better than the other two control models in terms of weighted mean (4.078) and convergence time (0–4 ms). It was also found that the basis for visitors to make experience visual satisfaction evaluations was basically whether the floral landscape was beautiful (39.18%) and whether the field of view was open (23.27%), and the auditory experience design evaluation scores were higher for acacia forests (5.436), red-leafed ridges (4.091), snapdragon forests (3.872), and metasequoia forests (3.657) in the gardens. The research in this paper reveals the focus of multi-sensory design in Chinese gardens, which can provide theoretical references for research in the field of Chinese garden design.