A deep understanding of traditional cultural elements and a keen sense of modern design can reinterpret and recreate traditional cultural elements, so that they can be integrated into modern art works. This paper combines visual communication semiotics and task-aware adaptive network algorithms to construct a pattern recognition model for integrating traditional culture and modern design. Aiming at the multi-domain testing problem in which the domain offsets between different testing domains vary greatly, and the model is difficult to adapt to a variety of domain offsets, which leads to poor testing stability, it proposes the task-aware adaptive network, and the strategy of learning the task-related parameters for the target task through self-adaptation. After the dataset testing and practical application, the model in this paper also improves 1.5%-3.5% compared with the latest methods in the results on CIFAR-10. The recognition accuracy of traditional cultural elements such as graphic, color, material, space, light, and runic pattern are all above 90%, with 95.3%, 96.1%, 91.5%, 92.8%, 91.7%, and 90.4%, respectively. The application results are much higher than the dataset test results. The model fulfills the design expectations and realizes the purpose of recognizing traditional cultural elements in design works. Finally, an innovative method for integrating traditional cultural elements with modern art design is proposed.