With the rapid development of the Internet, security issues are becoming more and more prominent, and since most information is transmitted through the Internet today, Internet security is particularly important. When the Internet was designed, only mutual compatibility and interoperability between networks were considered, and security issues were not fully considered. As a result, as the Internet continues to grow, security issues are becoming more and more serious. One of the more difficult attacks is the Distributed Denial of Service (DDoS) attack, which has many forms of attacks, is harmful, and is difficult to identify and defend. Therefore, building a global Internet security protection system to achieve effective protection against DDoS attacks is the main work of this research paper. In this paper, we propose an artificial intelligence DDoS attack protection system, which implements a controller and switch auto-detection model by extending the protocol and establishing an optimization model to realize a low-load and low-latency traffic monitoring scheme; for DDoS attacks. We propose the attack inspection algorithm SCVAE based on Variational Encoder (VAE) and Spectral Clustering. in order to mitigate DDoS attack traffic, the protection system uses the QoS traffic control method, builds the application flow hierarchy model, and filters the attack traffic endured by the system by setting the application flow bandwidth limit as well as the traffic priority dual policy. Finally, a Mininet-based simulation test environment is built to evaluate the model, and different test indexes are set for different system modules to evaluate their actual performance. The results of this paper show that in the network traffic monitoring test, the artificial intelligence DDoS attack protection algorithm can respond to the attack more quickly by reducing the average 73ms per sampling compared with other algorithms; in the attack traffic identification test, the comparison accuracy (P) is improved by 15.14%, the accuracy (AC) is improved by 13.26%, the recall (R) is reduced by 9.23%, and the F1 measurement criteria improved by 23%. The test verifies that the artificial intelligence DDoS attack protection system can achieve real-time monitoring of each performance parameter and also illustrates the feasibility and practicality of the research content of this paper, which strengthens the construction of the technical means of Internet security protection and further enhances the Internet security defense capability.