The rapid development of augmented reality technology and the widespread popularization of devices have caused changes in traditional interaction methods. An iterative, interactive learning model based on online interactive learning theory and supported by augmented reality technology is proposed in this paper. Based on this, the study constructs a machine learning-based teaching interaction analysis process model, which ranks the features according to the SHAP value in order to select the best learning interaction behavioral features for studying the teaching effect of ideological and political education in colleges and universities, and uses XGBoost to train the sampled behavioral data. Finally, a teaching experiment was carried out with the compulsory public course of Ideological and Moral Cultivation and the Foundation of Law in College X to verify the teaching effect of the model and the impact of each learning interaction behavior on students. The results show that from the beginning to the end of the semester, the student activity rate increased from 0.0520 to 0.0785, the teacher-student behavioral transition rate increased from 0.0054 to 0.1101, and the learner interaction rate also increased from 0.0088 to 0.0637. It indicates that the teaching course of ideological and political education in colleges and universities has become more effective and appealing in the augmented reality iterative interactive learning environment.