The traditional English listening test system does not focus on analyzing students’ strengths and weaknesses, and designing a test bank with targeted exercises will be beneficial for students to grasp their actual situation. In this paper, after combining the twin network structure with the pre-trained language model, the Fusion-LM language matching model is constructed, which is utilized to calculate the matching similarity of students’ language features in the test. The adaptive grouping module and automatic scoring module are designed with constraints in mind and the adaptive test question bank for college English listening comprehension is established together. The validity test of the test question bank revealed that the test results of each module of the system meet the requirements, and the errors generated by automatic scoring are basically less than 2 points. The results of the teaching control experiment on whether to apply the test bank show that there is a significant difference between the post-test scores of the experimental and control classes of college English listening utterance comprehension (P=0.025<0.05) and the listening and pronunciation recognition abilities of the experimental class students have also improved significantly after the teaching. This paper lays a necessary foundation for the effective development of university English listening teaching activities and provides an effective method for improving students’ listening comprehension abilities.