In this paper, a multi-scale feature fusion network is constructed by combining U-Net and Densebox algorithms, and feature extraction is performed by the deep residual network (ResNet). Taking colleges and universities as an example, a teacher reputation incentive model combining a multi-scale feature fusion network is established based on marginal productivity. The multi-scale feature fusion network is proposed to be used to realize the employee performance measurement level and personal appraisal system in the salary performance appraisal platform for teachers in colleges and universities. The appraisal test analysis of the teacher team’s performance and salary was carried out with the teacher team of School G as an example. The results show that the starting salary points of B1, C1, and D1 of the teacher team of school G are 2250, 1650, and 1260, respectively, in which the median difference in salary value between the highest grade B1 (9618) and the lowest grade I2 (784) is 8834 yuan. The performance pay management platform constructed in this paper effectively provides incentives and promotes the high-quality development of the university teaching team through strict assessment and fair distribution.