As a legal document for people traveling to and from the country, the authentication of passport is of great significance for personal identification and national security. In this paper, we design a passport image authentication method that includes UV anti-counterfeiting information enhancement and UV anti-counterfeiting pattern matching recognition. The UV anti-counterfeiting information enhancement algorithm combines the image quality information of the anti-counterfeiting region, solves the problem of unstable enhancement effect of the existing algorithms, and achieves the purpose of stabilizing the quality of UV anti-counterfeiting information and improving the enhancement effect. The growth rate of this paper’s algorithm in passport fluorescent foreground image processing in the mean, standard deviation, and clarity is 66.26%, 63.85%, and 55.75%, and at the same time effectively suppresses the non-fluorescent background image. The UV anti-counterfeiting pattern matching recognition algorithm based on color feature extraction and gradient feature extraction can effectively reduce the misidentification rate of real passports and the leakage rate of counterfeit passports, and its misidentification rate and leakage rate are 5.5% and 6.9%, respectively, which achieves better passport forgery identification results. The designed passport ultraviolet spectral image authentication method can be used in the passport authentication reading system to realize an informatization and intelligent passport authentication method.