Since overlapping handprints often have the potential to become a breakthrough point in criminal cases, extracting and separating overlapping handprints at the scene has become an urgent problem to be solved in criminal technical examination. In this context, this paper proposes a machine learning-based optical separation method for overlapping handprints, which uses SVM to simplify the classification of the features of the training samples and completes the measurement of the features based on the statistical properties of the image grayscale histogram. Using image blind source separation, independent component analysis and pixel iteration algorithm to separate the overlapped image to get two different spatial resolution images, SSIM is chosen to evaluate the quality of the separated image. Then, the target classification recognition and extraction techniques of the method in the paper are analyzed by overlapping handprint optical separation experiments. The results show that compared with the traditional algorithms, the TPR, FPR, overlap degree and accuracy of this paper’s method in dealing with overlapping handprint images with deep fuzzy features are at a minimum of 0.9258, 0.3723, 0.4645, 0.9572, which are all greater than several other algorithms, which proves that the overlapping handprint optical separation method proposed in this paper based on machine learning has obvious advantages.