Based on the cluster analysis algorithm, this paper collects a large amount of sample data from Huizhou buildings and groups these samples using the cluster analysis algorithm so as to identify different categories of architectural culture types. By dividing the group similarity of each category of buildings, the village cultural characteristics related to each category of buildings are extracted, including information on architectural style, structural features, decorative elements and so on. The results show that the automatic cultural classification feature values are 1.2, 2.5, and 2.9, which highlight the unique characteristics of the Huizhou village culture. Especially in Hongcun and Pai Fang Lane buildings, the cultural features are more significant, with matrix classification degrees of 4.7 and 4.022, respectively, fully highlighting the cultural heritage of these buildings. In addition, the clustering analysis algorithm has only a 9.5% error rate, and the classification accuracies all exceed the high level of 0.9, showing its excellent performance in the extraction of various cultural features.