Applied Mathematics and Nonlinear Sciences
Journal license

Journal

Applied Mathematics and Nonlinear Sciences


Volume
& Issue

Volume 9, Issue 1


Published
on

August 5, 2024


Pages


DOI

Article

Chinese Natural Language Processing: From Text Categorization to Machine Translation

Check for updates


Authors

Haitao Peng Affiliation:
School of Foreign Studies, Yulin Normal University, Yulin, Guangxi, 537000, China.


Abstract

The level and volume of automatic computerized processing of linguistic information has become one of the most important criteria for measuring whether a country has entered the information society. The study begins with statistical linguistics and aims to process complicated Chinese information. In this paper, after establishing the word database of the Chinese language, the language model is smoothed and compressed, the Chinese character information and Chinese language information are extracted, and the processing of Chinese grammar and Chinese semantic information is emphasized. Among them, Chinese grammar processing includes Chinese word analysis and basic phrase analysis based on the maximum entropy model, and Chinese semantic processing includes Bayesian-based word sense disambiguation, semantic role labeling based on the conditional random field model, and thesaurus-based semantic similarity calculation method. In addition, SECTILE-based Chinese text categorization and statistical linguistics-based machine translation methods are explored and tested for their effectiveness in Chinese natural language processing. The results show that the total average check accuracy and check the completeness of Chinese text are 78.65% and 72.24%, respectively, and the BLEU values of the translation methods are improved by [1.62,3.73] and [0.93,5.01] compared with the Baseline method, which is able to process the Chinese information accurately. The research plays an important role in the process of information processing based on Chinese language processing.


Keywords

Statistical linguistics, Chinese language processing, Maximum entropy model, Conditional random field model, Text categorization, Machine translation, 65Y04


Citation

Peng, H. (2024). Chinese natural language processing: From text categorization to machine translation. Applied Mathematics and Nonlinear Sciences, 9(1). https://doi.org/10.2478/amns-2024-1860

Published by: Engineering Journals

Engineering Journals Logo