Chinese Word Segmentation with Conditional Random Fields and Integrated Domain Knowledge
Chinese word segmentation is a difficult, important and widely-studied sequence modeling problem. Conditional Random Fields (CRFs) are a new discriminative sequence modeling technique that supports the incorporation of many rich features. This paper demonstrates the ability of CRFs to easily integrate domain knowledge, and thus reduce the need for labeled data. Using readily-available domain knowledge in the CRF’s feature definitions we show that, even training on as few as 140 segmented Chinese sentences, we can achieve world-class segmentation accuracy. Furthermore, when training on more data, our approach yields segmentation F1 of 97.5% with the Penn Chinese Treebank, and realistic, incomplete lexicons—a 50% reduction in error compared with the previously best published results of which we are aware. We also introduce two alternative prior distributions for CRF’s learned weights.