The Unsupervised Learning of Linguistic Structure
From a machine learning perspective,mostwork in natural languageprocessing can be seenas annotating observed linguistic inputs with unobservedstructure, for example detecting the topic of a text document or choosing a grammatical analysis for a sentence. The rules and patterns which drive such processing are typically either manually crafted by experts or derived automatically from human-annotated, labeled examples. On the other hand, language acquisition in humans is primarily unsupervised: children can learn languagerapidly, accurately, and robustly with lit tle or no explicit supervisory information. While a computerand achild areundeniably vastly different in both situationand context, thefactsof child language acquisition suggestthat there may be sufficient information in raw languagedata to allow the induction of hiddenlinguistic structures.The problemof unsupervisedlearning of languageis important not only in the abstract – thelogical questionof theacquisitionof natural language– but alsoonpractical grounds– labeled training data is expensiveand exists for only a relatively small number of langauges,while unlabeleddata is cheap, and more readily available. This thesisexaminesthe unsupervisedlearning of linguistic structure, primarily focusing on learning tree-st ructuredgrammatical analyses,in which treenodesdominate contiguousspansof wordscalled constituents. Constituents are the linguistically coherent units in a sentence, such as noun phrasesor verb phrases.An aim of a grammar induction systemis to figure out, givenjust the sentencesin a corpus, what treestructurescorrespond to them. It is an openproblemwhetherentirely unsupervisedmethodscan producelinguistically accurateparses of sentences. Due to the difficulty of the grammar induction task, the vast majority of statistical parsing work has focusedon supervisedapproaches,where one usesa treebank of fully parsedsentencesto induce a model which thenparsesunseensentences(Collins1997,Charniak 2000). In early work for this thesis(Klein and Manning 2001a, Klein and Manning 2001b, Klein and Manning 2002), we have shown a theoretically concise,robust method of inducing natural languagegrammar structures.This method not only parsesmuchbetter than therandom-parseunsupervisedparsing baseline,but it is also thefirst unsupervisedparsing systemto score higherthan the standard right-branching baselineon a per-consituent F-score on the ATIS parsing task (seefigure 2 and figure3). The restof the thesiswill explore extensionsof this initial work. First, we will thoroughly investigate the causesfor failure of previous systems. Second, although the above experiments usedparts-of-speech sequencesas input (rather than specific words), the full thesiswill integrate automatic word clustering into theacquisitionmodel. Finally, wewill examinethepotential for using broad syntatic constraint to allow the learning of semantic dependencies.Theseextensionsare describedbelow in more detail.