Journal
IEEE TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING
Volume 19, Issue 1, Pages 166-175Publisher
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TASL.2010.2045240
Keywords
Arabic; A* search; case-ending; corpus-based linguistics; coverage; diacritics; diacritization; disambiguation; factorized features; human language technologies (HLT); hybrid; language factorization; language modeling; language models; language processing; morphological analysis; morphology; natural language processing (NLP); n-grams; phonetic transcription; phonological analysis; statistical language model (SLM); statistical; stochastic; syntax; unfactorized features; vowelization
Categories
Ask authors/readers for more resources
This paper introduces a large-scale dual-mode stochastic system to automatically diacritize raw Arabic text. The first of these modes determines the most likely diacritics by choosing the sequence of full-form Arabic word diacritizations with maximum marginal probability via lattice search and long-horizon n-grams probability estimation. When full-form words are OOV, the system switches to the second mode which factorizes each Arabic word into all its possible morphological constituents, then uses also the same techniques used by the first mode to get the most likely sequence of morphemes, hence the most likely diacritization. While the second mode achieves a far better coverage of the highly derivative and inflective Arabic language, the first mode is faster to learn, i.e., yields better disambiguation results for the same size of training corpora, especially for inferring syntactical (case-ending) diacritics. Our presented hybrid system that benefits from the advantages of both modes has experimentally been found superior to the best performing reported systems of Habash and Rambow, and of Zitouni et al., using the same training and test corpus for the sake of fair comparison. The word error rates of (morphological diacritization, overall diacritization including the case endings) for the three systems are, respectively, as follows (3.1%, 12.5%), (5.5%, 14.9%), and (7.9%, 18%). The hybrid architecture of language factorizing and unfactorizing components may be inspiring to other NLP/HLT problems in analogous situations.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available