Word Similarity Calculation by Using the Edit Distance Metrics with Consonant Normalization
1
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
Edit distance metrics are widely used for many applications such as string comparison and spelling error corrections. Hamming distance is a metric for two equal length strings and Damerau-Levenshtein distance is a well-known metrics for making spelling corrections through string-to-string comparison. Previous distance metrics seems to be appropriate for alphabetic languages like English and European languages. However, the conventional edit distance criterion is not the best method for agglutinative languages like Korean. The reason is that two or more letter units make a Korean character, which is called as a syllable. This mechanism of syllable-based word construction in the Korean language causes an edit distance calculation to be inefficient. As such, we have explored a new edit distance method by using consonant normalization and the normalization factor.
Abstract
Edit distance metrics are widely used for many applications such as string comparison and spelling error corrections. Hamming distance is a metric for two equal length strings and Damerau-Levenshtein distance is a well-known metrics for making spelling corrections through string-to-string comparison. Previous distance metrics seems to be appropriate for alphabetic languages like English and European languages. However, the conventional edit distance criterion is not the best method for agglutinative languages like Korean. The reason is that two or more letter units make a Korean character, which is called as a syllable. This mechanism of syllable-based word construction in the Korean language causes an edit distance calculation to be inefficient. As such, we have explored a new edit distance method by using consonant normalization and the normalization factor.
참고문헌 (15)
- R. A. Wagner, "The string-to-string correction problem", Journal of the ACM, 21(1), pp.168~173 (1974)
- G. Navarro, "A guided tour to approximate string matching", ACM Computing Surveys, 33(1), pp.31~88 (2001)
- 노강호, "한글에 대한 편집 거리 문제", 정보과학회논문지 : 시스템 및 이론, 37(2), pp.103~109 (2010), 한국정보과학회
- B. Bae, "Edit distance calculation by phonetic rules and word-length normalization", Proceedings of the European Computing Conference (ECC'12), pp.315~319 (2012)
- R. W. Hamming, "Error detecting and error correcting codes", Bell System Technical Journal, 29(2), pp.147~160 (1950)
- V. I. Levenshtein, "Binary codes capable of correcting deletions, insertions and reversals", Soviet Physics-Doklady, 10(8), pp.707~710 (1966)
- F. J. Damerau, "A technique for computer detection and correction of spelling errors", Communications of the ACM, 7(3), pp.171~176 (1964)
- W. E. Winkler, "String comparator metrics and enhanced decision rules in the Fellegi-Sunter model of record linkage", Proceedings of the Survey Research Methods Section, pp.354~359 (1990)
- S. H. Cha, "Taxonomy of nominal type histogram distance measures", Proceedings of the American Conference on Applied Mathematics, pp.325~330 (2008)
- D. Gusfield, "Algorithms on Strings, Trees, and Sequences: Computer Science and Computational Biology" (1997), Cambridge University Press
- K. Kukich, "Techniques for automatically correcting words in text", ACM Computing Surveys, 24(4), pp.377~439 (1992)
- L. Ahmedi, "A database normalization tool using Semantic Web technologies", International Journal of Systems Applications, Engineering & Development, 5(4), pp.502~517 (2011)
- A. N. Khan, "Psyche mining with PsycheTagger : a computational linguistics approach to text mining", International Journal of Computers and Communications, 6(2), pp.119~127 (2012)
- H. Lee, "A study on the efficient education of pronunciation in Korean phonetic transformation rules", 석사, pp.2008
- S. Chang, "Slip of the Tongue that Slip of the Pen" (2000), Ministry of Culture and Tourism