A Text Similarity Measurement Method Based on Singular Value Decomposition and Semantic Relevance
2
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
The traditional text similarity measurement methods based on word frequency vector ignore the semanticrelationships between words, which has become the obstacle to text similarity calculation, together with thehigh-dimensionality and sparsity of document vector. To address the problems, the improved singular valuedecomposition is used to reduce dimensionality and remove noises of the text representation model. Theoptimal number of singular values is analyzed and the semantic relevance between words can be calculated inconstructed semantic space. An inverted index construction algorithm and the similarity definitions betweenvectors are proposed to calculate the similarity between two documents on the semantic level. Theexperimental results on benchmark corpus demonstrate that the proposed method promotes the evaluationmetrics of F-measure.
Abstract
The traditional text similarity measurement methods based on word frequency vector ignore the semanticrelationships between words, which has become the obstacle to text similarity calculation, together with thehigh-dimensionality and sparsity of document vector. To address the problems, the improved singular valuedecomposition is used to reduce dimensionality and remove noises of the text representation model. Theoptimal number of singular values is analyzed and the semantic relevance between words can be calculated inconstructed semantic space. An inverted index construction algorithm and the similarity definitions betweenvectors are proposed to calculate the similarity between two documents on the semantic level. Theexperimental results on benchmark corpus demonstrate that the proposed method promotes the evaluationmetrics of F-measure.
참고문헌 (16)
- N. K. Nagwani, "A comment on “a similarity measure for text classification and clustering", IEEE Transactions on Knowledge and Data Engineering, 27(1), pp.2589~2590 (2015)
- A. Awajan, "Semantic similarity based approach for reducing Arabic texts dimensionality", International Journal of Speech Technology, 19(2), pp.191~201 (2016)
- L. Xu, "Text similarity algorithm based on semantic vector space model", Proceedings of the 15th International Conference on Computer and Information Science, pp.1~4 (2016)
- R. lonescu, "Knowledge Transfer between Computer Vision and Text Mining: Similarity-Based Learning Approaches" (2016), Springer
- E. Blanco, "A semantic logic-based approach to determine textual similarity", IEEE, 2015, pp.23(4), 683~693
- M. Shirakawa, "Wikipedia-based semantic similarity measurements for noisy short texts using extended naive Bayes", IEEE Transactions on Emerging Topics in Computing, 3(2), pp.205~219 (2015)
- H. Z. Liu, "Accessing text semantic similarity using ontology", Journal of Software, 9(2), pp.490~497 (2014)
- W. Song, "Genetic algorithm for text clustering using ontology and evaluating the validity of various semantic similarity measures", Expert Systems with Applications, 36(5), pp.9095~9104 (2009)
- Y. Wang, "Document clustering with semantic analysis", Proceedings of the 39th Annual Hawaii International Conference on System Sciences, pp.54~63 (2006)
- R. M. Aliguliyev, "A new sentence similarity measure and sentence based extractive technique for automatic text summarization", Expert Systems with Applications, 36(4), pp.7764~7772 (2009)
- L. Gang, "Text information retrieval based on concept semantic similarity", Proceedings of the 5th International Conference on Semantics, Knowledge and Grid, pp.356~360 (2009)
- A. Hotho, "Ontologies improves text document clustering", Proceedings of the 3rd IEEE International Conference on Data Mining, pp.541~544 (2003)
- R. J. Bellegarda, "Exploiting latent semantic information in statistical language modeling", Proceedings of the IEEE, 88(8), pp.1279~1296 (2000)
- C. Buck, "Quick and reliable document alignment via TF", IDF-weighted cosine distance, 2016, 672~678
- A. Mirzal, "Clustering and latent semantic indexing aspects of the singular value decomposition", International Journal of Information and Decision Sciences, 8(1), pp.53~72 (2016)
- "CLUTO: a clustering toolkit" (2006)[URL]