텍스트 마이닝을 이용한 웹 포럼 불량글 탐지 모델
The Spam Detection Model for Web Forums using Text Mining Techniques
0
조회
0
다운로드
3
인용
0
참고문헌
키워드
초록
웹포럼상의 불량글은 사용자의 불편함을 증가시키고 사용자 의견 수렴의 원천으로서의 웹포럼의 가치를 하락시킨다. 웹포럼상의 글의 중요도는 일반적으로 해당글에 참여한 사용자 수나 댓글수로 측정하므로, 이러한 불량글은 웹포럼상의 의견수집 시 많은 불필요한 데이터를 포함하게 되고, 이렇게 분석된 결과는 왜곡된 결과를 내기도 한다. 본 연구에서는 웹포럼상의 불량글을 자동 분류하는 탐지 모델을 제안한다. 웹포럼상의 글에 대해 텍스트마이닝 기술을 적용하여 글의 구조적, 의미적, 형식적 텍스트 특성을 추출하고, 정상과 불량글을 두 개의 클래스로 한 분류 모델을 학습시킨다. 불량글을 자동으로 분류하는 모델을 학습시키기 위해서는 불량글과 정상글로 사전에 분류된 학습데이타가 필요하기 때문에, 평가자가 글의 불량성 여부를 평가하도록 한다. 자동화된 학습 모델을 구축하기 위해 데이터마이닝 및 텍스트마이닝에서 그 성능이 입증된, Naive Bayesian, Support Vector Machine (SVM), Decision Tree를 이용하였다. 이를 통해 불량글과 정상글을 구분하는 특성을 파악할 수 있으며, 학습된 분류 모델을 이용하여 새로운 글을 자동으로 분류할 수 있다. 본 연구에서는 제시하는 모델을 월마트(Walmart) 관련 최대 포럼인 YahooFinace내에 존재하는 Walmart-forum에 적용하였다.
Abstract
The spam in the discussion web forum causes user inconvenience and lowers the value of the web forum as the open source of user opinion. The importance of postings is evaluated in terms of the number of involved authors, so the spam distorts the analysis result by adding the unnecessary data in the opinion analysis. We propose the automatic detection model of spam postings in the web forum. We extract text features of posting contents using text mining techniques from the perspective of linguistics and then perform supervised learning to recognize spam from normal postings. Significant features are derived through the learning process and the automatic detection model is built based on those features. To build the automatic detection model of normal postings and spam, four evaluators are asked to recognize the spam posting in prior. We adopted the Naive Bayesian, Support Vector Machine (SVM), decision tree, which are known to perform well in data and text mining tasks. We can extract the text features to recognize the spam and detect automatically the newly posted spam. We apply the proposed model to the YahooFinace-Walmart forum, which is the world largest Walmart-related web forum.
참고문헌 (24)
- Sampson S., "Gathering customer feedback via the Internet: instruments and prospects", Industrial Management & Data Systems, 98(1-2), pp.71~ (1998)
- Gillin P., "The New Influencers, A Marketer’s Guide to the New Social Media" (2007), Quill Driver Books\Word Dancer Press
- Morinaga S., "Mining product reputations on the Web", The eighthACM SIGKDD international conference on Knowledgediscovery and data mining, 341 (2002)
- Liu Y, "ARSA: A Sentiment-Aware Model for Predicting Sales Performance Using Blogs" (2007), SIGIR
- Glance N., "Deriving Marketing Intelligence from Online Discussion" (2005), KDD
- Gruhl D., "The predictive power of online chatter", pp.78~87 (2005), KDD
- Wenger A., "Analysis of travel bloggers' characteristics and their communication about Austria as a tourism destination", Journal of Vacation Marketing, 14(169) (2008)
- Wanas N., "Automatic Scoring of Online Discussion Posts" (2008), WICOW
- Niu Y., "A Quantitative Study of Forum Spamming Using Context-based Analysis", Network & Distributed System Security (NDSS) Symposium (2007)
- Hayati P., "Toward spam 2.0: An evaluation of Web 2.0 anti-spam methods Industrial Informatics", INDIN 2009. 7th IEEE International Conference on, pp.875~880 (2009)
- Lin Y., "Splog detection using self-similarity analysis on blogtemporal dynamics", AIRWeb '07 Proceedings of the 3rdinternational workshop on Adversarial informationretrieval on the web (2007)
- Mishne G., "Blocking Blog Spam with Language Model Disagreement" (2005), AIRWeb
- Han S., "Collaborative blog spam filtering using adaptive percolation search" (2006), WWW
- Jindal N., "Opinion Spam and Analysis" (2008), WSDM’08
- Zinman A., "Is Britney Spears spam", Fourth Con-ference on Email and Anti-Spam Mountain View (2007)
- Benevenuto F., "Identifying Video Spammers in Online Social Networks" (2008), AIRWeb
- Statistical Identification of Language, "Dunning T.", New Mexico State University, 94~273, pp.Technical Report MCCS (1994)
- Paul K., "Analyzing Grammar: An Introduction", pp.35~ (2005), Cambridge University Press
- Robert F., "Syntax. Critical Concepts in Linguistics" (2006), Routledge
- Lewis D., "Naive (Bayes) at forty: The independence assumption in information retrieval", pp.4~15 (1998), Machine Learning
- Vapnik VN., "The nature of statistical learning theory" (1995), Springer-Verlag
- Quinlan JR., "Induction of decision trees. In Machine Learning", 81~106 (1986)
- Buckland M., "The relationship between Recall and Precision", Journal of the American Society for Information Science, 45(1), pp.12~19 (1999)
- Gwet K., "Handbook of Inter-Rater Reliability (Second Edition)" (2010), ISBN