트레이닝 데이터가 제한된 환경에서 N-Gram 사전을 이용한 트위터 스팸 탐지 방법
A Method for Twitter Spam Detection Using N-Gram Dictionary Under Limited Labeling
0
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
본 논문에서는 트레이닝 데이터가 제한된 환경에서 n-gram 사전을 이용하여 불건전 정보를 포함하는 스팸 트윗을 탐지하는 방법을 제안한다. 불건전 정보를 포함하는 스팸 트윗은 유사한 단어와 문장을 사용하는 경향이 있다. 이러한 특성을 이용하여 스팸 트윗과 정상 트윗에 대한 n-gram 사전을 구축하고 나이브 베이스 분류기를 적용하여 효과적으로 스팸 트윗을 탐지할 수 있음을 보인다. 반면에, 실시간으로 대용량의 데이터가 유입되는 트위터의 특성은 초기 트레이닝 집합 구성에 매우 큰 비용을 요구 한다. 따라서, 초기 트레이닝 집합이 매우 작거나 존재하지 않는 환경에서 적용할 수 있는 스팸 트윗 탐지 방법이 필요하다. 이를 위해 트위터의 리트윗 기능을 활용하여 의사 라벨을 생성하고 초기 트레이닝 집합의 구성과 n-gram 사전 업데이트에 활용하는 방법을 제안한다. 2016년 12월 1일부터 2016년 12월 7일까지 수집된 한국어 트윗 130만 건을 사용한 다양한 실험 결과는 비교 방법들보다 제안하는 방법의 성능이 우수함을 입증한다.
Abstract
In this paper, we propose a method to detect spam tweets containing unhealthy information by using an n-gram dictionary under limited labeling. Spam tweets that contain unhealthy information have a tendency to use similar words and sentences. Based on this characteristic, we show that spam tweets can be effectively detected by applying a Naive Bayesian classifier using n-gram dictionaries which are constructed from spam tweets and normal tweets. On the other hand, constructing an initial training set requires very high cost because a large amount of data flows in real time in a twitter. Therefore, there is a need for a spam detection method that can be applied in an environment where the initial training set is very small or non exist. To solve the problem, we propose a method to generate pseudo-labels by utilizing twitter's retweet function and use them for the configuration of the initial training set and the n-gram dictionary update. The results from various experiments using 1.3 million korean tweets collected from December 1, 2016 to December 7, 2016 prove that the proposed method has superior performance than the compared spam detection methods.
참고문헌 (33)
- "Number of Monthly Active Twitter Users Worldwide from 1st quarter 2010 to 4th quarter 2016 (in millions)"[URL]
- "Number of tweets per day?"[URL]
- L. M. Aiello, "Sensing Trending Topics in Twitter", IEEE Trans. Multimedia, 15(6), pp.1268~1282 (2013)
- T. Sakaki, "Earthquake Shakes Twitter Users : Real-Time Event Detection by Social Sensors", Proc. 19th International Conference on World Wide Web, pp.851~860 (2010), ACM
- A. I. Baqapuri, "Sentiment Classification of Tweets using Hierarchical Classification", Proc. IEEE International Conference on Communications (2016)
- "Almost 10% of Twitter Is Spam"[URL]
- "Twitter acknowledges 23 million active users are actually bots"[URL]
- C. Chen, "Spammers Are Becoming “Smarter” on Twitter", IEEE Trans. IT Professional., 18(2), pp.66~70 (2016)
- H. J. Choi, "A Twitter Spam Detection Method based on n-gram Dictionary", Proc. Korea Computer Congress, pp.227~229 (2017)
- K. Tao, "Groundhog Day : Near-Duplicate Detection on Twitter", Proc. 22nd International Conference on World Wide Web, pp.1273~1284 (2013), ACM
- K. M. Lee, "Uncovering social spammers : social honeypots + machine learning", Proc. 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.435~442 (2010), ACM
- F. Benevenuto, "Detecting spammers on Twitter", Presented at the 7th annual Collaboration Electronic Messaging Anti-Abuse Spam Conference (CEAS), 6 (2010)
- A. H. Wang, "Don’t follow me : spam detection in twitter", Proc. International Conference on Security and Cryptography (SECRYPT) (2010)
- S. Liu, "Statistical Detection of Online Drifting Twitter Spam", Proc. 11th ACM on Asia Conference on Computer and Communications Security, pp.1~10 (2016), ACM
- C. Chen, "A Performance Evaluation of Machine Learning-Based Streaming Spam Tweet Detection", IEEE Trans. Computational Social Systems, 2(3), pp.65~75 (2015)
- C. Chen, "Asymmetric Self-Learning for Tackling Twitter Spam Drift", Proc. IEEE Conference on Computer Communications Workshops, pp.208~213 (2015), IEEE
- G. Stringhini, "Detecting spammers on social networks", Proc. 26th Annual Computer Security Applications Conference, pp.1~9 (2010), ACM
- J. Song, "Spam filtering in Twitter using sender-reeiver relationship", Proc. 14th International Conference on Recent Advances in Intrusion Detection, pp.301~317 (2011), Springer
- C. Yang, "Empirical evaluation and new design for fighting evolving twitter spammers", IEEE Trans. Information Forensics and Security, 8(8), pp.1280~1293 (2013)
- K. Thomas, "Design and evaluation of a real-time URL spam filtering service", Proc. IEEE Symposium on Security and Privacy, pp.447~462 (2011)
- S. H. Lee, "Warningbird : A near real-time detection system for suspicious URLs in Twitter spammers", IEEE Trans. Information Forensics and Security, 8(8), pp.1280~1293 (2013)
- D. M. Freeman, "Using Naive Bayes to Detect Spammy Names in Social Networks", Proc. the 2013 ACM Workshop on Artificial Intelligence and Security, pp.3~12 (2013), ACM
- A. Herdağdelen, "Twitter n-gram corpus with demographic metadata", Language Resources and Evaluation, 47(4), pp.1127~1147 (2013)
- 이승재, "사용자 맞춤형 스팸 문자 필터링 시스템", 한국콘텐츠학회 논문지, 11(12), pp.122~135 (2011), 한국콘텐츠학회
- H. N. Lee, "A Study on Structuring Spam Short Message Service(SMS)filter", Proc. Symposium of the Korean Institute of communications and Information Sciences, pp.1072~1073 (2011)
- 이성욱, "카이제곱 통계량과 지지벡터기계를 이용한 스팸메일 필터", 정보처리학회논문지. 소프트웨어 및 데이터 공학, 17(3), pp.249~254 (2010), 한국정보처리학회
- 조인휘, "휴대폰 SMS를 위한 SVM 기반의 스팸 필터링 시스템", 한국통신학회논문지B, 34(9), pp.908~914 (2009), 한국통신학회
- Y. H. Kim, "Spam Twit Filtering using NaÏve Bayesian Algorithm and URL Analysis", Proc. Korean Institute of Information Scientists and Engineers, 38(2B), pp.375~378 (2011)
- "Streaming APIs"[URL]
- "Q3 Trend Report Highlights Real-Time Malware Campagigns And Increase In Phishing"[URL]
- V. Metsis, "Spam Filtering with Naive Bayes–Which Naive Bayes?", Proc. the Third Conference on Email and Anti-Spam, pp.28~69 (2006)
- J. Graovac, "Text Categorization Using n-Gram Based Language Independent Techniques", Proc. 35th Anniversary of Computational Linguistics, pp.124~135 (2014)
- "Weka3: Data Mining Software in Java"[URL]