A Development of LDA Topic Association Systems Based on Spark-Hadoop Framework
0
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
Social data such as users’ comments are unstructured in nature and up-to-date technologies for analyzingsuch data are constrained by the available storage space and processing time when fast storing and processingis required. On the other hand, it is even difficult in using a huge amount of dynamically generated social datato analyze the user features in a high speed. To solve this problem, we design and implement a topicassociation analysis system based on the latent Dirichlet allocation (LDA) model. The LDA does not requirethe training process and thus can analyze the social users’ hourly interests on different topics in an easy way. The proposed system is constructed based on the Spark framework that is located on top of Hadoop cluster. Itis advantageous of high-speed processing owing to that minimized access to hard disk is required and all theintermediately generated data are processed in the main memory. In the performance evaluation, it requiresabout 5 hours to analyze the topics for about 1 TB test social data (SNS comments). Moreover, through analyzingthe association among topics, we can track the hourly change of social users’ interests on different topics.
Abstract
Social data such as users’ comments are unstructured in nature and up-to-date technologies for analyzingsuch data are constrained by the available storage space and processing time when fast storing and processingis required. On the other hand, it is even difficult in using a huge amount of dynamically generated social datato analyze the user features in a high speed. To solve this problem, we design and implement a topicassociation analysis system based on the latent Dirichlet allocation (LDA) model. The LDA does not requirethe training process and thus can analyze the social users’ hourly interests on different topics in an easy way. The proposed system is constructed based on the Spark framework that is located on top of Hadoop cluster. Itis advantageous of high-speed processing owing to that minimized access to hard disk is required and all theintermediately generated data are processed in the main memory. In the performance evaluation, it requiresabout 5 hours to analyze the topics for about 1 TB test social data (SNS comments). Moreover, through analyzingthe association among topics, we can track the hourly change of social users’ interests on different topics.
참고문헌 (12)
- D. M. Blei, "Probabilistic topic models", Communication of the ACM, 55(4), pp.77~87 (2012)[DOI]
- K. Park, "A development of streaming big data analysis system using in-memory cluster computing framework: Spark", Lecture Notes in Electrical Engineering, 393, pp.157~163 (2016)
- M. Zaharia, "Spark: cluster computing with working sets", Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing (HotCloud'10), pp.1~7 (2010)
- J. Dean, "MapReduce: simplified data processing on large clusters", Proceedings of the 6th Conference on Symposium on Operating Systems Design & Implementation (OSDI'04), pp.137~149 (2004)
- M. Armbrust, "Spark SQL: relational data processing in Spark", Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (SIGMOD'15), pp.1383~1394 (2015)
- S. Kang, "Improving diversity using bandwagon effect for developing recommendation system", Far East Journal of Electronics and Communications, 17(3), pp.539~544 (2017)[DOI]
- D. M. Blei, "Latent Dirichlet allocation", Journal of Machine Learning Research, 3, pp.993~1022 (2003)
- M. D. Hoffman, "Online learning for latent Dirichlet allocation", Proceedings of the 23rd International Conference on Neural Information Processing Systems (NIPS'10), pp.856~864 (2010)
- V. K. Vavilapalli, "Apache Hadoop YARN: yet another resource negotiator", Proceedings of the 4th Annual Symposium on Cloud Computing (SOCC'13), pp.1~16 (2014)
- J. Park, "Distributed online machine learning for topic models", Communications of the Korean Institute of Information Scientists and Engineers, 32(7), pp.40~45 (2014)
- K. Shvachko, "The Hadoop distributed file system", Proceedings of 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), pp.1~10 (2010)
- M. Zaharia, "Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing", Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation (NSDI'12), pp.1~14 (2012)