Design and Implementation of Incremental Learning Technology for Big Data Mining
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
We usually suffer from difficulties in treating or managing Big Data generated from various digital media and/or sensors using traditional mining techniques. Additionally, there are many problems relative to the lack of memory and the burden of the learning curve, etc. in an increasing capacity of large volumes of text when new data are continuously accumulated because we ineffectively analyze total data including data previously analyzed and collected. In this paper, we propose a general-purpose classifier and its structure to solve these problems. We depart from the current feature-reduction methods and introduce a new scheme that only adopts changed elements when new features are partially accumulated in this free-style learning environment. The incremental learning module built from a gradually progressive formation learns only changed parts of data without any re-processing of current accumulations while traditional methods re-learn total data for every adding or changing of data. Additionally, users can freely merge new data with previous data throughout the resource management procedure whenever re-learning is needed. At the end of this paper, we confirm a good performance of this method in data processing based on the Big Data environment throughout an analysis because of its learning efficiency. Also, comparing this algorithm with those of NB and SVM, we can achieve an accuracy of approximately 95% in all three models. We expect that our method will be a viable substitute for high performance and accuracy relative to large computing systems for Big Data analysis using a PC cluster environment.
Abstract
We usually suffer from difficulties in treating or managing Big Data generated from various digital media and/or sensors using traditional mining techniques. Additionally, there are many problems relative to the lack of memory and the burden of the learning curve, etc. in an increasing capacity of large volumes of text when new data are continuously accumulated because we ineffectively analyze total data including data previously analyzed and collected. In this paper, we propose a general-purpose classifier and its structure to solve these problems. We depart from the current feature-reduction methods and introduce a new scheme that only adopts changed elements when new features are partially accumulated in this free-style learning environment. The incremental learning module built from a gradually progressive formation learns only changed parts of data without any re-processing of current accumulations while traditional methods re-learn total data for every adding or changing of data. Additionally, users can freely merge new data with previous data throughout the resource management procedure whenever re-learning is needed. At the end of this paper, we confirm a good performance of this method in data processing based on the Big Data environment throughout an analysis because of its learning efficiency. Also, comparing this algorithm with those of NB and SVM, we can achieve an accuracy of approximately 95% in all three models. We expect that our method will be a viable substitute for high performance and accuracy relative to large computing systems for Big Data analysis using a PC cluster environment.
참고문헌 (14)
- 김장원, "지시적 분석을 위한 연구자 활동 기반의 연구자 히스토리 추적 서비스", 정보과학회 컴퓨팅의 실제 논문지, 20(6), pp.359~363 (2014), 한국정보과학회
- 정도헌, "최대 개념강도 인지기법을 이용한 데이터베이스 자동선택 방법에 관한 연구", 정보관리학회지, 27(3), pp.265~281 (2010), 한국정보관리학회[DOI]
- Do-Heon Jeong, "The relationship between the specificity of S&T terms and auto-classification accuracy", Proceedings of the 14th Conference of Korean Society for Information Management, pp.31~36 (2007)
- 이재윤, "문서측 자질선정을 이용한 고속 문서분류기의 성능향상에 관한 연구", Journal of Information Science Theory and Practice, 36(4), pp.51~69 (2005), 한국과학기술정보연구원
- 이재윤, "지적 구조 분석을 위한 새로운 클러스터링 기법에 관한 연구", 정보관리학회지, 23(4), pp.215~231 (2006), 한국정보관리학회[DOI]
- 이원구, "상호운용적 분류체계 관리를 위한 반자동 분류체계 관리방안", 한국콘텐츠학회 논문지, 11(12), pp.466~474 (2011), 한국콘텐츠학회
- R. Burke, "Hybrid Recommender Systems : Survey and Experiments", User Modeling and User-Adapted Interaction, 12(4), pp.331~370 (2011)
- Francesco Ricci, "Recommender Systems Handbook" (2011), Springer
- G. Adomavicius, "Toward the Next Generation of Recommender Systems : A Survey of the State-of-the-Art and Possible Extensions", IEEE Transactions on Knowledge and Data Engineering, 17(6), pp.734~749 (2005)
- R. Plutchik, "The Nature of Emotions", American Scientist, 89(4), pp.344~350 (2001)
- J. A. Russell, "Everyday Conceptions of Emotion : An introduction to the Psychology, Anthropology and Linguistics of Emotion" (1995), Kluwer Academic Publishers
- Jeong-Won Lee, "Design of Moa Contents Curation Service System Based on Incremental Learning Technology", Proceedings of the Korea Contents Association 2018 Spring Symposium, pp.401~402 (2018)
- Jeong-Won Lee, "Design and Implementation of Contents Curation LOD(Linked Open Data) Cloud Service System Based on Incremental Learning Model for Big Data Mining", Proceedings of the Korea Contents Association 2018 International Conference on Convergence Content, pp.131~132 (2018)
- 이원구, "이기종 데이터 간 상호운용적 분류체계 관리를 위한 분류체계 자동화 방안", 한국정보통신학회논문지, 15(12), pp.2609~2618 (2011), 한국정보통신학회