빅데이터를 위한 H-RTGL 기반 단일 분류기 분산 처리 프레임워크 설계
Design of Distributed Processing Framework Based on H-RTGL One-class Classifier for Big Data
1
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
Purpose: The purpose of this study was to design a framework for generating one-class classification algorithm based on Hyper-Rectangle(H-RTGL) in a distributed environment connected by network. Methods: At first, we devised one-class classifier based on H-RTGL which can be performed by distributed computing nodes considering model and data parallelism. Then, we also designed facilitating components for execution of distributed processing. In the end, we validate both effectiveness and efficiency of the classifier obtained from the proposed framework by a numerical experiment using data set obtained from UCI machine learning repository. Results: We designed distributed processing framework capable of one-class classification based on H-RTGL in distributed environment consisting of physically separated computing nodes. It includes components for implementation of model and data parallelism, which enables distributed generation of classifier. From a numerical experiment, we could observe that there was no significant change of classification performance assessed by statistical test and elapsed time was reduced due to application of distributed processing in dataset with considerable size. Conclusion: Based on such result, we can conclude that application of distributed processing for generating classifier can preserve classification performance and it can improve the efficiency of classification algorithms. In addition, we suggested an idea for future research directions of this paper as well as limitation of our work.
Abstract
Purpose: The purpose of this study was to design a framework for generating one-class classification algorithm based on Hyper-Rectangle(H-RTGL) in a distributed environment connected by network. Methods: At first, we devised one-class classifier based on H-RTGL which can be performed by distributed computing nodes considering model and data parallelism. Then, we also designed facilitating components for execution of distributed processing. In the end, we validate both effectiveness and efficiency of the classifier obtained from the proposed framework by a numerical experiment using data set obtained from UCI machine learning repository. Results: We designed distributed processing framework capable of one-class classification based on H-RTGL in distributed environment consisting of physically separated computing nodes. It includes components for implementation of model and data parallelism, which enables distributed generation of classifier. From a numerical experiment, we could observe that there was no significant change of classification performance assessed by statistical test and elapsed time was reduced due to application of distributed processing in dataset with considerable size. Conclusion: Based on such result, we can conclude that application of distributed processing for generating classifier can preserve classification performance and it can improve the efficiency of classification algorithms. In addition, we suggested an idea for future research directions of this paper as well as limitation of our work.
참고문헌 (25)
- 백창화, "인공지능 서비스 특징 및 품질측정항목의 고찰과 제안", 품질경영학회지, 46(3), pp.677~694 (2018), 한국품질경영학회[DOI]
- Bekkerman, R., "Scaling up machine learning: Parallel and distributed approaches" (2011), Cambridge University Press
- Caruana, G., "An ontology enhanced parallel SVM for scalable spam filter training", Neurocomputing, 108, pp.45~57 (2013)
- Chang, E. Y., "Foundations of Large-Scale Multimedia Information Management and Retrieval", pp.213~230 (2011)
- Chao, T. S., "Introduction to semiconductor manufacturing technology" (2001), SPIE PRESS
- 조휘, "클라우드 컴퓨팅 기반의 자동차 부하정보 모니터링 시스템 개발", 품질경영학회지, 43(4), pp.573~588 (2015), 한국품질경영학회[DOI]
- Dai, W., "A mapreduce implementation of C4. 5 decision tree algorithm", International Journal of Database Theory and Application, 7(1), pp.49~60 (2014)
- Giacomelli, P., "Apache mahout cookbook" (2013), Packt Publishing
- Guo, W., "A resource aware MapReduce based parallel SVM for large scale image classifications", Neural Processing Letters, 44(1), pp.161~184 (2016)
- Hodge, V. J., "Hadoop neural network for parallel and distributed feature selection", Neural Networks, 78, pp.24~35 (2016)
- Huang, F., "Distributed latent dirichlet allocation via tensor factorization", NIPS Optimization Workshop, pp.1~5 (2014)
- Jeong, I., "Geometric one-class classifiers using hyper-rectangles for knowledge extraction", Expert Systems with Applications, 117, pp.112~124 (2019)
- Jin, R., "Efficient parallel spectral clustering algorithm design for large data sets under cloud computing environment", Journal of Cloud Computing: Advances, Systems and Applications, 2(1), pp.18~ (2013)
- Khan, S. S., "One-class classification: taxonomy of study and review of techniques", The Knowledge Engineering Review, 29(3), pp.345~374 (2014)
- 김도균, "가우시안 기반 Hyper-Rectangle 생성을 이용한 효율적 단일 분류기", 한국산업경영시스템학회지, 41(2), pp.56~64 (2018), 한국산업경영시스템학회[DOI]
- Kim, D. G., "A distributed processing framework based on H-RTGL for an efficient One-Class Classification of Big Data", Proceedings of the Korean Society for Quality Management Conference, pp.137~ (2020)
- Kim, H., "Deepspark: A spark-based distributed deep learning framework for commodity clusters", arXiv preprint arXiv:1602.08191, ~ (2016)
- Liu, Y., "The parallelization of back propagation neural network in mapreduce and spark", International Journal of Parallel Programming, 45(4), pp.760~779 (2017)
- Moritz, P., "Sparknet : Training deep networks in spark", arXiv preprint arXiv:1511.06051, ~ (2015)
- Padhy, R. P., "Big data processing with Hadoop-MapReduce in cloud systems", International Journal of Cloud Computing and Services Science, 2(1), pp.16~27 (2013)
- Parsian, M., "Data algorithms: Recipes for scaling up with hadoop and spark" (2015), O'Reilly Media, Inc
- Peralta, B., "Distributed mixture-of-experts for Big Data using PETUUM framework", 2017 36th International Conference of the Chilean Computer Science Society (SCCC), pp.1~7 (2017)
- "One-class classifier results" (2010)[URL]
- Xing, E. P., "Petuum : A new platform for distributed machine learning on big data", IEEE Transactions on Big Data, 1(2), pp.49~67 (2015)
- Zaharia, M., "Spark: Cluster computing with working sets", HotCloud, 10(10-10), pp.95~ (2010)