특허 빅데이터 분석을 위한 데이터 평활 및 이상치 제거
Big Data Smoothing and Outlier Removal for Patent Big Data Analysis
3
조회
0
다운로드
3
인용
0
참고문헌
키워드
초록
In general statistical analysis, we need to make a normal assumption. If this assumption is not satisfied, we cannot expect a good result of statistical data analysis. Most of statistical methods processing the outlier and noise also need to the assumption. But the assumption is not satisfied in big data because of its large volume and heterogeneity. So we propose a methodology based on box-plot and data smoothing for controling outlier and noise in big data analysis. The proposed methodology is not dependent upon the normal assumption. In addition, we select patent documents as target domain of big data because patent big data analysis is a important issue in management of technology. We analyze patent documents using big data learning methods for technology analysis. The collected patent data from patent databases on the world are preprocessed and analyzed by text mining and statistics. But the most researches about patent big data analysis did not consider the outlier and noise problem. This problem decreases the accuracy of prediction and increases the variance of parameter estimation. In this paper, we check the existence of the outlier and noise in patent big data. To know whether the outlier is or not in the patent big data, we use box-plot and smoothing visualization. We use the patent documents related to three dimensional printing technology to illustrate how the proposed methodology can be used for finding the existence of noise in the searched patent big data.
Abstract
In general statistical analysis, we need to make a normal assumption. If this assumption is not satisfied, we cannot expect a good result of statistical data analysis. Most of statistical methods processing the outlier and noise also need to the assumption. But the assumption is not satisfied in big data because of its large volume and heterogeneity. So we propose a methodology based on box-plot and data smoothing for controling outlier and noise in big data analysis. The proposed methodology is not dependent upon the normal assumption. In addition, we select patent documents as target domain of big data because patent big data analysis is a important issue in management of technology. We analyze patent documents using big data learning methods for technology analysis. The collected patent data from patent databases on the world are preprocessed and analyzed by text mining and statistics. But the most researches about patent big data analysis did not consider the outlier and noise problem. This problem decreases the accuracy of prediction and increases the variance of parameter estimation. In this paper, we check the existence of the outlier and noise in patent big data. To know whether the outlier is or not in the patent big data, we use box-plot and smoothing visualization. We use the patent documents related to three dimensional printing technology to illustrate how the proposed methodology can be used for finding the existence of noise in the searched patent big data.
참고문헌 (28)
- J. J. Berman, "Principles of Big Data" (2013), Morgan Kaufmann
- K. Krishnan, "Data Warehousing in the Age of Big Data" (2013), Morgan Kaufmann
- B. Chun, "A Study on Big Data Processing Mechanism & Applicability", International Journal of Software Engineering and Its Applications, 8(8), pp.73~82 (2014)
- S. Ha, "Standardization Requirements Analysis on Big Data in Public Sector based on Potential Business Models", International Journal of Software Engineering and Its Applications, 8(11), pp.165~172 (2014)
- S. Jeon, "Redundant Data Removal Technique for Efficient Big Data Search Processing", International Journal of Software Engineering and Its Applications, 7(4), pp.427~436 (2014)
- M. Riondato, "Sampling-based Randomized Algorithms for Big Data Analytics", PhD, pp.2014
- J. Lu, "Correction in a Small Sample from Big Data", IEEE Transactions on Knowledge and Data Engineering, 25(11), pp.2658~2663 (2013)
- A. T. Roper, "Forecasting and Management of Technology" (2011), John Wiley & Sons
- D. Hunt, "Patent Searching Tools & Techniques" (2007), Wiley
- J. Han, "Data Mining: Concepts and Techniques" (2012), Morgan Kaufmann
- "WIPS Corporation" (2016)[URL]
- "The United States Patent and Trademark Office" (2016)[URL]
- I. Feinerer, "Department of Statistics and Mathematics", pp.2008
- I. Feinerer, "Text mining infrastructure in R", Journal of Statistical Software, 25(5), pp.1~54 (2008)
- Package ‘tm’ Ver. 0.6, Text Mining Package, "I. Feinerer", CRAN, ~ (2016)
- S. Jun, "Technology Forecasting using Matrix Map and Patent Clustering", Industrial Management & Data Systems, 112(5), pp.786~807 (2012)
- B. L. Bowerman, "Forecasting, Time Series, and Regression, An Applied Approach, Independence", pp.2005, Brooks
- W. S. Cleveland, "LOWESS : A program for smoothing scatterplots by robust locally weighted regression", The American Statistician, 35(1), pp.54~ (1981)
- D. Ruppert, "Multivariate locally weighted least squares regression", The annals of statistics, pp.1346~1370 (1994)[DOI]
- G. Guo, "Image-based human age estimation by manifold learning and locally adjusted robust regression", IEEE Transactions on Image Processing, 17(7), pp.1178~1188 (2008)
- M. Akritas, "Probability and Statistics with R for Engineers and Scientists" (2016), Pearson
- 최준혁, "특허키워드분석을 위한 베이지안 회귀모형", 한국컴퓨터정보학회논문지, 21(1), pp.125~129 (2016), 한국컴퓨터정보학회
- S. Park, "Methodology of Technological Evolution for Three-dimensional Printing", Industrial Management & Data Systems, 116(1), pp.122~146 (2016)
- "R Development Core Team, R: A language and environment for statistical computing, R Foundation for Statistical Computing, Vienna, Austria" (2016)[URL]
- 최준혁, "A Technology Analysis Model using Dynamic Time Warping", 한국컴퓨터정보학회논문지, 20(2), pp.113~120 (2015), 한국컴퓨터정보학회
- S. Jun, "Technology Forecasting using Matrix Map and Patent Clustering", Industrial Management & Data Systems, 112(5), pp.786~807 (2012)
- S. Lee, "Key IPC Codes Extraction Using Classification and Regression Tree Structure", Advances in Intelligent Systems and Computing, 271, pp.101~109 (2014)