스파크 클러스터 환경에서의 대규모 로그 이상 패턴 분석
Anomalous Pattern Analysis of Large-Scale Logs with Spark Cluster Environment
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
본 연구는 Spark 클러스터 환경에서 대용량 로그를 분석하여 시스템 이상과의 연관성을 탐색한다. 로그를 활용한 이상 감지 연구는 증가하고 있으나, 클러스터의 다양한 컴포넌트의 로그를 충분히 활용하지 못하고 이상과 시스템의 연관성을 고려하지 않는다는 한계가 있다. 따라서 본 논문에서는 정상과 비정상 로그의 분포를 분석하고, 로그 템플릿의 출현 여부를 통해 이상 감지 가능성을 탐색한다. Hadoop과 Spark를 활용하여 정상과 비정상 로그 데이터를 생성하고, t-SNE와K-means 클러스터링을 통해 비정상 상황에서의 로그 템플릿을 찾아 이상 현상을 파악한다. 결과적으로, 비정상 상황에서만 발생하는 고유한 로그 템플릿을 확인하며 이를 통해 이상 현상 감지의 가능성을 제시한다.
Abstract
This study explores the correlation between system anomalies and large-scale logs within the Spark cluster environment. While research on anomaly detection using logs is growing, there remains a limitation in adequately leveraging logs from various components of the cluster and considering the relationship between anomalies and the system. Therefore, this paper analyzes the distribution of normal and abnormal logs and explores the potential for anomaly detection based on the occurrence of log templates. By employing Hadoop and Spark, normal and abnormal log data are generated, and through t-SNE and K-means clustering, templates of abnormal logs in anomalous situations are identified to comprehend anomalies. Ultimately, unique log templates occurring only during abnormal situations are identified, thereby presenting the potential for anomaly detection.
참고문헌 (18)
- Xu Zhang, "Robust log-based anomaly detection on unstable log data", Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software, pp.807~817 (2019)
- Min Du, "Deeplog:Anomaly detection and diagnosis from system logs through deep learning", Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp.1285~1298 (2017)
- Weibin Meng, "LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs", Proceedings of the 28th International Joint Conference on Artificial Intelligence, 7, pp.4739~4745 (2019)
- Lin Yang, "Semi-supervised log-based anomaly detection via probabilistic label estimation", Proceedings of 2021 IEEE, 2021, 1448~1460[DOI]
- Van-Hoang Le, "Log-based anomaly detection without log parsing", Proceedings of the 36th IEEE, 2021, 492~504
- "Loghub"[URL]
- Nengwen Zhao, "An Empirical Investigation of Practical Log Anomaly Detection for Online Service Systems", Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software, pp.1404~1415 (2021)
- Jieming Zhu, "Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics", Proceedings of 2023 IEEE 34th International Symposium on Software Reliability Engineering, pp.355~366 (2023)
- Wei Xu, "Detecting large-scale system problems by mining console logs", Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles, pp.117~132 (2009)
- Adam Oliner, "What Supercomputers Say: A Study of Five System Logs", Proceedings of 37th Annual IEEE, 2007, 575~584
- Qingwei Lin, "Log Clustering Based Problem Identification for Online Service Systems", Proceedings of 2016 IEEE, 2016, 102~111
- Jian-Guang Lou, "Mining Program Workflow from Interleaved Traces", Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining 2010, pp.613~622 (2010)
- Xiao Yu, "CloudSeer: Workflow Monitoring of Cloud Infrastructures via Interleaved Logs", Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating Systems, pp.489~502 (2016)
- R. Mahindru, "Log Anomaly to Resolution : AI Based Proactive Incident Remediation", Proceedings of IEEE, 2021, 1353~1357
- Baril Xavier, "Application Performance Anomaly Detection with LSTM on Temporal Irregularities in Logs", Proceedings of the 29th ACM International Conference on Information & Knowledge, pp.1961~1964 (2020)
- Chenxi Zhang, "DeepTraLog:Trace-Log Combined Microservice Anomaly Detection through Graph-based Deep Learning", Proceedings of the 44th International Conference on Software Engineering, pp.623~634 (2022)
- Yintong Huo, "EvLog: Identifying Anomalous Logs over Software Evolution", Proceedings of 2023 IEEE 34th International Symposium on Software Reliability Engineering, pp.391~402 (2023)
- "Apache Spark Examples"[URL]