정확하고 효율적인 로그 템플릿 발견 기법
Accurate and Efficient Log Template Discovery Technique
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
In this paper we propose a novel log template discovery algorithm which achieves high quality of discovered log templates through iterative log filtering technique. Log templates are the static string pattern of logs that are used to produce actual logs by inserting variable values during runtime. Identifying individual logs into their template category correctly enables us to conduct automated analysis using state-of-the-art machine learning techniques. Our technique looks at the group of logs column-wise and filters the logs that have the value of the highest proportion. We repeat this process per each column until we are left with highly homogeneous set of logs that most likely belong to the same log template category. Then, we determine which column is the static part and which is the variable part by vertically comparing all the logs in the group. This process repeats until we have discovered all the templates from given logs. Also, during this process we discover the custom patterns such as ID formats that are unique to the application. This information helps us quickly identify such strings in the logs as variable parts thereby further increasing the accuracy of the discovered log templates. Existing solutions suffer from log templates being too general or too specific because of the inability to detect custom patterns. Through extensive evaluations we have learned that our proposed method achieves 2 to 20 times better accuracy.
Abstract
In this paper we propose a novel log template discovery algorithm which achieves high quality of discovered log templates through iterative log filtering technique. Log templates are the static string pattern of logs that are used to produce actual logs by inserting variable values during runtime. Identifying individual logs into their template category correctly enables us to conduct automated analysis using state-of-the-art machine learning techniques. Our technique looks at the group of logs column-wise and filters the logs that have the value of the highest proportion. We repeat this process per each column until we are left with highly homogeneous set of logs that most likely belong to the same log template category. Then, we determine which column is the static part and which is the variable part by vertically comparing all the logs in the group. This process repeats until we have discovered all the templates from given logs. Also, during this process we discover the custom patterns such as ID formats that are unique to the application. This information helps us quickly identify such strings in the logs as variable parts thereby further increasing the accuracy of the discovered log templates. Existing solutions suffer from log templates being too general or too specific because of the inability to detect custom patterns. Through extensive evaluations we have learned that our proposed method achieves 2 to 20 times better accuracy.
참고문헌 (25)
- K. Kc, "Efficient log-based troubleshooting system for computing infrastructures", IEEE International Symposium on Reliable Distributed Systems (SRDS) (2011)
- B. C. Tak, "LOGAN: Problem Diagnosis in the Cloud Using Log-based Reference Models", IEEE International Conference on Cloud Engineering (IC2E), pp.62~67 (2016)
- X. Fu, "Digging deeper into cluster system logs for failure prediction and root cause diagnosis", 2014 IEEE International Conference on Cluster Computing (CLUSTER), pp.103~112 (2014), IEEE
- Min Du, "DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning", Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (2017), ACM
- W. Xu, "Detecting large-scale system problems by mining console logs", Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, SOSP '09, pp.117~132 (2009), ACM
- J.-G. Lou, "Mining invariants from console logs for system problem detection", Proceedings of the 2010 USENIX Conference on USENIX Annual Technical Conference, USENIX ATC'10 (2010), USENIX Association
- C. Lim, "A log mining approach to failure analysis of enterprise telephony systems", IEEE International Conference on Dependable Systems and Networks (2008)
- K. Nagaraj, "Structured comparative analysis of systems logs to diagnose performance problems", Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, NSDI'12 (2012)
- Risto Vaarandi, "A Data Clustering Algorithm for Mining Patterns from Event Logs", Proceedings of the 3rd IEEE Workshop on IP Operations & Management (IPOM), pp.119~126 (2003)
- R Vaarandi, "Mining event logs with SLCT and LogHound", 2008 IEEE Network Operations and Management Symposium (NOMS) (2008)
- A. Makanju, "A lightweight algorithm for message type extraction in system application logs", IEEE Transactions on Knowledge and Data Engineering (2012)
- A. A. Makanju, "Clustering event logs using iterative partitioning", KDD '09: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp.1255~1264 (2009)
- L. Tang, "Logsig: Generating system events from raw textual logs", Proceedings of ACM Conference on Information and Knowledge Management (CIKM) (2011)
- Pinjia He, "Drain: An Online Log Parsing Approach with Fixed Depth Tree", Proceedings of the IEEE International Conference on Web Services (ICWS 2017), pp.33~40 (2017)
- Q. Fu, "Execution anomaly detection in distributed systems through unstructured log analysis", ICDM'09: Proc. of International Conference on Data Mining (2009)
- "Apache Hadoop"[URL]
- "OpenStack"[URL]
- "Apache Cassandra"[URL]
- P. He, "An Evaluation Study on Log Parsing and Its Use in Log Mining", Proceedings of DSN'16: Proc. of the 46th Annual IEEE, 2016, ~
- L. Tang, "Logtree: A framework for generating system events from raw textual logs", Proceedings of the 2010 IEEE International Conference on Data Mining, ICDM '10 (2010), IEEE Computer Society
- M. Mizutani, "Incremental mining of system log format", SCC'13: Proc. of the 10th International Conference on Services Computing (2013)
- "Apache Hadoop YARN"[URL]
- Fei Wu, "Structural Event Detection from Log Messages", Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2017)
- Animesh Nandi, "Anomaly Detection Using Program Control Flow Graph Mining from Execution Logs", Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016)
- Rui Ding, "Log 2 : A Cost-Aware Logging Mechanism for Performance Diagnosis", Proceedings of the 2015 USENIX Conference on USENIX Annual Technical Conference, USENIX ATC'15 (2010), USENIX Association