A Mind Map Based Framework for Automated Software Log File Analysis
18
Dileepa Jayathilake A Mind Map Based Framework for Automated Software Log File Analysis Department of Electrical Engineering University of Moratuwa Sri Lanka ICSCA 2011
-
Upload
dileepa-jayathilake -
Category
Technology
-
view
930 -
download
1
description
Software log file analysis is involved heavily in both Software development and maintenance phases. It serves for various purposes such as verifying the conformance of the software functionality to the specification, software quality check and troubleshooting. Application log files or the logs generated by other monitoring tools are subjected to analysis for extracting information that can be vital in an investigation. These tasks demand expertise to a great deal and are labor intensive when performed manually. The lack of a commonly used technique to record expert knowledge stands as an impediment to automate the analysis tasks. The need for correlating information extracted from different locations in the same log file or multiple log files further ads to this complexity. This paper describes a framework based on mind maps which formulates a homogeneous platform for recording expert knowledge as well as for performing other tasks such as extracting information from log files, drawing inferences and creating reports. The framework includes a scripting language, a parallel application programming interface and a set of tools. Usage is illustrated by a proof of concept system built using the framework that creates a useful report after analyzing a log file generated by a widely used software monitoring tool. My speech in ICSCA 2011 - http://dileepaj.blogspot.com/2011/07/speech-in-icsca-2011.html
Transcript of A Mind Map Based Framework for Automated Software Log File Analysis
Dileepa Jayathilake
A Mind Map Based Framework for Automated Software Log File Analysis
Department of Electrical Engineering University of Moratuwa Sri LankaICSCA
2011
Background
Problem Identification
Solution Overview
Solution Design
Implementation
Conclusion
AG
EN
DA
BACKGROUNDFunctional Conformance
Quality Verification
Troubleshooting
System AdministratorsDomain Experts
DevelopersApplication Logs
Monitoring Tool Logs
LOG FILE ANALYSIS
Testers
Require Expertise
Labor Intensive
Error-prone
Advantage of Recurrence not used
BACKGROUND
PITFALLS IN MANUAL APPROACH
PROBLE
M
IDENTI
FICAT
ION
Challenges
Result
Automation abandoned
Proprietary Implementation
Costly
Rules not human readable
Difficult to add new rules
Less resilient to format changes
CHALLENGES
Reports not customizable
Different log formats & structure
Lack of a common platform
Making rules human & machine
readable
XML
Universal format
Ubiquitous use
Many tools available
Costly meta data
Less human readable
Associated languages are complex
Not every log is xml
Log File Grammars Formal definitions
Regular expression based
Assume line logs
Fail with complex log file structures
Unable to handle difficult syntax
Distant from XML
EXISTING SUPPORT
PROBLE
M
IDENTI
FICAT
ION
Handle arbitrary formats and structures of log files
In lined with XML
Friendly for non-developers
Ability to generate custom reports
A GENERIC LOG ANALYSIS FRAMEWORK
+
Resilient to log file format and structure changes
A knowledge representation which is both human and machine readable
EXPECTA
TIONS
SOLUTI
ON
OVERVIEW
InterpretationUnified mechanism for extracting information of interest from both text and binary log files with arbitrary structure and format
ProcessingEasy mechanism to build and maintain a rule base for inferences
PresentationFlexible means for generating custom reports from inferences
Log Files
Knowledge Representation Schema
SOLUTION OVERVIEW
SOLUTI
ON
OVERVIEW
Resembles human knowledge organization better
MIND MAPS
Easy to add content
Easy to visualize
Easy access to computers
Tree
Can utilize existing tree algorithms
Easily convertible to XML
Can utilize existing tools
Easy to combine
MIND MAP AS KNOWLEDGE UNIT
SOLUTI
ON
DESIGN
SOLUTI
ON
DESIGN
SYSTEM ARCHITECTURE
SOLUTI
ON
IMPL
EMENTATI
ONNEW SCRIPTING LANGUAGE
Mind map is the basic processing unit
Configurable syntax
Advanced filtering
Multiple executions in a single statementSupports basic and compound data typesBuilt-in and user defined functions
$Map1.TypeIs(#ERROR)::$MY.LeftSibling.IsNotNull, Level < 2.LeftSibling->Category.Unique.Count = $ERROR_CATEGORIES_COUNT
Follows the flow of a text in natural language
Uses statement chaining
No distant memory calls
More suitable for expressing rules
Independent small chunks of execution
$Found = FALSE$Map1.TypeIs(#ERROR) = $Set1$Set1.Unique = $Errors$Map1.TypeIs(#WARN) = $Set2$Set2.Unique = $WarningsForeach $Error in $Errors $Error->Category = $Cat $Warnings::Category==$Cat = $X If ( $X.Count > 0 ) $Found = TRUE Break EndIfEndFor
Suits Advanced Programming
Difficult for non-developers
Memory intensive
PROGRAMMING MODELS
SOLUTI
ON
IMPL
EMENTATI
ON
Interpretation Special support for splitting text and binary data Support for structural data extraction
Processing Presentation
Log Files
Mind Maps
SOLUTION SUMMARY
Rich platform to add and edit rules Support for combining mind maps Turing complete
Custom reports generated by scripts
SOLUTI
ON
IMPL
EMENTATI
ON
SOLUTI
ON
IMPL
EMENTATI
ON
USAGE SCENARIO
CONCLUSION
The new frameworkprovides a unified platform for generic
log analysis. It enables users to perform different tasks in a homogeneous fashion. In addition it formulates infrastructure for
a shared rule base.
FUTURE WORK
Interpretation Script library for common tool logs Declarative language
Processing Presentation Support for fuzzy rules Design driven reports
REFERENCES1. J. Valdman. Log file analysis. Technical Report DCSE/TR-2001-04, Department of
Computer Science and Engineering (FAV UWB), 2001.
2. Tony Buzan. The Mind Map Book. Penguin Books, 1996, ch. 2
3. John E. Hopcroft, Jeffery D. Ullman. Introduction to Automata Theory, Languages and Computation. Addison-Wesley, 1979, pp. 13-137
4. J. H. Andrews. Theory and practice of log file analysis. Technical Report 524, Department of Computer Science, University of Western Ontario, May 1998.
5. S. G. Eick, M. C. Nelson, J. D. Schmidt. Graphical Analysis of Computer Log Files. Communications of the ACM, Vol. 37, No. 12, pp. 50-56, 1994.
6. H. Saneifar, S. Bonniol, A. Laurent, P. Poncelet. Mining for relevant terms from log files. In: KDIR’09. Proc. of International Conference on Knowledge Discovery and Information Retrieval. Madeira, Portugal. 2009.
7. H. Saneifar, S. Bonniol, A. Laurent, P. Poncelet. Terminology extraction from log files. In: KDIR’09. Proc. Of 20th International Conference on Database and Expert Systems Applications. pp. 769-776. Lecture Notes in Computer Science, Springer 2009.
QUESTIONS
