Discussion for Anomaly & Prediction Engine
-
Upload
hisashiosanai -
Category
Software
-
view
304 -
download
0
Transcript of Discussion for Anomaly & Prediction Engine
Fujitsu Standard Tool
Copyright 2016 FUJITSU LIMITEDDiscussion for Anomaly & Prediction Engine04 Feb. 2016Hisashi Osanai
0
Copyright 2016 FUJITSU LIMITED0
AgendaCopyright 2016 FUJITSU LIMITEDPOC IntroductionPOC DemoSystem ConfigurationParallel distributed processing platformEx. Batch process / Stream processFindings/Problems from POCWhy Im interested in Monasca Current Concerns and Approach1
Copyright 2016 FUJITSU LIMITED1
POC DemoCopyright 2016 FUJITSU LIMITED
2
Copyright 2016 FUJITSU LIMITED2
Copyright 2016 FUJITSU LIMITEDDemo System ConfigurationMaster serverVisualization serverOSElasticSearchApache(httpd)KibanaJDKOScollection/storedefinitionHadoopSparkfluentdRabbitMQParallel distributed processing platformprocess definitionStream processSparkStreaming/SparkSQLDataconverterTarget server#1OSfluentdcollection definitionfluentdcollection/store definitionSlave server#3SparkOSHadoopJDKJDK
Data collection targetSlave server#2SparkOSHadoopJDKSlave server#1SparkOSHadoopJDKBatchprocessTaskcontrollerTarget server#2OSfluentdcollection definitionTarget server#nOSfluentdcollection definition3
Copyright 2016 FUJITSU LIMITED3
Copyright 2016 FUJITSU LIMITEDParallel distributed processing platformApache Spark(Core)SparkSQL(SQL query)SparkStreaming(Event stream processing)
Parallel distributed processing platformJob Definition(XML)RabbitMQ(Messagebroker)Fluentd(Data collector)HDFS(Distributed File System )ElasticSearch(Real time search engine)Kibana(Data visualization)
Stream data receptionData process with SQLCreate time-series dataAnalysis processEx. stream data analysis in the anomaly detection process
Enable to execute Stream process and Batch process Fast-acting data conversion based on XML-based Job Definition4
Copyright 2016 FUJITSU LIMITED4
Copyright 2016 FUJITSU LIMITEDEx. Batch processParallel distributed processing platformJob definition (XML)TASK:1Read master dataSparkBatchApplicationTASK:2Read Web access logWeb access log
Analysis TASK:3Query and Save
Spark Cluster
HDFS
HDFSAnalyze a lot of Web access log on file system5
Copyright 2016 FUJITSU LIMITED5
Copyright 2016 FUJITSU LIMITEDEx. Stream processParallel distributed processing platformJob definition (XML)RabbitMQReceiverRabbitMQTASK:1Process and store the CPU information
HDFSSparkStreamingApplicationTASK:2Process and store the MEM information
Analysis
Target server
Analyze statistics information (CPU/MEM) in real-time6
Copyright 2016 FUJITSU LIMITED6
Copyright 2016 FUJITSU LIMITEDFindings/Problems from POCNeeds manpower for data collection on target servers Have discussions with customers to define collecting data and then configure fluentd agents (Num of POCs is limited)
Difficult to store experiences of IT analytics Data and its format are different each customer so suitable anomaly detection libraries are also different
Difficult to catch up for anomaly detection librariesRapid tech evolution for Machine Learning such as Mllib, TensorFlow, CNTK and so on 7
Copyright 2016 FUJITSU LIMITED7
Copyright 2016 FUJITSU LIMITEDSeems to solve two problems from POCNeeds manpower for data collection on target servers Monasca provides agents for OpenStack env so we just use them.Difficult to store experiences of IT analytics Data come from Monasca agents and the format is stable. So we use the data as stable input and are looking for which libraries are suitable for this env which is monitored by Monasca
Add a catching function to MonascaBoosts Monasca salesA lot of our customers are interested in IT analyticsFujitsu sells Monasca-based product Why Im interested in Monasca8
Copyright 2016 FUJITSU LIMITED8
Copyright 2016 FUJITSU LIMITEDCurrent ConcernsPerformance for real time anomaly detection (Storm vs. ApacheStreaming)Rapid tech evolution for Machine Learning (Needs to have plugin arch for the libraries)
Approach (a base for discussion)How to move Anomaly & Prediction Engine (APE) dev ahead?IdeaFirst Rebase current prototype on Monasca master (If possible, I would like to do this with Rolands help)Then use it to find out problems
Current Concerns & Approach9
Copyright 2016 FUJITSU LIMITED9
Copyright 2016 FUJITSU LIMITED10