Operationalizing Apache Spark for the · PDF fileMapR OpenTSDB Optimization •Blob...
Transcript of Operationalizing Apache Spark for the · PDF fileMapR OpenTSDB Optimization •Blob...
The Collection of:
Sensors
Sensor Networks
Smart Machines
Computer Power
Analytics
PeopleOne element of the Industrial Internet of Things
Solving problems that were previously unsolvable
The Internet of ThingsThe Internet of Things
Network of physical objects or “things” embedded with electronics, software, sensors, and network connectivity, which enables these objects to collect and exchange data.
4BILLION
Connected People
25+MILLION
Apps
25+BILLION
Devices
50TRILLIONGBs of DATA
Use Cases
remote monitoring
SME’s manage
many machines
condition monitoring
detect early when
problems are small
share learning
learn on one
transfer to many
maintenance
sensorstreams
leverages existing infrastructure: plant historian sensor data streams, and EAM system
connect import combine analyze present
prescriptiveaction
Information Flow
Library
Benchmark Statistics
Equipment Metadata
Raw Data
Failure Signatures, PM Repository, Failure Code Hierarchy
Failure Rates,MTBF/MTTF/MTBR/MTBM,Average Repair Cost,Performance Data
Equipment Type,ISO 14224 Structure,Sensor Templates
Sensor Data,Maintenance Work Orders,Operational Events,Crowdsourced Info
Levels of Data
IoT and Big Data
Comet 67P vs Los Angeles
Scale
Data Points/Yr
Single Rig
315 B
100 rigs
31.5 T
Scale
Data Points/Yr
Mtell and OpenTSDB
Optimized for Time Series Data AccessQuerying sensor data by date range plus other filters
MapR OpenTSDB Optimization
• Blob Ingestion – 100x faster– Instead of inserting each point, buffer
data in memory and insert a blob
containing batch
– Move blob maker upstream of insertion
into the storage tier
– Less writes to disk (once / blob instead
of once/point) and reduced data size
(blob compresses raw data)
– 10 node MapR cluster achieved 100
million points/sec ingestion (10 million
points/sec/node)
What has happened?
What will happen?
What should we do?
DESCRIPTIVE
PREDICTIVE
PRESCRIPTIVE
Analytics in Maintenance
Types of Agents
Learns precise specific failure signature & performs live monitoring, providing
early warnings of recurrences
Failure Agent
Learns baseline normal & performs live monitoring to expose abnormal operations – updates as conditions
change
Anomaly Agent
Finds unrecorded failure patterns in training data & excludes suspect
data from baseline normal conditions
Hidden Failure Agent
anomaly agentknows all learned patterns
matching all normal operating states
failure agent 001knows precise signature of
patterns leading to bearing failure
failure agent 002knows precise signature of
patterns leading to drive coupling failure
failure agents 003+many other agents each
assigned to detect exact failure patterns
Many Agents Per Asset
…each one holds the
precise multi-
dimensional/temporal data
pattern of a machine in a
specific operating mode
capture worker
experiences & actual
measured sensor data
Mtell uses agents for
individual machines
created by you
…in minutes
one single job
…constantly monitor for
that exact pattern
& sound off “alarm bells”
Agents – Retained Knowledge
Find Degradation Earlier
Multi-variate
Temporal & multi-variate
Detect complex failure patterns that cannot be detected by humans, or other technologies, or seen in any single variable trend
Platform for IoT Analytics
Equipment Sets
& Taxonomy
make & model
operating context
population analysis
Equipment
asset hierarchy
sync from EAM
Sensor Mapping
data
streams
Live Agents
rules
maintenance scheduling
machine learning
M2M
population learning
transfer learning
Performance
usage, states
wear, fatigue
efficacy
benchmarking
advancedanalytics
analyst
automation
population
learning
deep
learning
fleet bench-
marking
reservoirsignature
library
sensor
data store
cloud
sync
fault/eff.
signatures
signature
search
operatingcentermgmt.
incident
response
immersive
visualization
adaptive
feedback
knowledge
capture
intelligentsignal
processing
audit
automation
instrument
reliability
derived
signal mgt.
interpola-
tion
globalequipmenttaxonomy
eqmt model
catalog
Industry
op. context
fleet sensor
templates
sensor
groups
Platform Functions
Transfer Learning Signature Library
Template
Signature
Pump 02
Pump 01
Library of Known
Failure Signatures
Time-Series
Sensor Data
A
B
C
A
B
C
Spark RDD
RDD – Resilient Distributed DataSet
• Read-only collection partitioned across a set of machines
• Can be rebuilt if partition lost
• Enables spark to outperform Hadoop 10x on iterative machine learning jobs
Query data via HTTP
Over time builds RDD Data-frame; distributed across nodes
Aim: query database only once
Distributed data storage system
Stores high-precision data points
Scales almost linearly
But lacks analytics
Mtell REST API
Any request
Any client
Flexible/scalable link to any Python-based machine learning libraries
Human friendly
Spark Integration
Automated & Self-ImprovingKnown Failure Signature
prescriptive maintenance
well in advancefixing a small problem
before it’s a big one
Automated & Self-ImprovingLearn New Failure Signature
search deeper & improve
7-day anomaly alert 30+ day failure signature alert
Predictive / Prescriptive Analytics
Maintenance costsdecrease dramatically
Machineslast longer
Net outputincreases dramatically
Critical Assets stopbreaking down
Alex [email protected]